Azure Data Factory V2 – Handling Daylight Savings using Azure Functions – Page 2.

In my previous blog, we set up an Azure Function to return current daylight saving time status for a timezone.

In this blog, we will create a new Version 2 Data Factory, and import an ARM template with a Pipeline to adjust for daylight savings. We can import this template into an existing Data Factory if required.

To begin, sign into the Azure Portal using an account with permissions to create new resources / services.

Click on the + Create a resource to create a new resource..

Type Data Factory into the search field and press Enter.

Select Data Factory , publisher Microsoft.

Click Create

Fill in details for the new services

    • Type in a Data Factory name.
    • Select the Subscription.
    • Select either Create new or Use existing resource group.
    • Select V2 (Preview)
    • Select a Location.

Click Create, and in a few minutes the Data Factory will be created.

Now we can import the ARM template with the Pipeline.
Go to Azure Custom Deployment to open the custom deployment page in Azure. Sign in if you need to.

Select Build your own template in the editor

Replace the text in the template with the following JSON.

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "factoryName": {
            "type": "string",
            "metadata": "Data Factory Name",
            "defaultValue": "Azure Data Factory"
        },
        "dstCheckURL": {
            "type": "string",
            "metadata": "URL for the DST Check function"
        }
    },
    "variables": {
        "factoryId": "[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]"
    },
    "resources": [
        {
            "name": "[concat(parameters('factoryName'), '/PL DST Adjustment')]",
            "type": "Microsoft.DataFactory/factories/pipelines",
            "apiVersion": "2017-09-01-preview",
            "properties": {
                "description": "Pipeline to adjust trigger run time depending on DST status.",
                "activities": [
                    {
                        "name": "Get DST Flag",
                        "description": "Call Azure Function to check current DST status.",
                        "type": "WebActivity",
                        "dependsOn": [],
                        "policy": {
                            "timeout": "7.00:00:00",
                            "retry": 0,
                            "retryIntervalInSeconds": 30,
                            "secureOutput": false
                        },
                        "userProperties": [],
                        "typeProperties": {
                            "url": "[parameters('dstCheckURL')]",
                            "method": "GET",
                            "headers": {}
                        }
                    },
                    {
                        "name": "Check DST Flag",
                        "description": "Check DST Flag and Delay 3600 Seconds (1 Hour) when FALSE",
                        "type": "IfCondition",
                        "dependsOn": [
                            {
                                "activity": "Get DST Flag",
                                "dependencyConditions": [
                                    "Succeeded"
                                ]
                            }
                        ],
                        "userProperties": [],
                        "typeProperties": {
                            "expression": {
                                "value": "@activity('Get DST Flag').output.isDST",
                                "type": "Expression"
                            },
                            "ifFalseActivities": [
                                {
                                    "name": "DST False",
                                    "description": "DST False - Delay 3600 Seconds.",
                                    "type": "Wait",
                                    "dependsOn": [],
                                    "userProperties": [],
                                    "typeProperties": {
                                        "waitTimeInSeconds": 3600
                                    }
                                }
                            ]
                        }
                    }
                ]
            },
            "dependsOn": []
        }
    ]
}

And press Save

The Template page should be displayed. Fill out the details as required, and click Purchase

This should deploy the pipeline into your Data Factory.
When that is complete. Open your Data Factory, select Author and Monitor
From the Let’s get started screen, select the pencil/Author icon on the left.

Click on Pipeline to expand. Hopefully a PL DST Adjustment pipeline is present.

To use this Pipeline, create a General / Execute Pipeline activity in your Pipeline, and in the Settings tab use the PL DST Adjustment pipeline as the Invoked Pipeline

Don’t forget to click on Advanced and set the Wait on completion setting so the current pipeline waits for the execution to complete.
This will give your pipeline a 60 minute delay when daylight savings is FALSE.

Note: When scheduling your pipeline, Schedule it to run at the time it will run when daylight savings is True.

Don’t forget to publish after completing your changes. Please feel free to let me know if there are any issues with this, or you would like more details on anything.

Thank you for visiting!!

4 Comments

  1. I needed to change the activity check to @activity(‘Get DST Flag’).output.isDST

    But apart from that very good. Pity in 2020 MS can still not be bothered to implement timezones, its a bit poor…

    Thanks again
    Richard

    • admin

      The Real Person!

      Author admin acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.

      Hi Richard,
      Thank you for the comment.
      I have noticed regression like this with some of the expressions over time.
      The JSON above is updated.

      -rob

  2. I’ve found a simple work around is to start your pipeline with the following if condition.

    if(Equals(addHours(convertTimeZone(utcNow(), ‘UTC’, ‘Central Standard Time’), 5, ‘HH’),utcNow(‘HH’)),true,false)

    Lets break the if down into the following lines…

    1. if(
    2. Equals(
    3. addHours(convertTimeZone(utcNow(), ‘UTC’, ‘Central Standard Time’), 5, ‘HH’)
    4. ,utcNow(‘HH’)
    5. )
    6. ,true,false)

    In line 3 we get current time in UTC (19:30) and convert to CST (15:30), add 5 hours, and then take just the hours value 19:30 >> 15:30 + 5:00 = 19

    In line 4 we are getting the timestamp in UTC and extracting just the hours (HH) and in this case it is 19.

    Finally this is wrapped in an “Equals” statement… if the two hours values are equal then true, else false.

    If true, we do nothing and continue on to the next step of the pipeline. If false then we are no longer in day light savings time and we wait one hour to continue. I have found this works even when the difference between UTC and CST spans midnight.

    Below is the json….

    {
    “name”: “If DST”,
    “type”: “IfCondition”,
    “dependsOn”: [],
    “userProperties”: [],
    “typeProperties”: {
    “expression”: {
    “value”: “@if(Equals(addHours(convertTimeZone(utcNow(), ‘UTC’, ‘Central Standard Time’), 5, ‘HH’),utcNow(‘HH’)),true,false)”,
    “type”: “Expression”
    },
    “ifFalseActivities”: [
    {
    “name”: “Wait 1 hour”,
    “type”: “Wait”,
    “dependsOn”: [],
    “userProperties”: [],
    “typeProperties”: {
    “waitTimeInSeconds”: 3600
    }
    }
    ]
    }
    },

    • Hi Kevin,
      This is a pretty nifty solution. Thank you for sharing!

      A lot easier without having to deploy the Azure Function.

      -rob

Leave a Reply

Your email address will not be published. Required fields are marked *