Written by Alex

Pass Parameters To Azure Function From Data Factory: Body, Query, Headers, Path

Azure Function, also known as Function App, is a popular Azure service that is part of the serverless stack. It allows you to run small pieces of code or “functions” in a highly scalable manner without worrying about the underlying infrastructure.

Azure Functions are used with Azure Data Factory (ADF) for a variety of reasons. For example, Azure Functions can perform specific data transformations or enrichments that cannot be done within Data Factory itself. Additionally, Azure Function can be used to trigger other services or applications as part of ADF pipeline execution.

Azure Function parameters can be passed from Data Factory using query string, path, request body, or headers. The choice of how to pass parameters depends on the implementation of the Azure Function and the HTTP method used to invoke the endpoint.

What You Will Learn:

Contents:

Overview

Azure Function can be invoked as an HTTP endpoint, where the caller passes input parameters as part of the HTTP message, and Azure Function does some work and returns result.

Now, if we put Azure Functions in the context of Azure Data Factory, we have a question of how to integrate these two together - the answer is to use Azure Function or Web activity available in Data Factory.

Additionally, in most cases, we want to pass some data (aka parameters or arguments), to the invoked Azure Function. For this, we can use mechanisms provided by the HTTP protocol: request body, query string, request path, or headers.

In the following sections, we cover how to pass parameters as part of the request body, query, path, or header. And then wrap up the conversation by discussing the use of HTTP methods in Azure Functions.

NOTE: Azure Function and Function App are used as synonyms in this post.

Prerequisites

The examples in this post assume that we have the following Azure resources created. Please familiarize yourself with them, but you don’t necessarily need to have these resources to follow the post.

Connecting To Azure Function

First, let’s figure out which Azure Data Factory Activity to use to invoke our Azure Function. There are many activities available, but we are interested in the following two:

Azure Function activity is a reasonable default, which satisfies most needs. And if we have a special use case, then we can always switch to Web Activity. In the following sections, we will be using Azure Function Activity in the examples, and mention Web Activity wherever applicable.

NOTE: Passing parameters is very similar when using Azure Function activity and Web activity, so you should be good following the examples even if you are using Web activity.

1. Body Parameters

According to Wikipedia, HTTP message body is the data bytes transmitted in a message immediately following the headers. We can think of it as a request payload where we can send any data we want.

To pass parameters to Azure Function from Data Factory as part of the request body, you should know:

NOTE: Interestingly, as of March 2023, even though Azure Data Factory doesn’t support specifying the body for a DELETE request (the body field is hidden), if you input the body for POST or PUT request, and then select back DELETE method and send the request, ADF will actually send the DELETE with the request body you specified. However, you shouldn’t rely on this behavior because it can change at any moment.

Here is the JSON code representation of the ADF pipeline. You can import this configuration into your Data Factory pipeline by clicking the “{ }” icon on the top right and pasting content there. Before saving, change “name” field in the JSON to match your pipeline name.

Pipeline JSON Code - Click to expand 
{
    "name": "AzureFunction",
    "properties": {
        "activities": [
            {
                "name": "Function1",
                "type": "AzureFunctionActivity",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "functionName": "Function1",
                    "method": "POST",
                    "body": {
                        "name": "Alex"
                    }
                },
                "linkedServiceName": {
                    "referenceName": "FuncContoso",
                    "type": "LinkedServiceReference"
                }
            }
        ],
        "annotations": [],
        "lastPublishTime": "2023-03-06T02:42:10Z"
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
}

Passing Azure Function parameters in request body Passing Azure Function parameters in request body

2. Query Parameters

Another common way to pass parameters as part of HTTP request is to use query string. It is a part of URL and specified after ?, and query parameters are separated by &, in the following example, we pass name and age parameters:

https://example.com/some/path?name=Alex&age=28.

NOTE: URL maximum length. Query parameters contribute to the overall length of the URL. And even though HTTP specification doesn’t limit the length of URL, different browser and server implementations enforce their own limits. In general, it is safe to have URLs under 2048 characters, however, you should do your research if you plan to have longer query strings or URLs.

Here is the JSON code representation of the ADF pipeline. You can import this configuration into your Data Factory pipeline by clicking the “{ }” icon on the top right and pasting content there. Before saving, change “name” field in the JSON to match your pipeline name.

Pipeline JSON Code - Click to expand 
{
    "name": "AzureFunction",
    "properties": {
        "activities": [
            {
                "name": "Function1",
                "type": "AzureFunctionActivity",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "functionName": "Function1?name=Alex&age=28",
                    "method": "GET",
                    "body": ""
                },
                "linkedServiceName": {
                    "referenceName": "FuncContoso",
                    "type": "LinkedServiceReference"
                }
            }
        ],
        "annotations": [],
        "lastPublishTime": "2023-03-07T05:55:54Z"
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
}

Passing Azure Function parameters in query string Passing Azure Function parameters in query string

3. Path Parameters

Path is a part of URL which identifies a resource within the host. Or, in simple words, it’s a part of URL that goes right after the host and before the optional query (?age=28). Path can consist of multiple parts separated by /, in the following example of Azure Function URL, /api/users/alex is path:

https://func-contoso.azurewebsites.net/api/users/alex?age=28

NOTE: In Azure Functions terminology, api is route prefix, and users/alex is route. Route prefix and route together form request path. Route prefix can be customized in host.json for the entire Function App. Route can be customized for a particular function in function.json (Python) or using attributes (C#).

Azure Functions allow customizing the request path and extracting parts of the route as parameters that can be accessed in the function body. Let’s assume that we defined our route to be users/{name}, where name is a route parameter.

Here is the JSON code representation of the ADF pipeline. You can import this configuration into your Data Factory pipeline by clicking the “{ }” icon on the top right and pasting content there. Before saving, change “name” field in the JSON to match your pipeline name.

Pipeline JSON Code - Click to expand 
{
    "name": "AzureFunction",
    "properties": {
        "activities": [
            {
                "name": "Function1",
                "type": "AzureFunctionActivity",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "functionName": "users/alex",
                    "method": "GET",
                    "body": ""
                },
                "linkedServiceName": {
                    "referenceName": "FuncContoso",
                    "type": "LinkedServiceReference"
                }
            }
        ],
        "annotations": [],
        "lastPublishTime": "2023-03-10T17:19:14Z"
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
}

Passing Azure Function parameters in request path Passing Azure Function parameters in request path

4. Header Parameters

HTTP headers are a part of HTTP request or response where we can provide additional context or metadata, they can be standard headers like Content-Type as well as custom ones. Note that any HTTP request can have headers.

NOTE: Header names are case-insensitive, this means that Content-Type and content-type header names are equivalent. However, header values can be case-sensitive depending on the use case.

Here is the JSON code representation of the ADF pipeline. You can import this configuration into your Data Factory pipeline by clicking the “{ }” icon on the top right and pasting content there. Before saving, change “name” field in the JSON to match your pipeline name.

Pipeline JSON Code - Click to expand 
{
    "name": "AzureFunction",
    "properties": {
        "activities": [
            {
                "name": "Function1",
                "type": "AzureFunctionActivity",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "functionName": "Function1",
                    "method": "GET",
                    "headers": {
                        "my-header": "some-value",
                        "x-functions-key": "7PIEewONq1GZS.."
                    },
                    "body": ""
                },
                "linkedServiceName": {
                    "referenceName": "FuncContoso",
                    "type": "LinkedServiceReference"
                }
            }
        ],
        "annotations": [],
        "lastPublishTime": "2023-03-08T17:53:05Z"
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
}

Passing Azure Function parameters in request headers Passing Azure Function parameters in request headers

HTTP Methods: GET, POST, PUT, DELETE, & Others

Azure Function can be triggered in various ways, one of them is through an HTTP request, this can be achieved by using Azure Functions HTTP trigger. After configuring the HTTP trigger, we get an HTTP endpoint in the following format (path can be customized, see path parameters):

https://<function_app_name>.azurewebsites.net/api/<function_name
For example, https://func-contoso.azurewebsites.net/api/Function1

Azure Function can accept any HTTP method, e.g. GET, POST, DELETE, etc. To make your Azure Function work with a particular HTTP method, we need to do the following two things:

  1. Specify that you want to handle a particular HTTP method in function.json (e.g. Python or JavaScript) or in the corresponding attribute (e.g. C# or Java).
  2. Write the function code to handle the request accordingly. Actually, the function can be accept all HTTP methods and have the same code that handles it. However, it’s probably not very useful.

When calling an Azure Function from Azure Data Factory, ultimately, we should use the HTTP method that our Azure Function supports and expects.

Summary

In this post, we explored how to pass parameters into Azure Function from Azure Data Factory in request body, query string, request path, and HTTP headers.

It’s important to remember that the way you pass parameters/arguments to your Azure Function depends on the function’s implementation, so you should take time to understand what parameters it expects.

With regards to Azure Data Factory, it provides all necessary means to call an Azure Function and pass parameters in the ways we discussed above. In this post, we focused on Azure Function and Web activities.

There are additional considerations that we need to make when using Azure Functions from Data Factory. Some topics such as authentication and function output will be covered in next posts, so stay tuned.