Fabric pipelines are growing daily and include more and more Synapse Analytics and Azure Data Factory features. One feature I have repeatedly used lately is the Azure Function activity to call Durable Functions for long-running operations that I can’t simply do with Synapse or Fabric tooling.
There could be many reasons for using Durable Functions; mine is that the Data Factory becomes slow and expensive when it calls APIs from within Pipelines or Dataflows or overloads the API with calls. I create batch files from the Pipeline, then utilise Durable Functions to process them by first queueing them in storage queues and executing in my own rhythm. You can also use Service Bus or other queueing components; I like the Durable Function experience.
The Azure Function activity in Fabric can call HTTP-triggered Functions, but Durable ones run asynchronously. That means they run in the background, and you need to query the status to see if they’ve finished. Fortunately, Durable Functions return async-HTTP-pattern compatible URLs, so it’s a breeze to do so.
For this example, we will use the code sample from our own GitHub repo, which you can find below. You can deploy that to any Function app of your choice, as long as it’s accessible from the Internet.
ThatBlueCloudLet’s dive in.
Create a Pipeline in Fabric
Go to your favourite Fabric Workspace and add a new Pipeline. Name it anything you like.
Add Azure Function Activity and Configure
Click on Azure Function activity under the Activities top ribbon. Name it as you wish, then go to the Settings page. Click on Add Connection. You can copy the Function App’s URL from its Azure Portal blade and the Function Key under your Function on the same page.
Azure Function activity currently doesn’t support Azure AD authentication or Service Principals, so we must use the Account Key authentication.
After creating the connection, configure the rest of the Settings as below. Function Relative URL is the route of your Function, which you can find under Azure Portal. In my example, it’s sayhello, but make sure you don’t include the api part in this box.
Now, save your changes and give it a run. If you configured it correctly, you should see a green light on the Output page with the following output:
As mentioned, it’s running asynchronously. Now we have to get the URL to listen from our Function activity, and wait for it to complete.
Subscribe to TBC Weekly Newsletter!
Receive articles, tips and tricks from us and the Fabric community
SubscribeListen To What The Function Says
We’ll use Web activity next to call the statusQueryGetUri URL that Function gave us. While this was a breeze in Synapse and ADF, it’s a bit tricky in Fabric as it requires a Connection (a.k.a. Linked Service) to be created for the Web activity.
This requirement for Connection screws up the relative path that it needs, as the status URL comes back as a full URL, not relative. We'll circumvent this by replacing the Function App’s URL with an empty string in the Web activity:
Then, configure the Method as GET on the Settings page. That’s it.
Save your changes and give it another go. Your Function should have executed nicely, and your Pipeline will drink a cold beer whilst waiting for its order:
If you click on the output of the Web activity, you should be seeing the Function App saying Hello That Blue Cloud!:
Conclusion
Function Apps are a big part of the Azure ecosystem and can help enrich your Pipeline experience in many scenarios. They may not be for business teams or citizen developers, but they certainly are easy to develop by IT teams. When Microsoft brings the Azure AD authentication capability to the Function activity, it’ll be even more accessible.
How are you using Azure Functions with your Fabric, Synapse or ADF instances? Let us know in the comments!
Remember to subscribe to our TBC Weekly newsletter! You'll get a summary of the latest articles from us and the Fabric and Azure communities.
