Matt How

Matt How's Blog

Using Lookup, Execute Pipeline and For Each Activity in Azure Data Factory V2

In my previous blog I looked how we can utilise pipeline parameters to variablise certain aspects of a dataset to avoid duplication of work. In this blog I will take that one step further and use parameters to feed into a For Each activity that can iterate over a data object and perform an action for each item. This blog assumes some prior knowledge of Azure Data Factory and it won’t hurt to read my previous blog

Previously I showed how to use a parameter file to copy a single table from Azure SQL DB down into a blob. Now lets use the For Each activity to fetch every table in the database from a single pipeline run. The benefit of doing this is that we don’t have to create any more linked services of data sets, we are only going to create one more pipeline that will contain the loop. The big picture here looks like below.


The first activity to note is the lookup activity. This can go off and fetch a value from either SQL or JSON based sources and then incorporate that value into activities further down the chain.


Here we are using SQL and you can see that we have supplied a SQL query that will fetch the schema and table names of our database. One “gotcha” is that even though you supply a SQL query, you still need to provide a dummy table name in the SQL dataset. It will use the query above at run time but won’t pass deployment without a table name. Also note that at this point, we do nothing with the returned value.

Next, we have the Execute Pipeline activity which can accept input parameters and pass those down into the executed pipelines (or child pipelines as per the diagram).


Within the type properties we can specify the parameters we want to pass in. The names here need to match whatever parameters we specify in the child pipeline but for the “value” we can make use of the new expression language to get a hold of the output of the previous lookup activity. We then reference the pipeline we want to execute, and that we need to wait for it to complete before continuing with our parent pipeline. Finally, we use the “dependsOn” attribute to ensure that our Execute Pipeline activity occurs AFTER our lookup has completed successfully.

At this point we have told the child pipeline which tables to copy and then told it to start. Our child pipeline now just needs to iterate over that list and produce our output files. To do this it only needs one activity which is the For Each. The For Each really has two components which are the outer configurables (such as the items to iterate over) and then the inner activity to perform on each item. The outer section looks like this:


Here we can configure the “isSequential” property which when set to “false” allows Data Factory to parallelise the inner activity, otherwise it will run each activity one after another. The other property is the “items” which is what the activity will iterate through. Because we fed the table list in to the “tableList” parameter from the Execute Pipeline activity we can specify that as our list of items.

Now for the inner activity:


Whilst this is a fairly chunky bit of JSON, those familiar with the copy activity in ADF V1 will probably feel pretty comfortable with this. The key difference is that we are again making use of expressions and parameters to make our template generic. You can see in the “output” attribute we are dynamically specifying the output blob name by using the schema and table name properties gleaned from our input data set. Also, in the source attribute we dynamically build our SQL query to select all the data from the table that we are currently on using the @item() property. This method of combing text and parameters is called string interpolation and allows us to easily mix static and dynamic content without the needed for additional functions or syntax.

That’s it! By making use of only a few extra activities we can really easily do tasks that would have taken much longer in previous versions of ADF. You can find the full collection of JSON objects using this link: Watch this space for the next blog which will look at custom logging of data factory activities!

Comments (7) -

  • Marius

    10/25/2017 8:58:37 PM | Reply

    Awesome post.  It's very hard to find information and practical examples.  The Microsoft documentation is fairly light at the moment.  Please do more!

  • Marius

    10/25/2017 9:16:09 PM | Reply

    Great post. Is the lookup activity the only way to get data from a source and feed that back into other activities in the pipeline?
    I'm executing a stored procedure on Azure SQL DW  using the stored proc activity. The  SP populates a dimension table and then outputs the number of records that was inserted/updated.  Is there a way to get that info back into the pipeline and pass it on the a next stored proc activity that updates a metadata table?

    Thanks in advance.

    • matthow

      10/30/2017 8:35:41 AM | Reply

      Hi Marius,
      Currently, Lookup is the only way to fetch a value during execution that can be fed into other activities within that same pipeline but you can use a stored proc as your SQL statement for the Lookup. So long as your stored proc always returns a value it will work fine - if no value is returned ADF V2 will complain about an invalid stored proc. Bear in mind, If you want to carry any values between pipelines you would need to write them out to a JSON file in blob and then lookup from that. However, as another way round your problem you can query the Data Factory Activity APIs and this can give you some of the detail you require. The best way to do this is to write an Azure Function which you can call from the new Web Activity in ADF V2. Currently I top and tail each pipeline with the Web Activity which calls a logging function so I know when pipelines start and end but also how many rows were copied. Hope that helps Smile

      • Mahindar

        1/25/2018 2:16:41 PM | Reply

        Hi matthow, Great post. Recently started learning ADF for the first time. Your post gave me a very good start.
        I have a small doubt
        How are you querying APIs (as you mentioned "query the Data Factory Activity APIs") ? . I'm able to post data(like @pipeline(),@activity()) to Azure Function and see it in log. But how to check complete api, Are there any other template functions which shows complete adf API?  what are you posting to Azure Function from web activity?.

  • DaveOD

    11/30/2017 5:24:34 PM | Reply

    Excellent post Matt, very succinct and informative. Greatly appreciated, thanks.

  • raman

    12/1/2017 7:10:46 AM | Reply

    Thanks for the practical implementation examples!!

  • Stephen Davies

    12/14/2017 3:07:11 PM | Reply

    This is a great blog post.