How to deploy Azure Data Factory pipeline and its dependencies programatically using PowerShell

Since ADF unfortunately doesn’t provide a built-in way for automated deployment, to do this you have to write a custom script. Then you can run it on build server such as VSTS or TeamCity.

If you’re in the interactive mode, you need to login and select the corresponding subscription. If you run the script on build server, the subscription should be selected for you automatically.

Select-AzureRmSubscription -SubscriptionName YourSubName

Here’s the parameters you can’t get programmatically so need to specify explicitly. Last parameter is a path to JSON files that represent the resources to be deployed: linked services, datasets, and pipelines.


First, create the factory itself. Flag -Force helps to continue on error if it already exists. Then load it into a variable:

New-AzureRmDataFactory -ResourceGroupName $ResourceGroupName -Name $DataFactoryName -Location $Location -Force -ErrorAction Stop
$dataFactory = Get-AzureRmDataFactory -ResourceGroupName $ResourceGroupName -Name $DataFactoryName -ErrorAction Stop

Next three loops read files based on the following convention:

  • Linked services start with LinkedService, e.g. LinkedService-SQL.json
  • Datasets start with Dataset, e.g. Dataset-Orders.json
  • Pipelines start with Pipeline, e.g. Pipeline-CopyOrders.json

First, create linked services. If any references a gateway then create it as well. Since the cmdlet doesn’t support flag -Force, we can use -ErrorAction Continue to continue on error if it already exists:

$files = Get-ChildItem $InputFolderPath -Recurse -Include *.json -Filter LinkedService* -ErrorAction Stop
foreach ($file in $files)  
    Write-Output "Creating linked services from $($file.FullName)"
    New-AzureRmDataFactoryLinkedService -DataFactory $dataFactory -File $file.FullName -Force -ErrorAction Stop

    $json = Get-Content $file.FullName -Raw -ErrorAction Stop
    $svc = $json | ConvertFrom-Json
    $gwName = $
    if ($gwName)
        Write-Output "Creating gateway $($gwName) from $($file.FullName)" 
        New-AzureRmDataFactoryGateway -DataFactory $dataFactory -Name $gwName -ErrorAction Continue

Then create datasets:

$files = Get-ChildItem $InputFolderPath -Recurse -Include *.json -Filter Dataset* -ErrorAction Stop
foreach ($file in $files)  
    Write-Output "Creating dataset from $($file.FullName)"
    New-AzureRmDataFactoryDataset -DataFactory $dataFactory -File $file.FullName -Force -ErrorAction Stop 

And finally, pipelines:

$files = Get-ChildItem $InputFolderPath -Recurse -Include *.json -Filter Pipeline* -ErrorAction Stop
foreach ($file in $files)  
    Write-Output "Creating pipeline from $($file.FullName)"
    New-AzureRmDataFactoryPipeline -DataFactory $dataFactory -File $file.FullName -Force -ErrorAction Stop 

That’s it. By this time all pipelines should be deployed, verified and started. Happy data movement!

Posted in Programming | Tagged , | 40 Comments

How to enable tracing in ASP.NET Web API using Application Insights

If you were wondering on other day what this line from the default template does:


Then know that it does exactly what is says: enables tracing using system.diagnostics API.

To forward this tracing to Application Insights, you need to install these two packages:

Install-Package Microsoft.AspNet.WebApi.Tracing
Install-Package Microsoft.ApplicationInsights.TraceListener

and add a trace listener:

  <trace autoflush="true">
      <add name="AppInsightsListener" type="Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener, Microsoft.ApplicationInsights.TraceListener" />

That’s it! The Trace category is now being populated: web api tracing ai

Happy tracing!

Posted in Programming | Tagged , , | Leave a comment

The ObjectContext instance has been disposed and can no longer be used for operations that require a connection

TL;DR: Don’t return task before disposing context, await it instead

If you’re getting an Entity Framework exception:

System.ObjectDisposedException: The ObjectContext instance has been disposed and can no longer be used for operations that require a connection
at System.Data.Entity.Core.Objects.ObjectContext.ReleaseConnection()
at System.Data.Entity.Core.Objects.ObjectContext.d__3d`1.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Data.Entity.Core.Objects.ObjectContext.d__39.MoveNext()

in the code such as this:

public Task<Order> GetOrders()
    using (var dbContext = new MyDbContext(connectionString))
        return dbContext.Orders.ToArrayAsync();

then it means that you’re disposing the context before the task is completed. Await it, indeed:

public async Task<Order> GetOrders()
    using (var dbContext = new MyDbContext(connectionString))
        return await dbContext.Orders.ToArrayAsync();

Happy awaiting!

Posted in Programming | Tagged , | Leave a comment

Could not load file or assembly ‘Newtonsoft.Json, Version=, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference

TL;DR: Add reference to Newtonsoft.Json.dll in every project

If you’re getting a runtime exception:

System.IO.FileLoadException: Could not load file or assembly ‘Newtonsoft.Json, Version=, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040).

which by the way is impossible to troubleshoot by searching in Google or on StackOverflow because it’s way too generic and has a millions of possible causes and solutions.

In our particular case, we had to make sure that every project that directly and (what is more likely) indirectly depends on this assembly, has a reference to the version you’re using, has the nuget package installed.

Checking you have correct assembly redirects in-place won’t hurt as well:

    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
        <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
        <bindingRedirect oldVersion="" newVersion="" />

Happy building!

Posted in Programming | Tagged | Leave a comment

How to generate ids programmatically using SQL Server and Entity Framework

TL;DR: create a sequence and execute stored proc to read from it

Generating ids in the database is used to be the default and near to the only one approach but now days this trend is fading away. Here’s some scenarios discovered by me recently where it doesn’t play well:
CQRS (see the comments, also Steven’s blog)
– bulk insert into many-to-many mapping table (see my previous post)

So here’s a solution. First, create a sequence (available in SQL Server 2012 or later, and in SQL Azure):

create sequence dbo.OrderId as int
start with 1
increment by 1

Then, read from it using the built-in stored procedure:

exec sys.sp_sequence_get_range @sequence_name, @range_size, @range_first_value OUT" 

And finally, execute the stored procedure programmatically. In this particular example, using Entity Framework:

 public async Task<int> GetSequenceStart(string sequenceName, int count)
    var sequenceNameParam = new SqlParameter
        ParameterName = "@sequence_name",
        SqlDbType = SqlDbType.NVarChar,
        Direction = ParameterDirection.Input,
        Value = sequenceName
    var countParam = new SqlParameter
        ParameterName = "@range_size",
        SqlDbType = SqlDbType.Int,
        Direction = ParameterDirection.Input,
        Value = count
    var outputParam = new SqlParameter
        ParameterName = "@range_first_value",
        SqlDbType = SqlDbType.Variant,
        Direction = ParameterDirection.Output

    await _dbContext.ExecuteSqlCommandAsync(
        "exec sys.sp_sequence_get_range @sequence_name, @range_size, @range_first_value OUT",

    return (int)outputParam.Value;

The drawbacks of this solution is that it doesn’t scale that well:
– if you’re using more than one database, you have to have one of them as a master and it will receive higher load.
– each application instance must be fully aware of such hierarchy what crosses the boundaries of horizontal scaling (aka scaling out).

Happy sequencing!

Posted in Programming | Tagged , | Leave a comment

How to push NuGet package to VSO feed v3

To push a nuget package to Visual Studio Online feed v3 use the following command:

nuget.exe push MyPackage.1.0.0.nupkg
    -ConfigFile nuget.config
    -ApiKey My_Feed

The crucial part here is to include -ApiKey My_Feed, otherwise you’ll get 404 Not Found error.

Happy packaging!

Posted in Programming | Tagged , | Leave a comment

How to map linking table for many-to-many relationship using Entity Framework 6

TL;DR: use synonym.

As promised earlier to myself, I blog about interesting challenges and the ways I solved them.

The other day, me and my team faced a limitation in EF6: you can either configure a linking table for many-to-many relationship between two entities or map this table to an entity. But not both simultaneously.

For instance, for the relationship between Orders and Addresses, when one order can be linked to multiple addresses (shipping, billing, legal, etc.) and one address can be used for multiple orders.

            .HasMany(x => x.Addresses)
            .WithMany(x => x.Orders)
            .Map(c => c.ToTable("OrderAddresses")

And in the same time you’d like to have the intermediate, linking entity to be mapped too. To query it independently or for bulk operations such as insert.


This case you’ll get an exception during context configuration validation:

System.InvalidOperationException: The navigation property ‘Addresses’ declared on type ‘Order’ has been configured with conflicting mapping information.

The solution would be to create a synonym (which in my project we keep in a separate schema called syn):

create schema syn
create synonym syn.OrderAddresses for dbo.OrderAddresses

and map the intermediate entity to it:

            .ToTable("OrderAddresses", "syn");

Note the overload accepting the schema name.

Now you can bulk insert into this table directly:

using (var scope = new TransactionScope())


We’re using EntityFramework.BulkInsert but also you can try EntityFramework.Utilities that supports both bulk insert and delete. However it also has a bug and doesn’t expose an overload accepting SqlBulkCopyOptions.

Happy mapping!

Posted in Programming | Tagged | Leave a comment

How to convert Google API Service Account certificate to base64

Recently in Google Developer Console by mistake I generated a new SSL certificate for my project’s Service Account so had to convert it again from p12 file to base64 representation and store its thumbprint separately.

Here’s how you can do it:

var cert = new X509Certificate2("project-595a8425b1d7.p12", "notasecret", X509KeyStorageFlags.Exportable);
var thumbprint = cert.Thumbprint;
var base64 = Convert.ToBase64String(cert.Export(X509ContentType.Pfx, "notasecret")); // or Pkcs12 which works as well

And just to make sure output string actually works and certificate contains private key:

var test = new X509Certificate2(Convert.FromBase64String(base64), "notasecret");
Debug.Assert(test.PrivateKey != null);
Posted in Programming | Tagged , | Leave a comment

How to extract private key from pfx and remove passphrase using OpenSSL

When I tried to enable SSL for BitTorrent Sync installed on my new NAS Synology 215j it turned out it requires not pfx but private and public keys separately in base64 encoded form.

Here’s the command to extract certificate itself. It will prompt for existing pfx’s passphrase (password):

openssl pkcs12 -in synology.pfx -clcerts -nokeys -out synology.cer

To extract private key. It will prompt for pfx’s passphrase and for a passphrase to add to the key:

openssl pkcs12 -in synology.pfx -nocerts -out synology.private.key

To remove the later passphrase. Now private key doesn’t contain any:

openssl rsa -in synology.private.key -out synology.key
Posted in Infrastructure | Tagged | Leave a comment

How to support different domain names in SSL behind reverse-proxy in IIS ARR

When you don’t want to enable SSL offloading so both internet facing web site and the one behind reverse proxy are access over SSL you will receive domain name mismatch.

The solution is simple. Either:

  • enable SSL Offloading
  • enable Require Server Name Indication in bindings settings:


Posted in Infrastructure | Tagged , | Leave a comment