Tag Archives: powershell

How to deploy Azure Data Factory pipeline and its dependencies programatically using PowerShell

Since ADF unfortunately doesn’t provide a built-in way for automated deployment, to do this you have to write a custom script. Then you can run it on build server such as VSTS or TeamCity.

If you’re in the interactive mode, you need to login and select the corresponding subscription. If you run the script on build server, the subscription should be selected for you automatically.

Login-AzureRmAccount
Select-AzureRmSubscription -SubscriptionName YourSubName

Here’s the parameters you can’t get programmatically so need to specify explicitly. Last parameter is a path to JSON files that represent the resources to be deployed: linked services, datasets, and pipelines.

param(
    [Parameter(Mandatory=$true)][string]$ResourceGroupName,
    [Parameter(Mandatory=$true)][string]$Location,
    [Parameter(Mandatory=$true)][string]$DataFactoryName,
    [Parameter(Mandatory=$true)][string]$InputFolderPath
)

First, create the factory itself. Flag -Force helps to continue on error if it already exists. Then load it into a variable:

New-AzureRmDataFactory -ResourceGroupName $ResourceGroupName -Name $DataFactoryName -Location $Location -Force -ErrorAction Stop
$dataFactory = Get-AzureRmDataFactory -ResourceGroupName $ResourceGroupName -Name $DataFactoryName -ErrorAction Stop

Next three loops read files based on the following convention:

  • Linked services start with LinkedService, e.g. LinkedService-Dev.json
  • Datasets start with Dataset, e.g. Dataset-Orders.json
  • Pipelines start with Pipeline, e.g. Pipeline-CopyOrders.json

First, create linked services. If any references a gateway then create it as well. Since the cmdlet doesn’t support flag -Force, we can use -ErrorAction Continue to continue on error if it already exists:

$files = Get-ChildItem $InputFolderPath -Recurse -Include *.json -Filter LinkedService* -ErrorAction Stop
foreach ($file in $files)  
{
    Write-Output "Creating linked services from $($file.FullName)"
    New-AzureRmDataFactoryLinkedService -DataFactory $dataFactory -File $file.FullName -Force -ErrorAction Stop

    $json = Get-Content $file.FullName -Raw -ErrorAction Stop
    $svc = $json | ConvertFrom-Json
    $gwName = $svc.properties.typeProperties.gatewayName
    if ($gwName)
    {
        Write-Output "Creating gateway $($gwName) from $($file.FullName)" 
        New-AzureRmDataFactoryGateway -DataFactory $dataFactory -Name $gwName -ErrorAction Continue
    }
}

Then create datasets:

$files = Get-ChildItem $InputFolderPath -Recurse -Include *.json -Filter Dataset* -ErrorAction Stop
foreach ($file in $files)  
{
    Write-Output "Creating dataset from $($file.FullName)"
    New-AzureRmDataFactoryDataset -DataFactory $dataFactory -File $file.FullName -Force -ErrorAction Stop 
}

And finally, pipelines:

$files = Get-ChildItem $InputFolderPath -Recurse -Include *.json -Filter Pipeline* -ErrorAction Stop
foreach ($file in $files)  
{
    Write-Output "Creating pipeline from $($file.FullName)"
    New-AzureRmDataFactoryPipeline -DataFactory $dataFactory -File $file.FullName -Force -ErrorAction Stop 
}

That’s it. By this time all pipelines should be deployed, verified and started. Happy data movement!

How to re-create symlinks of VM configs in Hyper-V using PowerShell

Hyper-V keeps VM configs at %ProgramData%\Microsoft\Windows\Hyper-V\Virtual Machines\ as a symlink to the original location.

You may get them broken due to various reasons, e.g.:

  • Server disaster
  • Upgrade to next version of OS/Hyper-V and then rollback
  • Migration

To restore functioning you need to create a symlink for each xml config, i.e.:

mklink %ProgramData%\Microsoft\Windows\Hyper-V\Virtual Machines\{guid}.xml d:\MyVM\Virtual Machines\{guid}.xml

But how to automate this if you have tens of VMs? Here’s the command:


Get-ChildItem -Recurse *.xml | New-Symlink -LiteralPath { Join-Path -Path '%ProgramData%\Microsoft\Windows\Hyper-V\Virtual Machines\' -ChildPath $_.Name } -TargetPath { $_.FullName }

How to select Azure subscription if you have more than one using PowerShell

If you have more than one Azure subscription in your account and try to upload a vhd using the instruction:

  1. Get-AzurePublishSettingsFile
  2. Import-AzurePublishSettingsFile d:\credentials.publishsettings
  3. Add-AzureVhd -LocalFilePath d:\my.vhd -Destination http://example.com/blob/container

You may get an error saying that selected account doesn’t have given blob.
That’s because the first subscription is selected by default and the target blob is in the another one.

To select the proper subscription use the following command:

Get-AzureSubscription | Select -Last 1 | Select-AzureSubscription

(For instance, the last one).