{{'2017-07-18T10:54:18.2503625Z' | utcToLocalDate }}
Richie Lee

Uploading Files To Data Lake Storage With PowerShell Part Two

Carrying on from our previous post on automating the process of uploading files to Azure Data Lake Store, we will check if a Resource Group exists, and if it does not then it will create it. Although the entire script is available on Git (posted below) I’m going to go into one function per post so that I can go in greater depth. Part One of this blog series focused on logging in to an Azure Subscription.

Today’s function starts on line 42 and is called Set-AzureResourceGroup. Before we go into it though, I want to take a moment to outline the importance of Approved Verbs in PowerShell. These things are important. Take a look at the defintion of “Set” on the Approved Verbs MSDN page, and you can see it describes exactly what this function does:

“The Set verb is used to modify an existing resource, optionally creating the resource if it does not exist, such as the Set-Variable cmdlet.”

Boom. In one verb we know exactly waht this function will do. No documentation. No comments that can go stale. Just an implied understanding that the functions were named using Approved Verbs.

Returning to today’s subject, Set-AzureResourceGroup:



Using the PowerShell Azure functions, we check if the ResourceGroup exists by executing “Get-AzureRmResourceGroup” and catch an exception, if one is thrown. We then check if that exception contains a string matching that "the Resource Group does not exist", and go on to create it. If it already exists we exit without making any changes.

I mentioned in Part One that the scripts were written with idempotence in mind: that is, an operation that has no side effects if called more than once. So this script can be run and re-run, and not change the state of anything else. This is important when writing scripts that are going to be used to deploy: you want them to repeat the same actions regardless of how many times they have been run. An example of this script not being idempotent would be if instead of exiting upon verifying that the Resource Group already exists, we would delete the Resource Group and re-create it, deleting all those objects that exist on the Resource Group. That would be a pretty ohrrific side-effect......

Lastly,  you may notice that I use Write-Verbose “” –Verbose. There’s lots of options to write to the console or to a log file (Write-Host, Write-Debug, Write-Information, Write-Output being a few of them) but my preference has been to use Write-Verbose mainly because you can use it when you are returning something else from a function. In other words, the output of Write-Output will also be returned by a function, which is not ideal. And whilst I really like Write-Host, and build/deploy tools like Octopus/TeamCity support writing these tothe scrrne when they’re executing a PowerShell script, it’s not considered best practice. And I’m aware that this function does nto return anything per se, it’s a habit I’ve really gotten into.

Next post will actually focus on the Data Lake Store. Exciting stuff!


comments powered by Disqus