Tokenizing your configuration files

I’ve been constantly working on a project where had to deploy a full blown application in Azure for Dev/Test using VSTS, PowerShell, DSC, ARM templates and the kitchen sink. The main idea was to deploy the application on the Azure VMs and treat them as cattle (as Jeffrey Snover would put it).

The whole point of this was to create a release workflow that would not require any human intervention to make the application work after it was deployed.

We went with the tactic of destroy and deploy which meant that an existing environment would be wiped and a new one would take its place. Hard-coding values in the applications configuration files, values like IPs, computer names or anything for that matter was not possible because the freshly deployed environment contained VMs that had all their “identifiers” randomized. Our way of identifying the VMs was to set Resource Tags using the ARM deployment template and VMs that would serve a role was tagged as such. So VMs that served the role of a web server would be tagged as web servers, app server role tagged as app servers and so on.

What we needed to do was to make an application that was configured for one very specific environment with a static configuration and make it work on in an environment where everything that mattered was dynamically provisioned.
We solved the problem by tokenizing all the configuration files in the build process and replace the tokens after the environment was provisioned.

So what’s a token or tokenizing a .config file?

Well in a nutshell a token is a string of text inside the configuration file that would be later identified by a RegEx matching pattern and have it replaced with an actual value. Usually these tokens are in the form of __REPLACEABLE_VALUE__ but they can be in any form as long as the prefix and suffix patterns __ (double underscore before and after the token) are configured in the tokenizing task.

Here’s an example of a tokenized web configuration file:

<?xml version="1.0"?>
<connectionStrings>
  <clear />
  <add name="LocalSqlServer" connectionString="data source=__DATA_SOURCE__;Initial Catalog=__INITIAL_CATALOG__;User ID=__USER_ID__;Password=__PASSWORD__" providerName="System.Data.SqlClient" />
</connectionStrings>

As you can see above, the web.config file is completely useless because there are no real values configured and we have tokens for data source, initial catalog, user and password.

So how do we replace those tokens with real data so that everything can work?

Well we have two options for doing this job; DSC Resource xTokenize or a VSTS extension.

The first iteration was to use PowerShell DSC and the DSC Resource xReleaseManagement to deploy the application and modify the tokens inside the VM. The way we handled this job was to create DSC configuration data files (.psd1) containing the relevant information from the freshly provisioned VMs and then deploy the application component along with the PowerShell DSC configuration and data file.

Here’s a sanitized snippet from the DSC configuration document along with the config data:

Configuration ...{

     Import-DscResource -ModuleName PSDesiredStateConfiguration, xWebAdministration, xReleaseManagement

    Node localhost {
            xTokenize ModifyConnectionString
            {
                path = "Path/To/ConfigFile/"
                recurse =                 $false
                tokens = @{DATA_SOURCE="$($Node.ConnectionDB),1433";INITIAL_CATALOG="$($Node.ConnectionCatalog)";USER_ID="$($Node.ConnectionUser)";PASSWORD="$($Node.ConnectionPass)"}
            }                   
    }
}
@{
    AllNodes = @(
        @{
            NodeName           = 'localhost'
            ConnectionDB       = 'SQLSERVER'
            ConnectionCatalog  = 'SQLDATABASE'
            ConnectionUser     = 'USER'
            ConnectionPass     = 'PASSWORD'
        }
    )
}

So what we’re doing here? We’re using the xReleaseManagement DSC module that contains a resource named xTokenize. Using that resource we can specify the path to the .config file and replace the tokens (__TOKEN__”) with real values. In the case above the values were generated in the Visual Studio Team Services release definition and we pushed the configuration document using the Azure DSC Extension.

Once the transformation of the Token Files was completed, the web application was spun up.

What we didn’t like after some time was that the transformation of the configuration files took place inside the VMs and we wanted that task to be done in the VSTS release definition. So I searched for a replacement for the Tokenizing resource and found this very nice VSTS extension made by Guillaum Ruchon VSTS Tokenize Extension which basically you install it in VSTS and add Tokenizing tasks.

The extension is pretty straightforward and no explanation is quite needed ?

The problem we were facing is that we had a solution for replacing tokens with actual values but the .config files were not tokenized and due to the unpredictable nature of the DEV team we couldn’t create the files on the fly. Our solution was to create a set of PowerShell scripts that would use regular expressions to match certain parts of the config files and replace them with tokens.

Here’s the first PowerShell scripts that I created for this job and used it to modify the web.config files and adapted it to target app.config files as well. I had to modify it for the application services configuration files because the structure was quite strange and unpredictable but the main idea remained the same.

function Set-ConnectionString
{
    param(
        [CmdletBinding()]
        [Parameter(Mandatory = $true, 
                   ValueFromPipeline = $true)]
                   [ValidateNotNullOrEmpty()]
        [String]
        $ConnectionFile
    ) 
   
    Write-Verbose -Message "Tokenizing '$ConnectionFile' File"
    $ConfigString =   [xml](Get-Content "$ConnectionFile")
    $RootDocument = $ConfigString.DocumentElement
    $ConnectionString = $RootDocument.add.connectionString `
            -replace '(?m)(?<=\bData Source=).*?[^;]+', '__DATA_SOURCE__' `
            -replace '(?m)(?<=\bInitial Catalog=).*?[^;]+', '__INITIAL_CATALOG__' `
            -replace '(?m)(?<=\bID=).*?[^;]+', '__USER_ID__' `
    -replace '(?m)(?<=\bPassword=).*','__PASSWORD__' 
    
    $RootDocument.add.connectionString = $ConnectionString
    $ConfigString.Save($ConnectionFile)
}

Here’s what’s going on in the script above; We’re populating a variable with the path to the .config file and then the script is loading the file in a variable and doing replace operations based on the (?m)(?<=\btargetedfield=).*?[^;]+ regular expression and then saving the file.

The regular expression looks for a value inside the file which in our case is Data Source= , Initial Catalog= , ID= and Password= and it replaces the value after the equal sign with a token. This may not be the perfect solution and I’m not a RegEx expert but it works fine in our case.

By doing all these operations inside the Visual Studio Team Services account, we created a one click button that can deploy the application to multiple environments while we’re drinking our coffee, eating a bagel and watching our favorite TV Series ?

That’s it for today, have a good one!