Cosmos DB as a config store for automations

My daily activity is figuring out new things that need to be eliminated from the clickfest so we don't have to do manual work, which is, in a nutshell, toil reduction. That's the gist of it. An end-to-end flow that builds, tests, and deploys code simplifies this process. However, some things shouldn't be "hard coded," and flexibility should be a big option.

Imagine this: I deploy code that targets two subscriptions, and the business asks for a third subscription. Based on this new requirement, I need to pull a request to add that new subscription and wait 15 minutes for it to be deployed.

However, a crucial element often overlooked is flexibility. Hardcoding configurations directly into deployments can quickly become a bottleneck. Imagine deploying code targeting two subscriptions, only to have the business request a third. Updating a hardcoded deployment requires a pull request, followed by a 15-minute wait.

This approach functions, but it needs more agility. What if you need to:

• Change a parameter on the fly?
• Temporarily turn off a specific functionality?
• Pause automation for a particular scope?

You get the point. Usually, the problem is solved by implementing configuration files for different purposes, but developers have been using feature flags with hot turn on/off for a long time. How do we, as infrastructure folks get some of that feature flag business in our code?

The solution is quite simple, as I found it and have been testing it for over two years now: Use CosmosDB as a config store. Is this the true purpose of CosmosDB? No, but it works perfectly for these use cases, or so I've seen so far.

Here's the deal: whenever you want to build something and have no idea if this should be extended to other resource groups or subscriptions, then plan for the future and go from there.

First before we start doing anything, install the CosmosDB module referenced below as it's gold for working with CosmosDB.

GitHub - PlagueHO/CosmosDB: PowerShell Module for working with Azure Cosmos DB databases, collections, documents, attachments, offers, users, permissions, triggers, stored procedures and user defined functions.

PowerShell Module for working with Azure Cosmos DB databases, collections, documents, attachments, offers, users, permissions, triggers, stored procedures and user defined functions. - PlagueHO/Cos…

GitHubPlagueHO

The process is simple: you write a JSON document in a collection, and you pull that configuration on each run.

Here's a function that you can use for connecting to the Cosmos database:

function Connect-CosmosDBContext {
  [CmdletBinding()]
  param (
    [Parameter(Mandatory = $true)]
    [string]
    $PrimaryKey,

    [Parameter(Mandatory = $true)]
    [string]
    $EncryptionKey,

    [Parameter(Mandatory = $true)]
    [string]
    $AccountName,

    [Parameter(Mandatory = $true)]
    [string]
    $DatabaseName
  )

  $accountKey = ConvertTo-SecureString -String $PrimaryKey -Key $($EncryptionKey).Split()
  $backoffPolicy = New-CosmosDbBackoffPolicy -MaxRetries 3 -Method Random -Delay 1000
  $cosmosDbContext = New-CosmosDbContext -Account $AccountName -Database $DatabaseName -Key $accountKey -BackoffPolicy $backoffPolicy
  return $cosmosDbContext
}

This is how it can be used:

$cosmosDbContext = Connect-CosmosDBContext -PrimaryKey $env:COSMOS_DB_KEY -EncryptionKey $env:CDBKEY -AccountName $env:COSMOS_DB_ACCOUNT_NAME -DatabaseName $env:COSMOS_DB_DATABASE_NAME

$configuration = Get-CosmosDbDocument -Context $cosmosDbContext -CollectionId 'configuration' -PartitionKey 'systemYXZ-configuration'

Function Usage:

The provided example demonstrates how to utilize the Connect-CosmosDBContext function. It assumes the primary key, encryption key, account name, and database name are stored as environment variables:

• $env:COSMOS_DB_KEY: Holds the primary key value.
• $env:CDBKEY: Holds the encryption key value.
• $env:COSMOS_DB_ACCOUNT_NAME: Holds the Cosmos DB account name.
• $env:COSMOS_DB_DATABASE_NAME: Holds the name of the database within your Cosmos DB account.

1. The function is called with the environment variable names passed as arguments.
2. This establishes a connection to your Cosmos DB account and creates the $cosmosDbContext object.
3. Subsequently, Get-CosmosDbDocument is used to retrieve a specific document from the "configuration" collection identified by the partition key "systemYXZ-configuration". This assumes you have a collection named "configuration" with documents identifiable by the provided partition key.

Now that we have the connection part, the flow looks something like this:

If you get the point from the scribble, that main idea is that the job or function pulls data from the Cosmos DB collection and then does what it does.

The configuration from the collection would be instantiated as a Cosmos DB object that looks and queries similarly like a hash table.

Now depending on the JSON schema you created, you can iterate between whatever with zero code changes.

Example of code use:

 Get the first subscription
$firstSubscription = $configuration.subscriptions[0]

# Access specific properties from the first subscription
$subscriptionName = $firstSubscription.subscriptionName
$earlyNotification = $firstSubscription.notification[0].earlyNotification

# Loop through all subscriptions and access properties
foreach ($subscription in $configuration.subscriptions) {
  Write-Host "Subscription Name: $($subscription.subscriptionName)"
  Write-Host "Early Notification: $($subscription.notification[0].earlyNotification)"
  Write-Host "" # Add an empty line for better readability
}

# Check if "shouldRemind" is true for any subscription
$hasReminder = $configuration.subscriptions | Where-Object { $_.shouldRemind -eq "true" }

if ($hasReminder) {
  Write-Host "There are subscriptions configured with reminders."
} else {
  Write-Host "No subscriptions are configured with reminders."
}

Picture this: you have rock-solid code with great tests written in Pester and a thorough process that needs to pass for a change to happen. It's 2 a.m., and a flag needs to be set to false because something happened on the Azure side. How bad will it be if the change takes 30 minutes?
I've been in such a situation, and if the change had taken 30 minutes, it would have been extremely bad. But having the feature flag in Cosmos and the function refreshed every time saved a lot of data.

Anyway, this is one thing I've been using Cosmos DB; the PowerShell Module referenced above is excellent for these operations. I might have built a jankier version if I had not found it.

I would go into more detail about this, but it's just how I found a significant usage of Cosmos DB besides other things I've been using it for.

Anyway, I hope you found this useful, and as always, have a good one!