Using credentials in your applications is not a new thing. We use credentials to connect to databases, storage accounts, and other services. We use credentials to integrate with other systems like identity managers and so on. The proof of concept way of using credentials is by hard coding those credentials inside your code, which is a bad practice in production but for demo purposes or POC purposes, then it can be let go; however, when you're looking at going in production, then you have to keep those credentials away from your code.
There are multiple solutions available that grant you the possibility of keeping your credentials outside your code and almost all of them if not all of them offer you SDKs to retrieve them.
Azure Key Vault is one solution that provides you a centralized way of keeping your credentials safe with versioning and clear auditing. Azure also grants you possibilities to reference secrets inside your PaaS services like App Services for example.
At this point in time, Kubernetes is the defacto standard in container orchestration with a lot of solutions out there that can integrate very easily with it. We have the Helm package manager which allows us to easily install systems inside our clusters which is very versatile and allows us to create deployment templates for our applications.
Kubernetes even has a Secret manager which allows you to keep secrets and set them as environment variables inside your deployments but with the caveat that those secrets are easy retrieval, for example, if I set a secret with a value of secret inside Kubernetes then that secret only has a layer of obfuscation applied to it by making it base64.
Again, same idea, ease of use to get started as fast as possible and deploy your application.
The problem appears when we want to leverage multiple services in tandem with our cluster. For example, we don't want to keep our databases inside our cluster but leverage a SQL or MysQL database as a service. All is great until we need to change or rotate the connection string or keys to said services.
One solution would be to create a custom service that keeps our secrets in sync but it can become cumbersome in the long run. Another solution would be to leverage native Azure Key Vault provider or a third party provider like https://akv2k8s.io/ provider.
In this article we will be seeing how we can use the native Key Vault provider for AKS provided by Microsoft.
Azure Key Vault CSI Provider
In a nutshell, the KV CSI provider allows you to use an Azure Key Vault as a secret store provider in your AKS cluster by using a CSI volume.
The main and important features of the AKV solution are:
- Mounts secrets in a pod using a CSI volume
- Support key rotation
- Syncs secrets to Kubernetes Secrets
You can install the CSI provider in two ways. One being the recommended one where you update your AKS cluster from Azure by running this command in the Cloud shell
export K8S_RG="<rgname>" export K8S_NAME="<clusname>" az aks enable-addons --addons azure-keyvault-secrets-provider --name $K8S_NAME --resource-group $K8S_RG --enable-secret-rotation --rotation-poll-interval 5m
The command basically takes a AKS RG and AKS Name and then it enables the secret provider addon inside it with secret rotation and a 5 minute poll interval.
This means that once you start leveraging secrets from Key Vault then every 5 minutes it will check if the secret changed and rotate it where it can. In some cases it won't be able to rotate the secret inside your pod so keep that in mind.
After we install the addon, we can see two new pods create in the kube-system namespace and a new managed identity inside your MC_ Resource Group
That managed identity will be used to pull secrets from a Key Vault where you grant it a Get access policy.
Now we need to start configuring the SecretProviderClass to pull the credentials that we need.
In an Azure Key Vault we have three types of secrets:
With the Secret Store CSI driver, we can pull any one of them and leverage versions or aliases if we want to.
Below you will find an example YAML file that we will have to deploy in the namespace where the pods that will consume secrets reside.
apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: kvazuresync spec: provider: azure secretObjects: - secretName: k8ssecrets type: Opaque labels: environment: 'production' data: - objectName: testsecret # name of the mounted content to sync. this could be the object name or object alias key: testSecret parameters: usePodIdentity: 'false' useVMManagedIdentity: 'true' userAssignedIdentityID: 'MSIclientID' keyvaultName: '<KVName>' # the name of the KeyVault objects: | array: - | objectName: testSecret objectType: secret objectAlias: "" objectVersion: "" tenantId: 'your AAD Tenant ID'
The GO object array in the form allows us to specify the object name, type, alias and version.
objects: | array: - | objectName: testSecret objectType: secret objectAlias: "" objectVersion: ""
The objectName maps directly to the KV object name, the type maps to one of the three types of credentials in Key Vault, the alias is for you to map another name and version maps to a specific KV version of the object which if left blank then it will take the latest version.
The excerpt from above will go to a key vault of our choice and pull the secret called testSecret and instantiate it in a Kubernetes secret named k8ssecrets with an object name.
This means that you will also get a file with the secret inside the pod and also have the possibility of injecting secrets as environment variables. You can use either or both.
After you modify and apply the YAML from above, you can now start consuming the secrets in your pods.
Modifying your YAML files to start using the KV secrets is pretty simple
Once your apply that deployment, you will have the TESTSECRET environment variable and the other secrets mounted in a volume in /mnt/secrets-store
Here's an example from a system that's running with this solution:
That's pretty much it. If you rotate a secret in the Key Vault, it will automatically rotate in the cluster.
That being said, playing around with the Secret Store provided is something easy to do and we have enough flexibility to do almost what ever we want.
That being said, have a good one!