Azure Key Vault integration with AKS clusters

Using credentials in your applications is not a new thing. We use credentials to connect to databases, storage accounts, and other services. We use credentials to integrate with other systems like identity managers and so on. The proof of concept way of using credentials is by hard coding those credentials inside your code, which is a bad practice in production but for demo purposes or POC purposes, then it can be let go; however, when you're looking at going in production, then you have to keep those credentials away from your code.

There are multiple solutions available that grant you the possibility of keeping your credentials outside your code and almost all of them if not all of them offer you SDKs to retrieve them.

Azure Key Vault is one solution that provides you a centralized way of keeping your credentials safe with versioning and clear auditing. Azure also grants you possibilities to reference secrets inside your PaaS services like App Services for example.

At this point in time, Kubernetes is the defacto standard in container orchestration with a lot of solutions out there that can integrate very easily with it. We have the Helm package manager which allows us to easily install systems inside our clusters which is very versatile and allows us to create deployment templates for our applications.

Kubernetes even has a Secret manager which allows you to keep secrets and set them as environment variables inside your deployments but with the caveat that those secrets are easy retrieval, for example, if I set a secret with a value of secret inside Kubernetes then that secret only has a layer of obfuscation applied to it by making it base64.

Again, same idea, ease of use to get started as fast as possible and deploy your application.

The problem appears when we want to leverage multiple services in tandem with our cluster. For example, we don't want to keep our databases inside our cluster but leverage a SQL or MysQL database as a service. All is great until we need to change or rotate the connection string or keys to said services.

One solution would be to create a custom service that keeps our secrets in sync but it can become cumbersome in the long run. Another solution would be to leverage native Azure Key Vault provider or a third party provider like https://akv2k8s.io/ provider.

In this article we will be seeing how we can use the native Key Vault provider for AKS provided by Microsoft.

Azure Key Vault CSI Provider

In a nutshell, the KV CSI provider allows you to use an Azure Key Vault as a secret store provider in your AKS cluster by using a CSI volume.

The main and important features of the AKV solution are:

• Mounts secrets in a pod using a CSI volume
• Support key rotation
• Syncs secrets to Kubernetes Secrets

You can install the CSI provider in two ways. One being the recommended one where you update your AKS cluster from Azure by running this command in the Cloud shell

export K8S_RG="<rgname>"
export K8S_NAME="<clusname>"

az aks enable-addons --addons azure-keyvault-secrets-provider --name $K8S_NAME --resource-group $K8S_RG --enable-secret-rotation --rotation-poll-interval 5m

The command basically takes a AKS RG and AKS Name and then it enables the secret provider addon inside it with secret rotation and a 5 minute poll interval.

This means that once you start leveraging secrets from Key Vault then every 5 minutes it will check if the secret changed and rotate it where it can. In some cases it won't be able to rotate the secret inside your pod so keep that in mind.

After we install the addon, we can see two new pods create in the kube-system namespace and a new managed identity inside your MC_ Resource Group

That managed identity will be used to pull secrets from a Key Vault where you grant it a Get access policy.

Now we need to start configuring the SecretProviderClass to pull the credentials that we need.

In an Azure Key Vault we have three types of secrets:

• Keys,
• Secrets
• Certificates

With the Secret Store CSI driver, we can pull any one of them and leverage versions or aliases if we want to.
Below you will find an example YAML file that we will have to deploy in the namespace where the pods that will consume secrets reside.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: kvazuresync
spec:
  provider: azure
  secretObjects:
    - secretName: k8ssecrets
      type: Opaque
      labels:
        environment: 'production'
      data:
        - objectName: testsecret # name of the mounted content to sync. this could be the object name or object alias
          key: testSecret

  parameters:
    usePodIdentity: 'false'
    useVMManagedIdentity: 'true'
    userAssignedIdentityID: 'MSIclientID'

    keyvaultName: '<KVName>' # the name of the KeyVault
    objects: |
      array:
        - |
          objectName: testSecret
          objectType: secret
          objectAlias: ""
          objectVersion: ""

    tenantId: 'your AAD Tenant ID'

The GO object array in the form allows us to specify the object name, type, alias and version.

objects: |
  array:
    - |
      objectName: testSecret
      objectType: secret
      objectAlias: ""
      objectVersion: ""

The objectName maps directly to the KV object name, the type maps to one of the three types of credentials in Key Vault, the alias is for you to map another name and version maps to a specific KV version of the object which if left blank then it will take the latest version.

The excerpt from above will go to a key vault of our choice and pull the secret called testSecret and instantiate it in a Kubernetes secret named k8ssecrets with an object name.

This means that you will also get a file with the secret inside the pod and also have the possibility of injecting secrets as environment variables. You can use either or both.

After you modify and apply the YAML from above, you can now start consuming the secrets in your pods.

Modifying your YAML files to start using the KV secrets is pretty simple

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kvexample
  namespace: default
  labels:
    app: kvexample
spec:
  selector:
    matchLabels:
      app: kvexample
  template:
    metadata:
      labels:
        app: kvexample
    spec:
      containers:
        - image: mcr.microsoft.com/powershell:latest
          imagePullPolicy: Always
          name: kvexample
          command:
            - '/bin/bash'
            - '-c'
            - 'sleep infinity'
          resources:
            requests:
              memory: '64Mi'
              cpu: '50m'
            limits:
              memory: '512Mi'
              cpu: '250m'
          volumeMounts:
            - name: secrets-store-inline
              mountPath: '/mnt/secrets-store'
              readOnly: true
          env:
            - name: TESTSECRET
              valueFrom:
                secretKeyRef:
                  name: k8ssecrets
                  key: testsecret
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: 'kvazuresync'

Once your apply that deployment, you will have the TESTSECRET environment variable and the other secrets mounted in a volume in /mnt/secrets-store

Here's an example from a system that's running with this solution:

That's pretty much it. If you rotate a secret in the Key Vault, it will automatically rotate in the cluster.

That being said, playing around with the Secret Store provided is something easy to do and we have enough flexibility to do almost what ever we want.

That being said, have a good one!