Roman Klimenko
DA

Sync from Azure Key Vault to Azure Kubernetes Service

azuredevops

In this walkthrough, we will create an Azure Key Vault and an Azure Kubernetes Service cluster, then synchronize certificates and secrets from the vault to the cluster.

Useful links:

We will use PowerShell 7 and run all commands in the same session. Start by defining the necessary variables:

$SUBSCRIPTION_ID = '...'
$LOCATION = '...'
$RG_NAME = '...'
$AKS_NAME = '...'
$AKV_NAME = '...' # must be globally unique

Before creating any resources, make sure you are using the right subscription:

az login
az account set --subscription $SUBSCRIPTION_ID

Enable the Secrets Store CSI Driver feature:

az feature register --namespace "Microsoft.ContainerService" --name "AKS-AzureKeyVaultSecretsProvider"

Enabling the feature takes a while. Check its status with this command:

az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/AKS-AzureKeyVaultSecretsProvider')].{Name:name,State:properties.state}"

# Eventually, it must return "Registered":

# Name                                                         State
# -----------------------------------------------------------  ----------
# Microsoft.ContainerService/AKS-AzureKeyVaultSecretsProvider  Registered

Re-register the Microsoft.ContainerService provider, then install or update the aks-preview extension:

az provider register --namespace Microsoft.ContainerService
az extension add --name aks-preview
az extension update --name aks-preview

Create a resource group:

az group create --name $RG_NAME --location $LOCATION

Create an Azure Key Vault with one secret and one certificate:


az keyvault create --name $AKV_NAME --resource-group $RG_NAME --location $LOCATION

az keyvault certificate get-default-policy > policy.json # get the default policy

az keyvault certificate create --name cert-demo --vault-name $AKV_NAME -p "@policy.json"
az keyvault secret set --vault-name $AKV_NAME --name "foo" --value "bar"

Next, create an Azure Kubernetes Service cluster:

az aks create `
  --resource-group $RG_NAME `
  --name $AKS_NAME `
  --node-vm-size Standard_B8ms `
  --node-count 1 ` # AKS creates 3 nodes by default, but for the demo we need only one
  --generate-ssh-keys `
  --network-plugin azure `
  --enable-addons azure-keyvault-secrets-provider ` # enable the Secrets Store CSI Driver
  --enable-managed-identity ;

  # Expected output:

  # {
  #   "aadProfile": null,
  #   "addonProfiles": {
  #     "azureKeyvaultSecretsProvider": {
  #       "config": {
  #         "enableSecretRotation": "false",
  #         "rotationPollInterval": "2m"
  #       },
  #       "enabled": true,
  #       "identity": {
  #         "clientId": "...",
  #         "objectId": "...",
  #         "resourceId": "/subscriptions/.../resourcegroups/MC_resourse-group-name_region/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurekeyvaultsecretsprovider-aks-name"
  #       }
  #     }
  #   },

Notice addonProfiles.azureKeyvaultSecretsProvider.identity, a managed identity that AKS creates automatically in the MC_ resource group. We will use this identity to connect to the Azure Key Vault.

Save the addonProfiles.azureKeyvaultSecretsProvider.identity.clientId in a variable:

$SERVICE_PRINCIPAL_CLIENT_ID = 'a819baaa-4aeb-43fc-92ce-b367176d5b88'

To enable the add-on in an existing AKS cluster, run:

az aks enable-addons --addons azure-keyvault-secrets-provider --name $AKS_NAME --resource-group $RG_NAME

Next, connect to the AKS cluster and enable automatic secret rotation:

az aks get-credentials --resource-group $RG_NAME --name $AKS_NAME
# check the CSI driver and the store provider statuses
kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver, secrets-store-provider-azure)'

# Expected output:
# NAME                                     READY   STATUS    RESTARTS   AGE
# aks-secrets-store-csi-driver-h52sr       3/3     Running   0          0h17m
# aks-secrets-store-provider-azure-7qlgd   1/1     Running   0          0h30m

az aks update -g $RG_NAME -n $AKS_NAME --enable-secret-rotation

Allow the managed identity to access the Azure Key Vault:

az keyvault set-policy -n $AKV_NAME --secret-permissions get --spn $SERVICE_PRINCIPAL_CLIENT_ID
az keyvault set-policy -n $AKV_NAME --certificate-permissions get --spn $SERVICE_PRINCIPAL_CLIENT_ID

These commands let the managed identity read secrets and certificates from the Azure Key Vault.

Now create a SecretProviderClass, the custom Kubernetes resource that connects to the Azure Key Vault:

# secretproviderclass.yml
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-keyvault-name # use the name of your Azure Key Vault
spec:
  provider: azure
  secretObjects:
  # The following section describes how AKV secret is mapped to the Kubernetes secret:
  - secretName: foo
    type: Opaque
    data:
    - objectName: foo
      key: foo
  # If we store a certificate as a Kubernetes secret, the secret type must be kubernetes.io/tls
  - secretName: cert-demo
    type: "kubernetes.io/tls"
    data:
    - objectName: cert-demo
      key: tls.key
    - objectName: cert-demo
      key: tls.crt
  parameters:
    keyvaultName: "azure-keyvault-name" # The name of the Azure Key Vault
    useVMManagedIdentity: "true"         
    userAssignedIdentityID: "..." # The clientId of the addon-created managed identity
    # this section describes the objects pulled from Azure Key Vault
    objects:  |
      array:
        - |
          objectName: foo
          objectType: secret
        - |
          objectName: cert-demo
          objectType: secret
    # the tenant ID containing the Azure Key Vault instance, you can find it in Azure Portal
    tenantId: "..."

Apply the SecretProviderClass:

kubectl apply -f ./secretproviderclass.yml

Finally, create a test-pod.yml file with the following content to test the setup:

kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline
spec:
  containers:
  - name: busybox
    image: k8s.gcr.io/e2e-test-images/busybox:1.29
    command:
      - "/bin/sleep"
      - "10000"
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-key-vault-name" # the name of your key vault

kubectl apply -f ./test-pod.yml

kubectl exec busybox-secrets-store-inline -- ls /mnt/secrets-store/
# Expected output:
# cert-demo
# foo

kubectl exec busybox-secrets-store-inline -- cat /mnt/secrets-store/foo
# Expected output:
# bar

kubectl exec busybox-secrets-store-inline -- cat /mnt/secrets-store/cert-demo
# Expected output:
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDVYhtyud6rbRJT
...
3fic6VM3cQR9FJxBxAq4vro=
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIDQjCCAiqgAwIBAgIQSRZYP7ncTSGCw6IEOxTIhjANBgkqhkiG9w0BAQsFADAe
...
5STNJyO/kEBkBMjlzZKlDkhuf4Tr1g==
-----END CERTIFICATE-----

kubectl get secrets
# Expected output:
# NAME                                    TYPE                                  DATA   AGE
# cert-demo                               kubernetes.io/tls                     2      9h
# foo                                     Opaque                                1      9h