Roman KlimenkoBlogPhotography

Sync certificates and secrets from Azure Key Vault to Azure Kubetnetes Service

November 01, 2021


In this walkthrough, we will create a new Azure Key Vault, and then create a new Azure Kubernetes Service, and then we will synchronize the certificates and secrets from the Azure Key Vault to the Azure Kubernetes Service.

Useful links:

We will use Powershell 7 and assume that all commands run in the same session. So we can start with defining the necessary variables:

$SUBSCRIPTION_ID = '...' $LOCATION = '...' $RG_NAME = '...' $AKS_NAME = '...' $AKV_NAME = '...' # must be globally unique

Ensure that we run the commands under the right Subscription:

az login az account set --subscription $SUBSCRIPTION_ID

Enable the Secrets Store CSI Driver feature:

az feature register --namespace "Microsoft.ContainerService" --name "AKS-AzureKeyVaultSecretsProvider"

It will take a while for the feature to be enabled. We can check the status of the feature by running this command:

az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/AKS-AzureKeyVaultSecretsProvider')].{Name:name,State:properties.state}" # Eventually, it must return "Registered": # Name State # ----------------------------------------------------------- ---------- # Microsoft.ContainerService/AKS-AzureKeyVaultSecretsProvider Registered

Now, re-register the Container Service extension and ensure it is up-to-date:

az provider register --namespace Microsoft.ContainerService az extension add --name aks-preview az extension update --name aks-preview

Create a resource group:

az group create --name $RG_NAME --location $LOCATION

Create an Azure Key Vault with one secret and one certificate:

az keyvault create --name $AKV_NAME --resource-group $RG_NAME --location $LOCATION az keyvault certificate get-default-policy > policy.json # get the default policy az keyvault certificate create --name cert-demo --vault-name $AKV_NAME -p "@policy.json" az keyvault secret set --vault-name $AKV_NAME --name "foo" --value "bar"

Next, let's create an Azure Kubernetes Service:

az aks create ` --resource-group $RG_NAME ` --name $AKS_NAME ` --node-vm-size Standard_B8ms ` --node-count 1 ` # AKS creates 3 nodes by default, but for the demo we need only one --generate-ssh-keys ` --network-plugin azure ` --enable-addons azure-keyvault-secrets-provider ` # enable the Secrets Store CSI Driver --enable-managed-identity ; # Expected output: # { # "aadProfile": null, # "addonProfiles": { # "azureKeyvaultSecretsProvider": { # "config": { # "enableSecretRotation": "false", # "rotationPollInterval": "2m" # }, # "enabled": true, # "identity": { # "clientId": "...", # "objectId": "...", # "resourceId": "/subscriptions/.../resourcegroups/MC_resourse-group-name_region/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurekeyvaultsecretsprovider-aks-name" # } # } # },

Pay attention to the addonProfiles.identity - a managed identity automatically created in the MC_ resource group. We will use this identity to connect to the Azure Key Vault.

Let's save the addonProfiles.identity.cliendId into a variable:

$SERVICE_PRINCIPAL_CLIENT_ID = 'a819baaa-4aeb-43fc-92ce-b367176d5b88'

If you update the existing AKS cluster, you will need to run this command in this way:

az aks enable-addons --addons azure-keyvault-secrets-provider --name $AKS_NAME --resource-group $RG_NAME

While we are here, let's connect to the AKS cluster and enable the secrets auto-rotation:

az aks get-credentials --resource-group $RG_NAME --name $AKS_NAME # check the CSI driver and the store provider statuses kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver, secrets-store-provider-azure)' # Expected output: # NAME READY STATUS RESTARTS AGE # aks-secrets-store-csi-driver-h52sr 3/3 Running 0 0h17m # aks-secrets-store-provider-azure-7qlgd 1/1 Running 0 0h30m az aks update -g $RG_NAME -n $AKS_NAME --enable-secret-rotation

Now, let's allow our managed identity to access the Azure Key Vault:

az keyvault set-policy -n $AKV_NAME --secret-permissions get --spn $SERVICE_PRINCIPAL_CLIENT_ID az keyvault set-policy -n $AKV_NAME --certificate-permissions get --spn $SERVICE_PRINCIPAL_CLIENT_ID

These commands let the managed identity read secrets and certificates from the Azure Key Vault.

Our next step is to create a SecretProviderClass - a custom Kubernetes resource that will be used to connect to the Azure Key Vault:

# secretproviderclass.yml apiVersion: kind: SecretProviderClass metadata: name: azure-keyvault-name # use the name of your Azure Key Vault spec: provider: azure secretObjects: # The following section describes how AKV secret is mapped to the Kubernetes secret: - secretName: foo type: Opaque data: - objectName: foo key: foo # If we store a certificate as a Kubernetes secret, the secret type must be - secretName: cert-demo type: "" data: - objectName: cert-demo key: tls.key - objectName: cert-demo key: tls.crt parameters: keyvaultName: "azure-keyvault-name" # The name of the Azure Key Vault useVMManagedIdentity: "true" userAssignedIdentityID: "..." # The clientId of the addon-created managed identity # this section describes the objects pulled from Azure Key Vault objects: | array: - | objectName: foo objectType: secret - | objectName: cert-demo objectType: secret # the tenant ID containing the Azure Key Vault instance, you can find it in Azure Portal tenantId: "..."

Apply the SecretProviderClass:

kubectl apply -f ./secretproviderclass.yml

Finally, let's test it:

Create a test-pod.yml with the following content:

kind: Pod apiVersion: v1 metadata: name: busybox-secrets-store-inline spec: containers: - name: busybox image: command: - "/bin/sleep" - "10000" volumeMounts: - name: secrets-store-inline mountPath: "/mnt/secrets-store" readOnly: true volumes: - name: secrets-store-inline csi: driver: readOnly: true volumeAttributes: secretProviderClass: "azure-key-vault-name" # the name of your key vault kubectl apply -f ./test-pod.yml kubectl exec busybox-secrets-store-inline -- ls /mnt/secrets-store/ # Expected output: # cert-demo # foo kubectl exec busybox-secrets-store-inline -- cat /mnt/secrets-store/foo # Expected output: # bar kubectl exec busybox-secrets-store-inline -- cat /mnt/secrets-store/cert-demo # Expected output: -----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDVYhtyud6rbRJT ... 3fic6VM3cQR9FJxBxAq4vro= -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- MIIDQjCCAiqgAwIBAgIQSRZYP7ncTSGCw6IEOxTIhjANBgkqhkiG9w0BAQsFADAe ... 5STNJyO/kEBkBMjlzZKlDkhuf4Tr1g== -----END CERTIFICATE----- kubectl get secrets # Expected output: # NAME TYPE DATA AGE # cert-demo 2 9h # foo Opaque 1 9h