Azure KeyVault with Flexvol on AKS
If you’re using Azure Kubernetes Service (AKS) then you have probably heard of https://github.com/Azure/aad-pod-identity. This is an Azure product that allows pods running on a Kubernetes cluster to assume an AD Service Principal or Managed Identity to access other Azure resources, such as Cosmos DB or Key Vault.
In this post we’ll be looking at the Key Vault (KV) use case and combining it with another Azure Kubernetes offering, https://github.com/Azure/kubernetes-keyvault-flexvol, to present a secret value from Key Vault to an arbitrary application pod running on an AKS cluster as a file mount. The key win here, obviously, is that the application containers themselves can completely outsource secrets management and Azure integration to AKS. Another is that, although there’s a bit of work to set this up, once it’s there using it is a cinch.
There’s four main parts to the story:
- Installing the aad-pod-identity and kubernetes-keyvault-flexvol components on your cluster.
- Configuring your Managed Identity with access to the KV.
- Adding a couple of bespoke Kubernetes resources (AzureIdentity and AzureIdentityBinding), which are implemented as CRDs.
- Adding the Flexvol integration to your target pods.
The great news is that all this can be accomplished using standard Kubernetes techniques, like Helm charts and resource metadata, that you are already familiar with and can plug in to your existing config management and deployment pipelines. So let’s get stuck in:
1. Install aad-pod-identity and kubernetes-keyvault-flexvol:
If you’re using Helm:
aad-pod-identity has a well-featured official chart: https://github.com/Azure/aad-pod-identity/tree/master/charts/aad-pod-identity.
It installs some CRDs and two runtime components: the mic controller deployment and the nme daemonset, which proxies Azure API calls from pods and impersonates the designated identity on the pod’s behalf- rather like kube2iam does on AWS.
At the time of writing kubernetes-keyvault-flexvol doesn’t appear to have an official Helm chart, but until one’s available it’s simple enough to roll your own from:
https://github.com/Azure/kubernetes-keyvault-flexvol/tree/master/deployment
If you’re using addon manager or similar to manage your deployments:
https://github.com/Azure/aad-pod-identity/tree/master/deploy/infra
https://github.com/Azure/kubernetes-keyvault-flexvol/tree/master/deployment
There are choices to be made here as the integration can work in a number of ways, as described here: https://github.com/Azure/kubernetes-keyvault-flexvol
You can use a single identity for the cluster, including the built-in VMSS identity, or a different Managed Identity in each case. Broadly speaking, in a single-tenant, dedicated cluster use case you might choose the former but for a multi-tenant cluster where you want more fine-grained control over your identity and clearer separation for it from the cluster life-cycle then a different identity for each client application is the way to go. That is certainly our use case, so we’ll be adding a Managed Identity and following this documentation: https://github.com/Azure/kubernetes-keyvault-flexvol#option-2-pod-identity
Note that both the flexvoland nmedaemonsets require elevated privileges and should be placed in their own namespaces accordingly.
My aad-pod-identity chart release templates like this:
And my keyvault-flexvolume template comes out like this:
2. Configuring the Managed Identity
This creates and sets up the Managed Identity in Azure.
We’re using Azure Pipelines to manage this two-step process, but essentially it does this:
For the next step make a note of ID_CLIENTID
3. Create the AzureIdentity and AzureIdentityBinding objects
These are the aad-pod-identity CRD resources in your AKS cluster. Again, we’re chaining this together in an Azure Pipeline and have wrapped the resources into a Helm chart, but the resulting manifests look something like this:
4. Add additional config to target pods
There’s two things to add:
- A label to plug in the AzureIdentityBinding:
aadpodidbinding - The flexvol volume mount.
Here’s an example:
Note how the value of KEY_FILE is a combination of the volumeMount mountPath and the flexVolume keyvaultobjectnames
If you tail the logs of the aad-pod-identity components you should see a bit of activity, including useful errors if there’s any issue, when the AzureIdentity and binding is created (mic) and when the pods start on the relevant node (nme).
That’s pretty much it.
However, the more circumspect reader may be thinking “So, all a pod needs is the aadpodidbinding label with the right value and it gets that AzureIdentity?” That struck me as well and the answer is “Yes”, so consider mitigation carefully before diving in. The primary mitigation here is logical namespace separation of your tenants and application components, which should always be grouped on a least-privilege basis; see the aad-pod-identity chart values optionforceNameSpaced.
A secondary mitigation is ensuring least-privilege for the Managed Identity and logical separation of your Key Vaults to minimise the blast-radius of any compromise.
These are both good-practice in their own right.
A more general question, though, is “So, am I OK with people adding whatever labels they want to deployments on my cluster without me even knowing about it?” If they can do that then what else can they do? We’re using OPA to provide governance around questions of this kind and adding a rule to forbid the label aadpodidbinding, except on pods where we allow it, is one of the things we’re including in our roll-out of this feature. Indeed, that might be the subject of the next blog if anyone’s interested.
