Pod Stuck in ContainerCreating? How Secrets Reach a Pod in AKS (Key Vault CSI Driver)

October 1, 2026 · How It Actually Works: Real Systems as State Machines (part 14)

▶ Watch on YouTube & subscribe to The Stack Underflow

Your Spring Boot app on AKS needs a database password. It must not live in Git or in the container image, yet it has to be there when the container starts. The video follows that one secret from Azure Key Vault into a running pod as a hierarchical state machine, including every place it gets stuck. This page is the written companion: the same path, the failure messages you’ll actually see, and what rotation really updates.

The one-line version: no secret, no start. The kubelet won’t start your containers until the CSI driver has fetched the secret, so a broken trust chain shows up as a pod stuck in ContainerCreating.

Last verified against Microsoft Learn (AKS, Key Vault, Entra ID), kubernetes.io and the Secrets Store CSI Driver docs: 1 October 2026. Scope: AKS with the Azure Key Vault provider for Secrets Store CSI Driver add-on, Microsoft Entra Workload ID, Linux nodes, Kubernetes 1.37 docs and a Spring Boot 4.1 app. Not covered: External Secrets Operator internals, Service Connector, node-level managed identity, Windows nodes.

The pegs

The video uses one picture, a bank vault and its couriers. These are memory aids (analogies), not facts.

Where the pegs break: Entra ID, not the vault, checks the badge; Key Vault then checks the role on the access token. And the drawer is open to anyone allowed to exec into the containers.

Chapter 1: one-time setup

Two independent things are set up once, side by side:

1. The vault. The password goes into Key Vault as a secret. The app’s managed identity gets the Key Vault Secrets User role on the vault (Microsoft: “Read secret contents including secret portion of a certificate with private key”). Microsoft recommends one vault per application per environment. Starting with Key Vault API version 2026-02-01, Azure RBAC is the default access model for new vaults; access policies are labelled legacy.
2. The trust chain. The cluster needs --enable-oidc-issuer and --enable-workload-identity, plus the azure-keyvault-secrets-provider add-on. You create a managed identity for the app, annotate the app’s service account with its client ID, and create a federated credential:

az identity federated-credential create --name aksfederatedidentity \
  --identity-name $UAMI --resource-group $RESOURCE_GROUP \
  --issuer ${AKS_OIDC_ISSUER} \
  --subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}

It takes a few seconds for a new federated credential to propagate, and a brand-new role assignment can take several minutes.

Chapter 2: deploy time, three ways to hand over a secret

The pod template gets the app’s service account, the label azure.workload.identity/use: "true" (Microsoft documents it as required in the pod template spec), and a CSI volume pointing at the class. In the class, an empty objectVersion means “latest”.

Chapter 3: pod startup, no secret no start

Before any container starts, the kubelet on the node prepares the pod’s volumes, secret data included. Kubernetes says Secrets are required by default, and none of a pod’s containers start until all non-optional Secrets are available; the kubelet retries and reports an event. For a CSI volume the kubelet calls the Secrets Store CSI driver on that node; for a plain Secret volume it fetches the Secret from the API server.

Chapter 4: the courier’s round

The driver runs on every node as a DaemonSet. Asked to publish the volume:

1. It looks up the SecretProviderClass, which must be in the pod’s own namespace.
2. It receives a service account token for the pod from the kubelet, and the Azure provider exchanges it with Microsoft Entra ID.
3. Entra ID checks the token’s signature with the cluster’s public signing keys, found through OIDC discovery, and looks for a federated credential matching the issuer, subject and audience.
4. Entra ID returns an access token for the managed identity (the vault’s day pass), and the provider asks Key Vault for each object.
5. Each object becomes a file in the pod’s tmpfs volume. Optionally, secretObjects mirrors the content into a Kubernetes Secret, to feed environment variables. That synced Secret exists only while a pod mounts the volume.

Where it fails, and what you’ll see

The pod phase stays Pending and kubectl get pods shows ContainerCreating. kubectl describe pod shows a FailedMount warning carrying the provider’s error:

One contrast worth remembering: if a missing plain Kubernetes Secret feeds an environment variable, the container shows CreateContainerConfigError, not ContainerCreating. Fix the cause, and the kubelet’s next retry succeeds.

Chapter 5: running

Three things now happen independently: the app reads its secret, the driver may check the vault again, and people may try to read Secrets.

How Spring Boot reads it.

• The mounted file, through spring.config.import=configtree:/mnt/secrets-store/: the file name becomes the property name, the content its value. Spring Boot’s config tree caches the value on first read.
• An environment variable from the synced Secret, via relaxed binding (SPRING_DATASOURCE_PASSWORD). Spring Boot’s docs warn that environment variables have drawbacks for secrets.
• Spring Cloud Azure’s Key Vault property source, which calls the vault from the app (using DefaultAzureCredential, which can use workload identity). Its property source refreshes every 30 minutes by default; the docs don’t promise that beans built at startup, like the DataSource, change.

What rotation really updates. On the add-on, autorotation is off by default (az aks show reports "enableSecretRotation": "false", "rotationPollInterval": "2m"). Turn it on with:

az aks addon update --resource-group myResourceGroup --name myAKSCluster \
  --addon azure-keyvault-secrets-provider --enable-secret-rotation

Then the provider polls for changes based on the rotation poll interval, two minutes by default, and updates the pod mount and the Kubernetes Secret defined in secretObjects. The courier restocks the drawer and the envelope, not the taped note: environment variables keep the old value, and so does Spring’s cached config tree. subPath mounts get no updates at all. Microsoft’s guidance is to restart the pod for environment variables, for example with a tool such as Reloader doing a rolling restart.

Who else can read it. Whoever may get, list or watch Secrets in the app’s namespace can read the synced Secret, and so can anyone allowed to create a pod there. Grant only the verbs a workload needs, and isolate secrets by namespace. On the vault side, with logging turned on, Key Vault records each SecretGet with the caller’s identity and IP address.

The options side by side

Pause & Prove

1. Autorotation is on. You store a new password version in Key Vault. Your app reads the password from an environment variable fed by the synced Kubernetes Secret. After the poll interval, does the app see the new value?

No. The driver updates the mounted files and the synced Secret, but an environment variable is set when the container starts and never refreshed. The app needs a restart, for example a rolling restart.

2. The Key Vault CSI mount fails for a new pod on AKS. What does kubectl get pods show?

• CrashLoopBackOff. Tempting, but Kubernetes describes it as a container that is “failing and restarting repeatedly”, with a backoff delay between restarts. Here no container has started yet.
• ContainerCreating. ✓ The pod stays Pending because no container starts until its volumes are ready; kubectl describe pod shows a FailedMount warning with the provider’s error.
• Running. Not until the mount succeeds.
• Evicted. Eviction terminates pods, for example for lack of resources or node maintenance. A failed mount isn’t an eviction; the kubelet keeps retrying the mount.

3. Peg drill

Key Vault · workload identity · CSI driver · Kubernetes Secret · environment variable. (The bank vault · the courier’s ID badge · the courier · an envelope in the mailroom, base64 is not a lock · a note taped to the desk.)

Before and after this video

• Before: Base64 Is Not Encryption: secrets and cache words, the primer for every term on this page.
• Next, same bookstore: Spring Boot Redis cache as a state machine.

Sources

Re-checked on 1 October 2026 unless noted:

• AKS: Key Vault provider for Secrets Store CSI Driver: add-on defaults (enableSecretRotation: false, rotationPollInterval: 2m), OIDC issuer and workload identity flags, subPath limitation, role names
• AKS: CSI driver configuration options: autorotation updates the mount and synced Secret, two-minute default, enable command, Reloader, sync lifetime
• AKS: Access Key Vault with the CSI driver identity provider: token exchange, Key Vault Secrets User, federated credential command, SecretProviderClass sample, pod label
• AKS: Workload ID overview: cluster as token issuer, OIDC signing keys, required pod label, propagation delay
• Troubleshoot the Key Vault Secrets Provider add-on: FailedMount messages, ForbiddenByConnection, context deadline exceeded, SecretProviderClass not found
• Azure Workload Identity troubleshooting: AADSTS70021 and the trailing slash
• Key Vault: Azure RBAC guide: role descriptions, RBAC default from API 2026-02-01, vault per app per environment, role assignment delay
• Azure built-in roles: Security: Key Vault Secrets User = “Read secret contents”, data actions getSecret and readMetadata only; About Key Vault certificates: each certificate has an addressable secret
• Spring Cloud Azure secret management: 30-minute default refresh
• Kubernetes: Secrets and Good practices for Secrets: unencrypted in etcd by default, base64, pod creators and list/watch read Secrets, Secrets required by default
• Kubernetes: Pod lifecycle: Pending phase, CrashLoopBackOff, eviction
• From the video’s source check (29 September 2026): the kubelet source (ContainerCreating, CreateContainerConfigError), Secrets Store CSI Driver concepts, Spring Boot external config and the v4.1.1 ConfigTreePropertySource source (config tree caching), Key Vault logging

Change notes

• 1 Oct 2026: first published. Note: the video says the Key Vault Secrets User role can “read secret contents, nothing more”. Azure’s built-in roles reference describes the role as “Read secret contents” and grants only two data actions: get a secret’s value and read a secret’s metadata. The Key Vault RBAC guide’s longer description, “Read secret contents including secret portion of a certificate with private key”, covers the same read access: every Key Vault certificate also gets an addressable secret, from which the certificate (with its private key, if exportable) can be retrieved. Either way the role is read-only.

Not affiliated with or endorsed by Microsoft, The Linux Foundation, or VMware Broadcom (Spring). Found a mistake? Tell us in the video’s comments and we’ll correct this page.