Pods cannot reach cloud APIs after enabling Kubernetes workload identity

TL;DR — After enabling workload identity, the most common breakage is that your pod is no longer using the old node-level credentials, but it also is not getting a valid projected service account token or the Kubernetes service account is not bound to the cloud identity. Start by checking whether the pod has a projected token file and whether the cloud-side trust/binding matches the pod's service account namespace/name. Reading time: ~6 min

The scenario

You roll out workload identity on a Tuesday afternoon because static cloud keys in secrets finally need to die. The deploy itself looks fine, pods are Running, readiness checks pass, and then the app starts throwing 401s and timeouts when it tries to hit object storage, a secrets API, or a cloud database auth endpoint. Nothing changed in application code, but every request that depends on cloud auth is now failing. You need to know whether this is identity plumbing, SDK behavior, or plain old egress.

Symptoms

• Application logs start showing auth failures immediately after the rollout:

403 Forbidden
401 Unauthorized
AccessDenied
InvalidIdentityToken
invalid_grant
permission denied

• SDK-specific errors such as:

NoCredentialProviders: no valid providers in chain
DefaultCredentialsError: failed to load credentials
failed to refresh token
error exchanging Kubernetes service account token for cloud access token

• Pod logs mention missing token files or unreadable projected volumes:

open /var/run/secrets/tokens/oidc-token: no such file or directory
permission denied reading token file

• Direct HTTP calls to the provider token endpoint fail from inside the pod:

curl: (6) Could not resolve host: sts.examplecloud.com
curl: (7) Failed to connect to sts.examplecloud.com port 443
HTTP/1.1 403 Forbidden
HTTP/2 401

• The app works if you exec into the pod and export a long-lived access key manually, but fails with ambient credentials.
• Existing pods created before the identity change may still work; newly created pods fail.

Likely causes

| Kubernetes service account is not bound/mapped to the cloud identity | Very common | ```bash
kubectl get sa -n <ns> <sa> -o yaml
``` |
| Token issuer or audience does not match what the cloud trust expects | Common | ```bash
kubectl exec -n <ns> <pod> -- sh -c 'TOKEN=$(cat /var/run/secrets/tokens/* 2>/dev/null | head -c 4000); python3 - <<"PY"
import os,base64,json
t=os.popen("cat /var/run/secrets/tokens/* 2>/dev/null").read().strip().split(".")
print(json.dumps(json.loads(base64.urlsafe_b64decode(t[1]+"==")),indent=2)) if len(t)>=2 else print("no token")
PY'
``` |
| NetworkPolicy, proxy, firewall, or DNS blocks egress to the cloud token/API endpoints | Common | ```bash
kubectl exec -n <ns> <pod> -- sh -c 'getent hosts <token-endpoint-host> && curl -sv https://<token-endpoint-host>/ 2>&1 | sed -n "1,20p"'
``` |
| SDK is still preferring old env vars or metadata-based credentials and never attempts workload identity | Common | ```bash
kubectl exec -n <ns> <pod> -- sh -c 'env | grep -E "AWS_|AZURE_|GOOGLE_|CLOUD_|*_CREDENTIALS|*_TOKEN" | sort'
``` |
| Pod spec disables service account token automount or uses the wrong service account | Less common | ```bash
kubectl get pod -n <ns> <pod> -o jsonpath='{.spec.serviceAccountName}{"\n"}{.spec.automountServiceAccountToken}{"\n"}'
``` |

## Step-by-step diagnosis
1. Check which service account the failing pod is actually using.

```bash
kubectl get pod -n <ns> <pod> -o jsonpath='{.spec.serviceAccountName}{"\n"}{.spec.automountServiceAccountToken}{"\n"}'

This is your problem if the output shows the wrong service account name, or false for automountServiceAccountToken when your setup relies on projected tokens. Jump to Fixes → Pod uses the wrong service account or token automount is disabled.

1. Check whether the pod has a projected token file at all.

kubectl exec -n <ns> <pod> -- sh -c 'find /var/run/secrets -maxdepth 3 -type f | sort; echo; id'

This is your problem if you do not see a token under a projected path such as /var/run/secrets/tokens/... and your SDK/provider expects one. Jump to Fixes → Projected service account token is missing or not mounted.

1. Inspect the Kubernetes service account manifest for workload identity annotations/labels your provider requires.

kubectl get sa -n <ns> <sa> -o yaml

This is your problem if the service account has no workload-identity-related annotation/label expected by your platform, or references the wrong cloud principal. Jump to Fixes → Kubernetes service account is not bound or mapped to the cloud identity.

1. Decode the token claims and compare iss, sub, and aud with the cloud-side trust policy.

kubectl exec -n <ns> <pod> -- sh -c 'python3 - <<"PY"
import glob,base64,json
files=glob.glob("/var/run/secrets/tokens/*")
if not files:
    print("no token files")
    raise SystemExit(1)
t=open(files[0]).read().strip().split(".")
print(json.dumps(json.loads(base64.urlsafe_b64decode(t[1]+"==")), indent=2))
PY'

This is your problem if iss does not match the issuer configured in the cloud trust, aud is wrong for your provider's STS/token exchange endpoint, or sub does not match the expected system:serviceaccount:<namespace>:<serviceaccount>. Jump to Fixes → Token issuer or audience does not match cloud trust.

1. Test DNS and HTTPS connectivity from the pod to both the token endpoint and the target API.

kubectl exec -n <ns> <pod> -- sh -c 'getent hosts <token-endpoint-host> <api-host>; echo; curl -sv https://<token-endpoint-host>/ 2>&1 | sed -n "1,20p"; echo; curl -sv https://<api-host>/ 2>&1 | sed -n "1,20p"'

This is your problem if DNS lookup fails, TCP connect fails, or you get a proxy/firewall denial before auth even happens. Typical output shape:

* Could not resolve host: sts.examplecloud.com
curl: (6) Could not resolve host: sts.examplecloud.com

or:

*   Trying 10.20.30.40:443...
* connect to 10.20.30.40 port 443 failed: Connection timed out
curl: (28) Failed to connect

Jump to Fixes → NetworkPolicy, proxy, firewall, or DNS blocks egress.

1. Check for env vars or mounted secrets that force the SDK down a different credential path.

kubectl exec -n <ns> <pod> -- sh -c 'env | grep -E "AWS_|AZURE_|GOOGLE_|CLOUD_|_CREDENTIALS|_TOKEN|HTTP_PROXY|HTTPS_PROXY|NO_PROXY" | sort'

This is your problem if you still have old static credential env vars, a credentials file path, or a proxy that intercepts token exchange traffic. Jump to Fixes → SDK is preferring old credentials or wrong credential source.

Fixes

Projected service account token is missing or not mounted

Patch the workload to use a service account and enable token automount if you disabled it.

kubectl patch deploy -n <ns> <deploy> --type='merge' -p '{"spec":{"template":{"spec":{"serviceAccountName":"<sa>","automountServiceAccountToken":true}}}}'
kubectl rollout restart deploy -n <ns> <deploy>

If your provider requires an explicit projected token volume with a specific audience, add it to the pod spec:

spec:
  serviceAccountName: app-sa
  containers:
  - name: app
    volumeMounts:
    - name: wi-token
      mountPath: /var/run/secrets/tokens
      readOnly: true
  volumes:
  - name: wi-token
    projected:
      sources:
      - serviceAccountToken:
          path: oidc-token
          expirationSeconds: 3600
          audience: <expected-audience>

Verify it worked:

kubectl exec -n <ns> deploy/<deploy> -- ls -l /var/run/secrets/tokens

Kubernetes service account is not bound or mapped to the cloud identity

Edit the Kubernetes service account so it points at the intended cloud principal, using the annotation/label format your provider expects, then restart pods.

kubectl annotate sa -n <ns> <sa> <provider-binding-key>=<cloud-principal> --overwrite
kubectl rollout restart deploy -n <ns> <deploy>

If your provider also requires a cloud-side trust policy, update it so the subject matches the Kubernetes service account exactly:

{
  "issuer": "https://<cluster-oidc-issuer>",
  "subject": "system:serviceaccount:<ns>:<sa>",
  "audience": "<expected-audience>"
}

Verify it worked:

kubectl get sa -n <ns> <sa> -o yaml | sed -n '1,120p'

Token issuer or audience does not match cloud trust

Fix either side so the token claims and trust policy agree. If the pod projects a token with the wrong audience, patch the pod spec and redeploy.

volumes:
- name: wi-token
  projected:
    sources:
    - serviceAccountToken:
        path: oidc-token
        audience: <provider-required-audience>
        expirationSeconds: 3600

If the cloud trust points at the wrong issuer URL or subject, update the trust configuration in your provider dashboard/CLI to match the token claims you decoded from the pod.

Use this claim shape as the source of truth:

{
  "iss": "https://<cluster-issuer>",
  "sub": "system:serviceaccount:<ns>:<sa>",
  "aud": ["<provider-required-audience>"]
}

Verify it worked:

kubectl exec -n <ns> <pod> -- sh -c 'python3 - <<"PY"
import glob,base64,json
f=glob.glob("/var/run/secrets/tokens/*")[0]
t=open(f).read().strip().split(".")
print(json.loads(base64.urlsafe_b64decode(t[1]+"=="))["aud"])
PY'

NetworkPolicy, proxy, firewall, or DNS blocks egress

If you use NetworkPolicy, allow egress from the app namespace to DNS and the provider token/API endpoints. Example policy skeleton:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-cloud-api-egress
  namespace: <ns>
spec:
  podSelector:
    matchLabels:
      app: <app>
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
  - to: []
    ports:
    - protocol: TCP
      port: 443

If a proxy is required, set HTTPS_PROXY and NO_PROXY correctly for both the token endpoint and in-cluster addresses. If a proxy is breaking TLS inspection, bypass it for the provider endpoints.

⚠️ Broad egress rules can increase your attack surface. If your CNI supports FQDN policies, prefer provider endpoint hostnames over to: [].

Verify it worked:

kubectl exec -n <ns> <pod> -- sh -c 'curl -sS -o /dev/null -w "%{http_code}\n" https://<token-endpoint-host>/'

SDK is preferring old credentials or wrong credential source

Remove static credential env vars and secret mounts from the workload so the SDK falls through to workload identity.

kubectl set env deploy/<deploy> -n <ns> AWS_ACCESS_KEY_ID- AWS_SECRET_ACCESS_KEY- GOOGLE_APPLICATION_CREDENTIALS- AZURE_CLIENT_SECRET- AZURE_TENANT_ID- AZURE_CLIENT_ID-
kubectl rollout restart deploy -n <ns> <deploy>

If your app sets an explicit credentials file path in code or config, delete that setting and let the provider SDK use its default chain. Also check for stale init containers writing credential files into a shared volume.

Verify it worked:

kubectl exec -n <ns> <pod> -- sh -c 'env | grep -E "AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY|GOOGLE_APPLICATION_CREDENTIALS|AZURE_CLIENT_SECRET"; test $? -eq 1 && echo clean'

Pod uses the wrong service account or token automount is disabled

Patch the workload template so every new pod uses the intended service account.

kubectl patch deploy -n <ns> <deploy> --type='merge' -p '{"spec":{"template":{"spec":{"serviceAccountName":"<sa>","automountServiceAccountToken":true}}}}'
kubectl rollout restart deploy -n <ns> <deploy>

If the service account was changed recently, old ReplicaSet pods may still be running with the previous spec. Confirm after rollout that all pods are new.

Verify it worked:

kubectl get pod -n <ns> -l app=<app> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.serviceAccountName}{"\n"}{end}'

Prevention

• Add a startup probe or init check that fails fast when the token file is missing.

test -r /var/run/secrets/tokens/oidc-token

• Add a CI policy check for workloads in protected namespaces: reject manifests without serviceAccountName or with automountServiceAccountToken: false.

yq '.spec.template.spec.serviceAccountName' deploy.yaml

• Emit a synthetic auth check from each workload after deploy: call the provider token endpoint and one low-risk API operation, then alert on non-2xx/3xx.

curl -sS -o /dev/null -w '%{http_code}\n' https://<token-endpoint-host>/

• Pin SDK major versions and test workload identity in integration tests. Credential-chain behavior does change between majors.

{
  "dependencies": {
    "<cloud-sdk>": "^<known-good-major.minor>"
  }
}

• Alert on Kubernetes audit/events for service account changes in app namespaces.

kubectl get events -A --field-selector involvedObject.kind=ServiceAccount --sort-by=.lastTimestamp

• Keep a one-pager in the repo with the exact expected token claims: issuer, audience, and subject format. During incidents, compare decoded claims from the pod to that doc instead of guessing.

[Pod] ---- 1. read projected token ----> [K8s SA token]
[Pod] ---- 2. exchange token ----> [Cloud STS]
[Cloud STS] ---- 3. issue access token ----> [Pod]
[Pod] ---- 4. call cloud API ----> [Cloud API]

This article was written by an AI system and published pending human review. Verify anything you intend to act on.