Migrate off service account keys using impersonation and attached identities

TL;DR — Replace service account key files in this order: local/admin workflows first with impersonation, then CI/CD with workload identity or OIDC federation, then runtime workloads with attached service accounts. Only after every caller works without a JSON key should you disable key creation and delete existing keys.\n> Reading time: ~5 min\n\n## Goal\nWhen you are done, every app, script, CI job, and runtime workload that used a service account key will authenticate without a downloaded key file: developers use impersonation, compute uses an attached service account, and key creation is blocked so no new long-lived credentials can appear.\n\n## Prerequisites\n- Access to your cloud IAM admin role that can grant roles on service accounts and delete keys\n- CLI installed and current enough to support impersonation and workload identity features; check with:\nbash\ngcloud --version\naws --version\naz version\n\n- A list of service accounts currently in use and where each key file is referenced: repos, CI variables, Kubernetes secrets, VM startup scripts, local .env files\n- The exact service account email/name you are migrating, for example deploy-bot@PROJECT_ID.iam.gserviceaccount.com\n- Permission to update CI secrets/variables and runtime configuration\n- For Kubernetes or VMs: access to the cluster or instance templates to change the runtime identity\n\n## Steps\n\n### Step 1: Inventory every place a key file is used\nRun a repo and environment search before changing IAM.\nbash\nrg -n "GOOGLE_APPLICATION_CREDENTIALS|client_email|private_key_id|BEGIN PRIVATE KEY|service-account.*json|gcloud auth activate-service-account|aws_access_key_id|AZURE_CLIENT_SECRET" .\n\nprintenv | rg "GOOGLE_APPLICATION_CREDENTIALS|AWS_ACCESS_KEY_ID|AZURE_CLIENT_SECRET"\n\nIf you store secrets in Kubernetes, list likely references:\nbash\nkubectl get secrets -A | rg "service-account|gcp|google|aws|azure|json|key"\n\nSuccess looks like a concrete list of files, env vars, CI jobs, and workloads to migrate.\n\n### Step 2: Grant impersonation to humans and automation that still need API access\nFor Google Cloud, grant the minimum role that allows minting short-lived access on the target service account. Replace values first.\nbash\nexport PROJECT_ID="my-project"\nexport SA="deploy-bot@${PROJECT_ID}.iam.gserviceaccount.com"\nexport USER="alice@example.com"\n\ngcloud iam service-accounts add-iam-policy-binding "$SA" \\\n --member="user:${USER}" \\\n --role="roles/iam.serviceAccountTokenCreator"\n\nFor a CI principal or group, use its real member string, for example principalSet://..., serviceAccount:..., or group:devops@example.com.\nSuccess looks like Updated IAM policy for serviceAccount [...].\n\n### Step 3: Switch local/admin commands to impersonation\nStop using gcloud auth activate-service-account --key-file=.... Use your own login plus impersonation.\nbash\ngcloud auth login\ngcloud config set project "$PROJECT_ID"\ngcloud config set auth/impersonate_service_account "$SA"\ngcloud auth print-access-token | head -c 20 && echo\n\nFor one-off commands without changing global config:\nbash\ngcloud --impersonate-service-account="$SA" storage ls\n\nFor Application Default Credentials used by SDKs locally:\nbash\ngcloud auth application-default login\nexport GOOGLE_IMPERSONATE_SERVICE_ACCOUNT="$SA"\n\nSuccess looks like commands succeeding without any --key-file argument and no GOOGLE_APPLICATION_CREDENTIALS set.\n\n### Step 4: Move CI/CD off key files to federation or platform identity\nIf your CI platform can issue OIDC tokens, exchange them for cloud credentials instead of storing a JSON key. The exact UI varies by provider, but the action is always: create a trust relationship from your CI issuer to a cloud principal, then remove the stored key secret. For Google Cloud jobs using gcloud after federation is set up, the job should look like this shape, not activate-service-account:\nbash\ngcloud auth login --brief --cred-file="$GOOGLE_EXTERNAL_ACCOUNT_JSON"\ngcloud config set project "$PROJECT_ID"\ngcloud --impersonate-service-account="$SA" run deploy my-service --image "$IMAGE" --region us-central1\n\nIf your CI runs on a cloud-hosted runner with an attached identity, use that runtime identity directly and then impersonate the target deployment service account if needed.\nSuccess looks like the CI job no longer reading a secret named like GCP_SA_KEY, AWS_SECRET_ACCESS_KEY, or AZURE_CLIENT_SECRET.\n\n### Step 5: Attach a service account to runtime workloads instead of mounting keys\nFor VMs: update the instance or template to use the runtime service account you want. In your provider dashboard, open the VM or instance template and set the attached service account to the target identity; if using a managed group, roll a new template and replace instances. For Kubernetes, bind the workload identity your platform supports and remove the mounted JSON key secret from the pod spec. A generic before/after change is: remove env: GOOGLE_APPLICATION_CREDENTIALS=/var/secrets/key.json and the secret volume, then annotate or bind the pod/service account to the cloud identity.\n\nIf you need a concrete validation from inside the workload, query the metadata endpoint from a shell in the running container or VM. On Google-style metadata servers:\nbash\ncurl -s -H "Metadata-Flavor: Google" \\\n http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email\n\nTypical success output shape:\ntext\ndeploy-bot@my-project.iam.gserviceaccount.com\n\nSuccess looks like the workload reporting the attached service account email and continuing to call APIs without a key file present.\n\n> ⚠️ Do not disable key creation yet. If you do this before CI and runtime are migrated, deployments or background jobs will fail immediately with auth errors.\n\n### Step 6: Remove key-file references from code and config\nDelete explicit key loading paths so libraries use ambient credentials. Common fixes:\nbash\n# remove env var from shell startup or deployment config\nunset GOOGLE_APPLICATION_CREDENTIALS\n\n# remove secret mount references from manifests before applying updated YAML\nkubectl apply -f deploy.yaml\n\nIn code, delete constructors that point to a JSON file path and use the default credential chain.\nSuccess looks like startup logs no longer mentioning a credentials file path.\n\n### Step 7: Prove nothing still depends on keys, then disable and delete them\nList existing service account keys and delete user-managed ones after verification. For Google Cloud:\nbash\ngcloud iam service-accounts keys list --iam-account "$SA"\n\n# delete one key\ngcloud iam service-accounts keys delete KEY_ID --iam-account "$SA" --quiet\n\nThen block future key creation using your org policy or equivalent IAM guardrail in your cloud provider. In your provider's organization policy dashboard, set the policy that disables service account key creation for the relevant folders/projects, then enforce it.\nSuccess looks like key listing returning no active user-managed keys and new key creation attempts being denied.\n\n## Verify it works\nRun these checks after migration:\nbash\n# 1) No local key path in environment\nprintenv | rg "GOOGLE_APPLICATION_CREDENTIALS|AWS_ACCESS_KEY_ID|AZURE_CLIENT_SECRET"\n\n# 2) Impersonated local command works\ngcloud --impersonate-service-account="$SA" projects describe "$PROJECT_ID" --format='value(projectNumber)'\n\n# 3) Existing keys are gone\ngcloud iam service-accounts keys list --iam-account "$SA"\n\n# 4) New key creation is blocked; expect PERMISSION_DENIED or policy denial\ngcloud iam service-accounts keys create /tmp/test-key.json --iam-account "$SA"\n\nExpected results:\ntext\n# step 1: no output\n\n# step 2: prints a project number, exit code 0\n123456789012\n\n# step 3: no user-managed keys listed\nListed 0 items.\n\n# step 4: denied\nERROR: (gcloud.iam.service-accounts.keys.create) PERMISSION_DENIED: Request is prohibited by organization's policy.\n\nFor runtime verification from inside a pod or VM, the metadata query should return the attached service account email, and your app logs should stop showing errors like could not read json credentials file or invalid_grant.\n\n## Common pitfalls\n\n### You disabled key creation before migrating CI or workloads\nMistake: enforcing the org policy first.\nSymptom: builds or jobs fail immediately with messages like ERROR: could not read json credentials or permission denied opening /secrets/key.json.\nFix: temporarily exempt the project or revert the policy, migrate the failing caller to federation or attached identity, then re-enforce the policy.\n\n### You granted the wrong IAM role for impersonation\nMistake: giving only a viewer/editor role on the project, not Service Account Token Creator on the target service account.\nSymptom: PERMISSION_DENIED: The caller does not have permission iam.serviceAccounts.getAccessToken.\nFix: grant roles/iam.serviceAccountTokenCreator on the service account itself to the exact user/group/principal that is impersonating.\n\n### A hidden environment variable still points to a deleted key file\nMistake: removing the file but leaving GOOGLE_APPLICATION_CREDENTIALS or equivalent in CI variables, shell startup, or deployment config.\nSymptom: startup fails with no such file or directory, even though metadata identity is attached correctly.\nFix: delete the env var and restart the process so the SDK falls back to ambient credentials.\n\n### The workload has an attached identity, but the app still uses explicit file-based auth\nMistake: code still calls a credentials constructor with a JSON path.\nSymptom: metadata endpoint works, but the app keeps failing on the missing file path.\nFix: remove the explicit credentials-file code path and use the provider SDK default credential chain.\n\n### You deleted the wrong key or wrong service account\nMistake: multiple similar service account names, deleting before mapping callers.\nSymptom: one unrelated legacy job breaks while your target app works.\nFix: list keys and callers first, migrate one service account at a time, and delete keys only after a successful end-to-end verification for that specific identity.\n\n### CI federation is configured, but the trust condition does not match the job identity\nMistake: repository/branch/audience claims in the trust policy do not match the actual token claims.\nSymptom: login step fails with invalid_target, unauthorized_client, or a generic permission denied during token exchange.\nFix: print the CI OIDC token claims in a safe debug job, compare them to the trust policy, and update the exact subject/audience/branch condition values.

---

This article was written by an AI system and published pending human review. Verify anything you intend to act on.