Managed identity gets 403 from Key Vault despite role assignment

TL;DR — A 403 from a key vault with an existing role assignment usually means one of six things: you assigned the role to the wrong principal, the vault is using access policies instead of RBAC, the assignment is at the wrong scope or missing a data-plane role, the assignment has not propagated yet, the vault network rules are blocking the caller, or the app is using a token for the wrong resource. The fastest win is to confirm the object ID of the managed identity your app is actually using, then verify the vault permission model and a data-plane role at the vault scope. Reading time: ~6 min

The scenario

You push a Tuesday afternoon deploy that switches your app from a client secret to managed identity for secret reads. The app starts, health checks pass, but every call that needs a secret fails with 403 and your logs say the role assignment is already there. In the cloud portal, someone can point to a shiny "Secrets User" role on the vault and swear the permissions are correct. Meanwhile the app is still returning 500s because it cannot fetch the database password from the vault.

Symptoms

• Application logs show 403 on secret/key/certificate reads, for example:

Azure.RequestFailedException: Operation returned an invalid status code 'Forbidden'
Status: 403 (Forbidden)
ErrorCode: Forbidden

Content:
{"error":{"code":"Forbidden","message":"Caller is not authorized to perform action on resource.\nIf role assignments, deny assignments or role definitions were changed recently, please observe propagation time.\nCaller: appid=<guid>;oid=<guid>;iss=https://sts.windows.net/<tenant-guid>/\nAction: 'Microsoft.KeyVault/vaults/secrets/readMetadata/action'\nResource: '/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<vault>/secrets/<name>'\nAssignment: (not found)\nDenyAssignmentId: null\nDecisionReason: null\nVault: <vault>;location=<region>"}}

• Or a network-shaped 403 instead of an authz-shaped 403:

{"error":{"code":"ForbiddenByFirewall","message":"Client address is not authorized and caller is not a trusted service.\nClient address: 203.0.113.10 from unknown subnet\nCaller: appid=<guid>;oid=<guid>;iss=https://sts.windows.net/<tenant-guid>/\nVault: <vault>;location=<region>"}}

• curl against the vault endpoint from the workload host returns 403 even before your SDK retries:

curl -i https://<vault>.vault.azure.net/secrets/<name>?api-version=7.4 -H "Authorization: Bearer $TOKEN"

HTTP/1.1 403 Forbidden
Content-Type: application/json; charset=utf-8
x-ms-keyvault-region: <region>

{"error":{"code":"Forbidden","message":"Caller is not authorized..."}}

• Portal/CLI shows a role assignment exists, but the app still fails.
• The same secret read works from your user account or from a different app instance, but not from this deployment slot/pod/VM.

Likely causes

| Vault is using access policies, not RBAC permission model | Very common | ```bash
az keyvault show -n "$VAULT" --query properties.enableRbacAuthorization -o tsv
``` |
| Role assignment exists, but at the wrong scope or wrong role for data plane | Common | ```bash
az role assignment list --assignee-object-id "$MI_OBJECT_ID" --scope "$VAULT_ID" -o table
``` |
| RBAC propagation delay / stale token | Common | ```bash
az account get-access-token --resource https://vault.azure.net --query expiresOn -o tsv
``` |
| Vault firewall/private endpoint/network ACLs blocking the caller | Common | ```bash
az keyvault show -n "$VAULT" --query properties.networkAcls -o json
``` |
| App requested a token for the wrong audience/resource | Less common | ```bash
python - <<'PY'
import os,base64,json
jwt=os.environ['TOKEN'].split('.')[1]+'=='
print(json.loads(base64.urlsafe_b64decode(jwt))['aud'])
PY
``` |

## Step-by-step diagnosis
1. Confirm which managed identity your workload is actually using.
   - For a resource with system-assigned or user-assigned identity, inspect the resource identity block:
```bash
az resource show --ids "$APP_RESOURCE_ID" --query identity -o json

• What indicates the problem: the principalId or listed user-assigned identity does not match the principal you granted on the vault.
• Jump to: Fixes → Wrong managed identity object.

1. Check whether the vault is using RBAC or legacy access policies.
   • Run:

az keyvault show -n "$VAULT" --query '{rbac:properties.enableRbacAuthorization,accessPolicies:properties.accessPolicies[].objectId}' -o json

• What indicates the problem: rbac is false. In that mode, RBAC role assignments on the vault do not authorize data-plane secret/key/cert reads; access policies do.
• Jump to: Fixes → Vault permission model is access policies, not RBAC.

1. Check the role assignment at the vault scope and verify it is a data-plane role.
   • Get the vault resource ID and list assignments:

VAULT_ID=$(az keyvault show -n "$VAULT" --query id -o tsv)
az role assignment list --assignee-object-id "$MI_OBJECT_ID" --scope "$VAULT_ID" -o table

• What indicates the problem: no rows, or only management-plane roles like Reader, Contributor, or a role assigned at an unrelated scope. For secret reads you want a data-plane role such as Key Vault Secrets User or broader if justified.
• Jump to: Fixes → Role assignment scope or role type is wrong.

1. If the assignment was just added or changed, rule out propagation and stale tokens.
   • Fetch a fresh token from the workload context if possible. On a host with managed identity available:

TOKEN=$(curl -s -H Metadata:true "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net" | jq -r .access_token)

• Then call the vault directly:

curl -i "https://$VAULT.vault.azure.net/secrets/$SECRET?api-version=7.4" -H "Authorization: Bearer $TOKEN"

• What indicates the problem: the role assignment exists and is correct, but direct calls still return authz 403 for a short period after the change; later retries succeed without config changes.
• Jump to: Fixes → RBAC propagation delay or stale token.

1. Check whether the 403 is actually from network ACLs or private endpoint routing.
   • Inspect network ACLs:

az keyvault show -n "$VAULT" --query '{defaultAction:properties.networkAcls.defaultAction,bypass:properties.networkAcls.bypass,ipRules:properties.networkAcls.ipRules,virtualNetworkRules:properties.networkAcls.virtualNetworkRules,publicNetworkAccess:properties.publicNetworkAccess}' -o json

• What indicates the problem: error code ForbiddenByFirewall, publicNetworkAccess disabled, or the caller subnet/IP is not in allowed rules and the workload is not reaching the private endpoint.
• Jump to: Fixes → Vault firewall or private endpoint blocks the caller.

1. Verify the token audience is the vault data plane.
   • Decode the token you are actually sending:

python - <<'PY'
import os,base64,json
parts=os.environ['TOKEN'].split('.')
payload=parts[1] + '=' * (-len(parts[1]) % 4)
print(json.dumps(json.loads(base64.urlsafe_b64decode(payload)), indent=2))
PY

• What indicates the problem: aud is not https://vault.azure.net (or the SDK equivalent scope https://vault.azure.net/.default). Common wrong values are https://management.azure.com/ or Microsoft Graph.
• Jump to: Fixes → Token audience/resource is wrong.

[App] ---- 1. get MI token ----> [IMDS]
  |
  | 2. Bearer token aud=vault?
  v
[Key Vault] <---- 3. data-plane authz ---- [RBAC/Policies]
  |
  | 4. network ACL / private endpoint
  v
[403 Forbidden]

Fixes

Wrong managed identity object

If your app resource has both system-assigned and user-assigned identities, or you granted a role to the app registration/service principal instead of the managed identity principal, fix the assignment to the actual principalId used by the workload.

Find the principal ID:

az resource show --ids "$APP_RESOURCE_ID" --query identity -o json

Assign the role at vault scope to that object ID:

VAULT_ID=$(az keyvault show -n "$VAULT" --query id -o tsv)
az role assignment create --assignee-object-id "$MI_OBJECT_ID" --assignee-principal-type ServicePrincipal --role "Key Vault Secrets User" --scope "$VAULT_ID"

If your code should use a specific user-assigned identity, pin it in code/config instead of relying on default selection. Example with environment variable used by common SDKs:

export AZURE_CLIENT_ID="<user-assigned-managed-identity-client-id>"

Verify it worked:

az role assignment list --assignee-object-id "$MI_OBJECT_ID" --scope "$VAULT_ID" -o table

Vault permission model is access policies, not RBAC

If enableRbacAuthorization is false, RBAC role assignments do not grant data-plane access. Either switch the vault to RBAC, or add an access policy for the managed identity.

⚠️ Switching permission model can break existing access if you have only access policies today. Review current consumers before changing it.

Option A: switch to RBAC:

az keyvault update -n "$VAULT" --enable-rbac-authorization true

Then add the data-plane role:

VAULT_ID=$(az keyvault show -n "$VAULT" --query id -o tsv)
az role assignment create --assignee-object-id "$MI_OBJECT_ID" --assignee-principal-type ServicePrincipal --role "Key Vault Secrets User" --scope "$VAULT_ID"

Option B: stay on access policies and grant secret get/list:

az keyvault set-policy -n "$VAULT" --object-id "$MI_OBJECT_ID" --secret-permissions get list

Verify it worked:

az keyvault show -n "$VAULT" --query properties.enableRbacAuthorization -o tsv

Role assignment scope or role type is wrong

Management-plane roles like Reader and Contributor do not let the app read secrets. Grant a data-plane role at the vault scope unless you have a reason to scope narrower.

List current assignments:

az role assignment list --assignee-object-id "$MI_OBJECT_ID" --scope "$VAULT_ID" -o table

Grant the minimum data-plane role for secrets:

az role assignment create --assignee-object-id "$MI_OBJECT_ID" --assignee-principal-type ServicePrincipal --role "Key Vault Secrets User" --scope "$VAULT_ID"

If the app needs keys or certificates instead, use the matching data-plane role rather than broad admin roles.

Verify it worked:

az role assignment list --assignee-object-id "$MI_OBJECT_ID" --scope "$VAULT_ID" -o table

RBAC propagation delay or stale token

Role assignments are not always effective instantly, and your process may cache tokens.

Force a fresh token and retry from the workload host:

unset TOKEN
TOKEN=$(curl -s -H Metadata:true "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net" | jq -r .access_token)
curl -i "https://$VAULT.vault.azure.net/secrets/$SECRET?api-version=7.4" -H "Authorization: Bearer $TOKEN"

If your app caches credentials aggressively, restart the process/pod after the assignment change.

Verify it worked:

curl -s -o /dev/null -w "%{http_code}\n" "https://$VAULT.vault.azure.net/secrets/$SECRET?api-version=7.4" -H "Authorization: Bearer $TOKEN"

Expected: 200.

Vault firewall or private endpoint blocks the caller

A valid identity with a valid role still gets 403 if the vault network rules deny the source.

Inspect ACLs:

az keyvault show -n "$VAULT" --query properties.networkAcls -o json

If public access is allowed and you need to permit a known egress IP:

az keyvault network-rule add -n "$VAULT" --ip-address "203.0.113.10"

If the workload should use a private endpoint, fix DNS/routing so <vault>.vault.azure.net resolves to the private IP from the workload network. Test from the host:

nslookup "$VAULT.vault.azure.net"
curl -I "https://$VAULT.vault.azure.net/"

Typical healthy shape is a vault response, not a timeout. If DNS points public while publicNetworkAccess is disabled, fix your private DNS zone/linking in your provider's dashboard or IaC.

Verify it worked:

curl -i "https://$VAULT.vault.azure.net/secrets/$SECRET?api-version=7.4" -H "Authorization: Bearer $TOKEN"

Expected: no ForbiddenByFirewall.

Token audience/resource is wrong

If your app asks managed identity for a token for ARM or Graph, the vault will reject it even if RBAC is correct.

Request the correct resource/scope.

For IMDS-style token requests:

curl -s -H Metadata:true "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net"

For SDKs using scopes, use:

https://vault.azure.net/.default

Example in code config:

{
  "KeyVaultScope": "https://vault.azure.net/.default"
}

Verify it worked by decoding the token and checking aud:

python - <<'PY'
import os,base64,json
p=os.environ['TOKEN'].split('.')[1]
p += '=' * (-len(p) % 4)
print(json.loads(base64.urlsafe_b64decode(p))['aud'])
PY

Expected output:

https://vault.azure.net

Prevention

• Add a startup probe that performs a real secret metadata read, not just token acquisition. Example shell check for a container health probe:

TOKEN=$(curl -s -H Metadata:true "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net" | jq -r .access_token) && curl -fsS "https://$VAULT.vault.azure.net/secrets/$SECRET?api-version=7.4" -H "Authorization: Bearer $TOKEN" >/dev/null

• Pin the intended user-assigned identity in app config so deployments do not silently switch identities:

export AZURE_CLIENT_ID="<managed-identity-client-id>"

• In CI, assert the vault permission model before applying RBAC-based IaC:

test "$(az keyvault show -n "$VAULT" --query properties.enableRbacAuthorization -o tsv)" = "true"

• In CI/CD, validate the exact assignee object ID and role at the exact vault scope after deploy:

az role assignment list --assignee-object-id "$MI_OBJECT_ID" --scope "$VAULT_ID" --query "[?roleDefinitionName=='Key Vault Secrets User']" -o json

• Alert on authz and firewall-specific 403s separately so you do not chase RBAC during a network outage. Example log filters to split on error code:

error.code == "Forbidden"
error.code == "ForbiddenByFirewall"

• If you use private endpoints, add a post-deploy DNS check from inside the workload network:

nslookup "$VAULT.vault.azure.net"

Expected result: private IP when public access is disabled.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.