AWS AccessDenied: identify which policy type denied the request

TL;DR — Most AWS "AccessDenied" incidents are not "missing IAM permission" in the simple sense; the useful question is which policy layer produced the deny or failed to allow the action. Start with the exact error text and CloudTrail event, then check in this order: explicit deny in the message, Organizations SCP, permissions boundary/session policy, resource policy, and service-specific layers like KMS key policy or VPC endpoint policy. Reading time: ~6 min

The scenario

You kick off a routine deploy on a Tuesday afternoon and the pipeline dies on a step that has worked for months: uploading an artifact to S3, decrypting a secret with KMS, or updating a Lambda role. The AWS CLI just says AccessDenied, your app logs only show a 403 from some downstream AWS API, and the person who last touched IAM is on leave. You know the principal exists and the credentials are valid because sts get-caller-identity works. What you need now is not a lecture on IAM — you need to find the policy type that actually said no.

Symptoms

• AWS CLI returns one of these verbatim patterns:

An error occurred (AccessDenied) when calling the PutObject operation: User: arn:aws:sts::123456789012:assumed-role/ci-deploy-role/GitHubActions is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::my-bucket/releases/app.tar.gz" because no identity-based policy allows the s3:PutObject action

An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::123456789012:assumed-role/app-role/i-0abc is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/db because no resource-based policy allows the secretsmanager:GetSecretValue action

An error occurred (AccessDenied) when calling the Decrypt operation: User is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:us-east-1:123456789012:key/1111-2222-3333-4444 because no identity-based policy allows the kms:Decrypt action

An error occurred (AccessDenied) when calling the CreateFunction operation: User: arn:aws:iam::123456789012:user/alice is not authorized to perform: lambda:CreateFunction on resource: * with an explicit deny in a service control policy

• AWS SDK/app logs show HTTP 403 with service-specific wrappers like AccessDeniedException, Client.UnauthorizedOperation, or UnauthorizedOperation.
• CloudTrail event has errorCode like AccessDenied, AccessDeniedException, Client.UnauthorizedOperation, or VpceAccessDenied.
• IAM Policy Simulator says allowed, but the real call still fails.
• The same action works from your admin user but fails from an assumed role, CI role, ECS task role, Lambda execution role, or SSO session.

Likely causes

Step-by-step diagnosis

1. Confirm which principal is actually making the call.

aws sts get-caller-identity

If the ARN is not the role/user/session you expected, this is your problem. Common examples: local shell still using old profile, CI assuming a different role, ECS task using task role not execution role. Fix the credential source first, then retry before doing deeper IAM work.

1. Re-run the failing command with --debug and copy the exact service, action, resource ARN, and request ID.

aws s3api put-object --bucket my-bucket --key releases/app.tar.gz --body app.tar.gz --debug 2>&1 | tee /tmp/aws-debug.log

This is your problem if the error text already names the policy type, for example:

... because no identity-based policy allows the s3:PutObject action
... because no resource-based policy allows the secretsmanager:GetSecretValue action
... with an explicit deny in a service control policy

Jump straight to the matching fix section.

1. Pull the CloudTrail event for the failed request.

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=PutObject --max-results 10

If needed, filter by time and principal in your log store or CloudTrail Lake. In the event JSON, look for userIdentity.arn, errorCode, errorMessage, and resources. If errorMessage names SCP, permissions boundary, or VPC endpoint policy, jump to that fix. If CloudTrail shows a different principal than step 1, fix the credential chain.

1. Simulate the principal policy for the exact action/resource.

aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::123456789012:role/ci-deploy-role --action-names s3:PutObject --resource-arns arn:aws:s3:::my-bucket/releases/app.tar.gz

This is your problem if EvalDecision is implicitDeny or explicitDeny and the matched statements are in the role/user policies. If simulation says allowed but the real call fails, skip to step 6; you are likely dealing with resource policy, session policy, SCP, VPC endpoint policy, or KMS.

1. Check for permissions boundary on the principal.

aws iam get-role --role-name ci-deploy-role --query 'Role.PermissionsBoundary'

If a boundary ARN is present, fetch and inspect it:

aws iam get-policy-version --policy-arn arn:aws:iam::123456789012:policy/DeployBoundary --version-id v3

This is your problem if the boundary omits the action/resource or has an explicit deny. Jump to the permissions boundary fix.

1. Check whether the principal is an assumed role session with extra restrictions.

aws sts get-caller-identity

If the ARN looks like arn:aws:sts::123456789012:assumed-role/role-name/session-name, inspect the code or CI config that called AssumeRole. Look for Policy, PolicyArns, Tags, or SSO permission set/session constraints. This is your problem if the base role allows the action but only assumed sessions fail.

1. Check the resource policy on the target resource.

• S3 bucket policy:

aws s3api get-bucket-policy --bucket my-bucket --query Policy --output text | jq .

• Secrets Manager resource policy:

aws secretsmanager get-resource-policy --secret-id arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/db --query ResourcePolicy --output text | jq .

• SQS queue policy:

aws sqs get-queue-attributes --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/myq --attribute-names Policy --query 'Attributes.Policy' --output text | jq .

This is your problem if the caller principal/account is missing, a condition excludes it, or there is an explicit deny.

1. If the account is in AWS Organizations, check SCPs.

aws organizations describe-effective-policy --policy-type SERVICE_CONTROL_POLICY --target-id 123456789012

If your org does not expose an effective policy in your setup, inspect attached SCPs from the management account or delegated admin. This is your problem if an SCP explicitly denies the action or your org uses allow-list SCPs that do not include the service/action.

1. If the call goes through a VPC endpoint, inspect the endpoint policy.

aws ec2 describe-vpc-endpoints --filters Name=service-name,Values=com.amazonaws.us-east-1.s3 --query 'VpcEndpoints[].{Id:VpcEndpointId,Policy:PolicyDocument}'

This is your problem if the endpoint policy excludes the action, bucket, secret, or principal. Typical CloudTrail clue: VpceAccessDenied.

1. For KMS-backed operations, inspect the key policy and grants even if the top-level error came from S3, EBS, Secrets Manager, or Lambda.

aws kms get-key-policy --key-id arn:aws:kms:us-east-1:123456789012:key/1111-2222-3333-4444 --policy-name default | jq -r .Policy | jq .

This is your problem if the principal/account/service is not allowed in the key policy or a condition blocks it. KMS is often the hidden second deny after the first service appears correctly configured.

Fixes

Identity policy does not allow the action/resource

Attach or update the principal policy with the exact action and resource ARN.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::my-bucket/releases/*"]
    }
  ]
}

Apply it to a managed policy version or inline policy:

aws iam create-policy-version --policy-arn arn:aws:iam::123456789012:policy/ci-deploy --policy-document file://policy.json --set-as-default

Verify it worked:

aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::123456789012:role/ci-deploy-role --action-names s3:PutObject --resource-arns arn:aws:s3:::my-bucket/releases/app.tar.gz

Organizations SCP blocks the action

From the management account or delegated admin, update the SCP attached to the account/OU.

⚠️ Editing an SCP can widen access across multiple accounts. Test on the narrowest OU/account possible before attaching broadly.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowDeployWritesToS3",
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["*"]
    }
  ]
}

If you use deny-based SCPs, remove or narrow the matching deny statement instead. Verify it worked by retrying the exact call and checking CloudTrail no longer reports explicit deny in a service control policy.

Resource policy does not allow the caller

Add the caller principal/account to the resource policy and fix any conditions. Example S3 bucket policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::123456789012:role/ci-deploy-role"},
      "Action": ["s3:PutObject"],
      "Resource": "arn:aws:s3:::my-bucket/releases/*"
    }
  ]
}

Apply it:

aws s3api put-bucket-policy --bucket my-bucket --policy file://bucket-policy.json

Verify it worked:

aws s3api put-object --bucket my-bucket --key releases/test.txt --body /etc/hosts

Permissions boundary blocks it

Update the boundary policy to include the action/resource, or attach a less restrictive boundary to the role.

aws iam put-role-permissions-boundary --role-name ci-deploy-role --permissions-boundary arn:aws:iam::123456789012:policy/DeployBoundaryV2

If the boundary itself needs editing:

aws iam create-policy-version --policy-arn arn:aws:iam::123456789012:policy/DeployBoundaryV2 --policy-document file://boundary.json --set-as-default

Verify it worked:

aws iam get-role --role-name ci-deploy-role --query 'Role.PermissionsBoundary'

Session policy or AssumeRole/SSO session restrictions block it

Remove or widen the inline session policy passed to AssumeRole, or fix session tags/conditions. Example assume-role call that is too restrictive:

aws sts assume-role --role-arn arn:aws:iam::123456789012:role/ci-deploy-role --role-session-name build-123 --policy file://session-policy.json

If you control the caller, remove --policy for a test or update session-policy.json. In CI, inspect the action/plugin or SDK call that assumes the role. Verify it worked by assuming the role again and retrying the failing API call with the temporary credentials.

VPC endpoint policy denies the call

Update the endpoint policy to allow the needed action/resource/principal.

{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::my-bucket", "arn:aws:s3:::my-bucket/*"]
    }
  ]
}

Apply it:

aws ec2 modify-vpc-endpoint --vpc-endpoint-id vpce-0123456789abcdef0 --policy-document file://vpce-policy.json

Verify it worked by retrying from the same subnet/path that uses the endpoint; testing from your laptop may bypass the endpoint and give a false pass.

KMS key policy/grant blocks the action

Add the principal to the key policy or create a grant if the service pattern calls for it.

{
  "Sid": "AllowCiDeployRoleUseOfKey",
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::123456789012:role/ci-deploy-role"},
  "Action": ["kms:Decrypt", "kms:Encrypt", "kms:GenerateDataKey"],
  "Resource": "*"
}

Apply the updated policy:

aws kms put-key-policy --key-id arn:aws:kms:us-east-1:123456789012:key/1111-2222-3333-4444 --policy-name default --policy file://key-policy.json

Verify it worked:

aws kms decrypt --key-id arn:aws:kms:us-east-1:123456789012:key/1111-2222-3333-4444 --ciphertext-blob fileb://blob.bin --output text --query Plaintext | base64 -d

Prevention

• Add a CI smoke test that exercises the exact role and exact API call after IAM changes.

aws sts assume-role --role-arn "$DEPLOY_ROLE_ARN" --role-session-name ci-smoke >/tmp/creds.json
export AWS_ACCESS_KEY_ID=$(jq -r .Credentials.AccessKeyId /tmp/creds.json)
export AWS_SECRET_ACCESS_KEY=$(jq -r .Credentials.SecretAccessKey /tmp/creds.json)
export AWS_SESSION_TOKEN=$(jq -r .Credentials.SessionToken /tmp/creds.json)
aws iam simulate-principal-policy --policy-source-arn "$DEPLOY_ROLE_ARN" --action-names s3:PutObject kms:Decrypt --resource-arns arn:aws:s3:::my-bucket/releases/test arn:aws:kms:us-east-1:123456789012:key/1111-2222-3333-4444

• Alert on CloudTrail access denials by policy type. In your log pipeline, parse errorMessage and group by substrings like service control policy, resource-based policy, permissions boundary, and VpceAccessDenied.
• Pin role assumption behavior in code and CI. If you pass session policies, keep them in versioned files in the repo instead of generated strings, and log the final assumed role ARN plus session name on startup/deploy.
• For KMS-backed resources, document the dependency pair in IaC: the service permission and the KMS key permission. Example: an S3 upload with SSE-KMS needs both s3:PutObject and kms:GenerateDataKey/kms:Decrypt depending on the path.
• Keep resource policies and endpoint policies in IaC and run policy diff review in PRs.

terraform plan -out=tfplan && terraform show -json tfplan | jq '.resource_changes[] | select(.type|test("aws_(s3_bucket_policy|kms_key_policy|vpce|iam_policy)"))'

• When using Organizations, maintain a small test account in each OU with a scripted permission probe so SCP regressions are caught before production deploys.

aws organizations list-policies-for-target --target-id "$ACCOUNT_ID" --filter SERVICE_CONTROL_POLICY

This article was written by an AI system and published pending human review. Verify anything you intend to act on.