Use CloudTrail to reconstruct who did what during AWS access incidents

TL;DR — To reconstruct who did what in AWS, pull CloudTrail events for the time window, normalize the actor from userIdentity, and correlate STS AssumeRole with the follow-on API calls. The most common reason investigations stall is looking only at Username or only in one Region; query all Regions and inspect userIdentity.arn, sessionIssuer, sourceIPAddress, and requestParameters together. Reading time: ~5 min

Goal

When you finish, you will have a defensible timeline of AWS API activity for a specific incident window: which principal made each call, from which IP, whether MFA was present, what role/session was used, and which resources were touched.

Prerequisites

• AWS account access with permission to read CloudTrail and, if used, Athena and S3 logs storage
• AWS CLI v2 — check with:

aws --version

• jq 1.6+ — check with:

jq --version

• A time window in UTC, for example 2026-08-05T09:00:00Z to 2026-08-05T12:00:00Z
• At least one starting indicator: IAM user name, role ARN, access key ID, source IP, AWS account ID, or resource name/ARN
• If your organization uses an org trail or centralized logging bucket, the account/role that can read it

Steps

Step 1: Confirm CloudTrail coverage and where logs live

Run these commands in the account you are investigating:

aws cloudtrail describe-trails --include-shadow-trails | jq '.trailList[] | {Name, HomeRegion, IsMultiRegionTrail, S3BucketName, S3KeyPrefix, IsOrganizationTrail}'

aws cloudtrail get-trail-status --name <trail-name>

You should see at least one trail with IsMultiRegionTrail: true or know exactly which Regions to query, and IsLogging: true in the status output.

Step 2: Pull raw events for the incident window with lookup-events

Start broad. Query by time first, then narrow by actor, access key, or resource.

START="2026-08-05T09:00:00Z"
END="2026-08-05T12:00:00Z"
REGION="us-east-1"

aws cloudtrail lookup-events \
  --region "$REGION" \
  --start-time "$START" \
  --end-time "$END" \
  --max-results 50 \
  --output json > /tmp/ct-events.json

jq '.Events[0] | {EventTime, EventName, Username, CloudTrailEvent}' /tmp/ct-events.json

You should get a JSON file with Events entries; CloudTrailEvent is a JSON string containing the full event.

Step 3: Normalize the actor, IP, MFA, and resource from each event

Convert the embedded JSON string into structured fields you can sort and grep.

jq -r '.Events[] | .CloudTrailEvent | fromjson | [
  .eventTime,
  .awsRegion,
  .eventSource,
  .eventName,
  (.userIdentity.type // "-"),
  (.userIdentity.arn // .userIdentity.sessionContext.sessionIssuer.arn // "-"),
  (.userIdentity.accessKeyId // "-"),
  (.sourceIPAddress // "-"),
  (.userAgent // "-"),
  (.userIdentity.sessionContext.attributes.mfaAuthenticated // "-"),
  (.recipientAccountId // "-"),
  (.requestParameters.roleArn // .requestParameters.bucketName // .requestParameters.functionName // .requestParameters.instanceId // "-")
] | @tsv' /tmp/ct-events.json | column -t -s $'\t'

You should see one line per event with columns for time, service, action, actor ARN, access key ID, source IP, MFA, and a best-effort target resource.

Step 4: If the actor used a role, find the AssumeRole event that created the session

For role sessions, the important identity is often in the STS event, not the later API call.

aws cloudtrail lookup-events \
  --region "$REGION" \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time "$START" \
  --end-time "$END" \
  --output json > /tmp/assume-role.json

jq -r '.Events[] | .CloudTrailEvent | fromjson | [
  .eventTime,
  .userIdentity.arn,
  .sourceIPAddress,
  .requestParameters.roleArn,
  .requestParameters.roleSessionName,
  .responseElements.assumedRoleUser.arn
] | @tsv' /tmp/assume-role.json | column -t -s $'\t'

You should see who assumed which role, from which IP, with which session name, and the resulting assumed-role ARN to match against later events.

Step 5: If you have an access key ID, pivot directly on it

This is the fastest path when GuardDuty, billing, or an app log gives you an access key.

ACCESS_KEY_ID="AKIAEXAMPLE123456789"

aws cloudtrail lookup-events \
  --region "$REGION" \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue="$ACCESS_KEY_ID" \
  --start-time "$START" \
  --end-time "$END" \
  --output json > /tmp/by-key.json

jq -r '.Events[] | .CloudTrailEvent | fromjson | [
  .eventTime,
  .eventName,
  .eventSource,
  .userIdentity.arn,
  .sourceIPAddress,
  .errorCode,
  .errorMessage
] | @tsv' /tmp/by-key.json | column -t -s $'\t'

You should see all API calls made with that key in the window, including failed calls with AccessDenied or other error codes.

Step 6: Query the S3-backed CloudTrail logs with Athena for complete timelines

lookup-events is useful but limited. For larger windows, multiple Regions, or organization trails, use Athena on the raw logs.

Create the table if you do not already have one. Replace the S3 path with your trail bucket and prefix.

CREATE EXTERNAL TABLE IF NOT EXISTS cloudtrail_logs (
  eventVersion STRING,
  userIdentity STRUCT<
    type:STRING,
    principalId:STRING,
    arn:STRING,
    accountId:STRING,
    accessKeyId:STRING,
    userName:STRING,
    sessionContext:STRUCT<
      attributes:STRUCT<mfaAuthenticated:STRING,creationDate:STRING>,
      sessionIssuer:STRUCT<type:STRING,principalId:STRING,arn:STRING,accountId:STRING,userName:STRING>
    >
  >,
  eventTime STRING,
  eventSource STRING,
  eventName STRING,
  awsRegion STRING,
  sourceIPAddress STRING,
  userAgent STRING,
  errorCode STRING,
  errorMessage STRING,
  requestParameters STRING,
  responseElements STRING,
  additionalEventData STRING,
  requestID STRING,
  eventID STRING,
  readOnly STRING,
  resources ARRAY<STRUCT<ARN:STRING,accountId:STRING,type:STRING>>,
  eventType STRING,
  recipientAccountId STRING,
  serviceEventDetails STRING,
  sharedEventID STRING,
  vpcEndpointId STRING
)
PARTITIONED BY (region STRING, year STRING, month STRING, day STRING)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://<trail-bucket>/<optional-prefix>/AWSLogs/<account-id>/CloudTrail/';

Then load partitions:

MSCK REPAIR TABLE cloudtrail_logs;

Now run the investigation query:

SELECT eventTime, awsRegion, eventSource, eventName,
       COALESCE(userIdentity.arn, userIdentity.sessionContext.sessionIssuer.arn) AS actor_arn,
       userIdentity.accessKeyId AS access_key_id,
       sourceIPAddress, userAgent,
       userIdentity.sessionContext.attributes.mfaAuthenticated AS mfa,
       errorCode,
       json_extract_scalar(requestParameters, '$.roleArn') AS role_arn
FROM cloudtrail_logs
WHERE from_iso8601_timestamp(eventTime)
      BETWEEN from_iso8601_timestamp('2026-08-05T09:00:00Z')
          AND from_iso8601_timestamp('2026-08-05T12:00:00Z')
  AND recipientAccountId = '<account-id>'
ORDER BY from_iso8601_timestamp(eventTime);

You should get a time-ordered result set across the trail data, not just the most recent 50 events.

Step 7: Build the final timeline and preserve evidence

Export the query results or normalized CLI output to a file you can attach to the incident.

jq -r '.Events[] | .CloudTrailEvent | fromjson | {
  eventTime,
  awsRegion,
  eventSource,
  eventName,
  actorArn: (.userIdentity.arn // .userIdentity.sessionContext.sessionIssuer.arn // "-"),
  principalType: (.userIdentity.type // "-"),
  accessKeyId: (.userIdentity.accessKeyId // "-"),
  sourceIPAddress: (.sourceIPAddress // "-"),
  mfa: (.userIdentity.sessionContext.attributes.mfaAuthenticated // "-"),
  errorCode: (.errorCode // "-"),
  requestParameters: (.requestParameters // {}),
  resources: (.resources // [])
}' /tmp/ct-events.json > incident-timeline.json

You should end with a machine-readable timeline file you can diff, search, and hand to responders.

Verify it works

Run one of these checks against your output:

jq 'length' incident-timeline.json

jq -r '.[] | [.eventTime, .eventName, .actorArn, .sourceIPAddress, .mfa] | @tsv' incident-timeline.json | head

Expected shape:

2026-08-05T09:14:22Z  ConsoleLogin       arn:aws:iam::123456789012:user/alice                 203.0.113.10   true
2026-08-05T09:16:03Z  AssumeRole         arn:aws:iam::123456789012:user/alice                 203.0.113.10   true
2026-08-05T09:16:05Z  GetCallerIdentity  arn:aws:sts::123456789012:assumed-role/Admin/alice   203.0.113.10   true
2026-08-05T09:17:11Z  PutBucketPolicy    arn:aws:sts::123456789012:assumed-role/Admin/alice   203.0.113.10   true

If you can point from the initial identity to any AssumeRole event and then to the follow-on API calls with matching assumed-role ARN, the reconstruction is working.

Common pitfalls

Looking only in one Region

Mistake: running lookup-events only in your default CLI Region. Symptom: obvious actions are missing, especially global-service-related or cross-Region activity. Fix: run aws configure get region, then query every Region in scope or use the multi-Region trail’s S3 logs via Athena.

Trusting Username as the actor

Mistake: using the top-level Username field as the identity source of truth. Symptom: events appear to come from a generic role name or have blank usernames. Fix: read CloudTrailEvent.userIdentity.arn and userIdentity.sessionContext.sessionIssuer.arn instead.

Missing the STS hop

Mistake: reviewing only the sensitive API call and not the preceding AssumeRole or GetSessionToken event. Symptom: you know which role acted but not who started the session. Fix: query EventName=AssumeRole in the same window and match responseElements.assumedRoleUser.arn to later events.

Assuming CloudTrail records every data-plane action by default

Mistake: expecting object-level S3 or Lambda invoke details without data events enabled. Symptom: management events exist, but object reads/writes or function invokes are absent. Fix: confirm trail event selectors; if data events were not enabled before the incident, CloudTrail cannot reconstruct them retroactively.

Ignoring failed API calls

Mistake: filtering out errors too early. Symptom: you miss reconnaissance or privilege-escalation attempts because only successful calls remain. Fix: include errorCode and errorMessage in your output and review AccessDenied, UnauthorizedOperation, and ConsoleLogin failures first.

Athena table points at the wrong S3 prefix

Mistake: creating the table at s3://bucket/ instead of the CloudTrail account path. Symptom: MSCK REPAIR TABLE finds no partitions or queries return zero rows. Fix: set LOCATION to s3://<trail-bucket>/<prefix>/AWSLogs/<account-id>/CloudTrail/ and rerun MSCK REPAIR TABLE.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.