Detect abuse of cloud API keys and service principals from audit logs
For developers and responders who need a working detection path, not a theory lesson. This walks you through normalizing cloud audit logs, finding suspicious API key and service principal use with concrete queries, and verifying the alerts against real log fields.
TL;DR — Pull your cloud audit logs into one queryable place, then alert on four things first: use from a new country/IP/ASN, impossible travel or sudden fan-out, privileged actions outside normal hours, and authentication method changes for service principals. The single most common miss is not preserving the original principal identifier and source IP fields during normalization, which makes every downstream detection noisy or blind. Reading time: ~5 min
Goal
When you finish, you will have audit logs from your cloud provider(s) in a local JSONL file or your SIEM, plus working detections that flag suspicious API key and service principal activity with concrete queries and a repeatable verification method.
Prerequisites
- Read access to your cloud audit logs for at least the last 7 days
- Permission to create alerts or scheduled searches in your log platform, or shell access to a workstation where you can run
jq >= 1.6— check withjq --version python >= 3.11for quick local analysis — check withpython3 --versionripgrep >= 14for fast field discovery — check withrg --version- A sample of raw audit events exported as newline-delimited JSON in
audit.jsonl - The exact field names for: principal ID, principal type, source IP, user agent, action name, result/status, timestamp, and target resource
- A list of known automation IP ranges, CI/CD runners, NAT egress IPs, and expected service principal names
Steps
Step 1: Inspect raw audit events and map the fields you actually have
Run these commands against your exported logs:
wc -l audit.jsonl
head -n 3 audit.jsonl | jq .
rg -n 'source[ipI]|client[ipI]|caller[ipI]|principal|servicePrincipal|accessKey|apiKey|userAgent|eventName|operationName|action' audit.jsonl | head -n 40
You should see real field names such as sourceIPAddress, callerIpAddress, userAgent, principalId, servicePrincipalId, eventName, or operationName in the output.
Step 2: Normalize the minimum useful fields into one schema
Create a normalized view locally first. Adjust the right-hand field names to match your provider logs exactly.
cat > normalize.py <<'python'
import json, sys
for line in sys.stdin:
e = json.loads(line)
out = {
"ts": e.get("eventTime") or e.get("time") or e.get("timestamp"),
"principal_id": e.get("principalId") or e.get("userIdentity", {}).get("principalId") or e.get("servicePrincipalId"),
"principal_type": e.get("principalType") or e.get("userIdentity", {}).get("type") or ("ServicePrincipal" if e.get("servicePrincipalId") else None),
"auth_type": e.get("authenticationType") or e.get("credentialType") or e.get("userIdentity", {}).get("accessKeyId"),
"src_ip": e.get("sourceIPAddress") or e.get("callerIpAddress") or e.get("clientIpAddress"),
"user_agent": e.get("userAgent") or e.get("callerUserAgent"),
"action": e.get("eventName") or e.get("operationName") or e.get("action"),
"result": e.get("errorCode") or e.get("resultType") or e.get("status"),
"target": e.get("resourceId") or e.get("requestParameters", {}).get("resource") or e.get("targetResourceName"),
"raw": e,
}
print(json.dumps(out))
python
python3 normalize.py < audit.jsonl > normalized.jsonl
head -n 2 normalized.jsonl | jq .
You should see one JSON object per line with ts, principal_id, principal_type, src_ip, action, and result populated.
Step 3: Build a baseline of known-good principals, IPs, and actions
Create allowlists from the last 7 days before you alert on anomalies.
jq -r 'select(.principal_id and .src_ip) | [.principal_id,.src_ip] | @tsv' normalized.jsonl | sort -u > known_principal_ip.tsv
jq -r 'select(.principal_id and .action) | [.principal_id,.action] | @tsv' normalized.jsonl | sort -u > known_principal_action.tsv
jq -r 'select(.principal_type == "ServicePrincipal" or .principal_type == "AssumedRole" or .principal_type == "Application") | .principal_id' normalized.jsonl | sort -u > service_principals.txt
printf '198.51.100.10\n203.0.113.0/24\n' > known_egress.txt
You should end up with flat files listing normal principal-to-IP and principal-to-action combinations.
Step 4: Detect first-seen IP use and suspicious fan-out
Run these exact checks locally. They translate directly into SIEM scheduled searches.
python3 - <<'python'
import json, collections, ipaddress
known = set()
with open('known_principal_ip.tsv') as f:
for line in f:
p, ip = line.rstrip('\n').split('\t')
known.add((p, ip))
counts = collections.defaultdict(set)
for line in open('normalized.jsonl'):
e = json.loads(line)
p, ip = e.get('principal_id'), e.get('src_ip')
if p and ip:
counts[p].add(ip)
if (p, ip) not in known:
print(f'NEW_IP\t{e.get("ts")}\t{p}\t{ip}\t{e.get("action")}')
for p, ips in counts.items():
if len(ips) >= 5:
print(f'FANOUT\t{p}\tunique_ips={len(ips)}')
python
You should see either no output or lines shaped like NEW_IP 2026-10-30T12:44:02Z spn-prod-deploy 45.83.12.9 CreateRoleAssignment.
Step 5: Detect privileged actions by service principals and API keys
Start with action-name matching. Tune later.
jq -r 'select((.principal_type == "ServicePrincipal" or .auth_type != null) and (.action | test("(?i)(create|delete|attach|detach|put|update|set|add|assign|grant|policy|role|token|secret|key)"))) | [.ts,.principal_id,.principal_type,.src_ip,.action,.result,.target] | @tsv' normalized.jsonl | column -t -s $'\t' | head -n 50
You should see a compact table of potentially sensitive actions; expected automation should be obvious from principal name and source IP.
Step 6: Detect auth changes and failed-then-success patterns
These are high-signal for stolen credentials and brute-force against non-human identities.
jq -r 'select(.principal_id) | [.principal_id,.ts,.src_ip,.action,.result,.auth_type] | @tsv' normalized.jsonl > timeline.tsv
python3 - <<'python'
from collections import defaultdict, deque
rows = defaultdict(list)
for line in open('timeline.tsv'):
p, ts, ip, action, result, auth = line.rstrip('\n').split('\t')
rows[p].append((ts, ip, action, result, auth))
for p, events in rows.items():
last_auth = None
fails = 0
for ts, ip, action, result, auth in events:
if auth and last_auth and auth != last_auth:
print(f'AUTH_CHANGE\t{ts}\t{p}\t{last_auth}->{auth}\t{ip}\t{action}')
last_auth = auth or last_auth
r = (result or '').lower()
if any(x in r for x in ['fail', 'denied', 'unauthorized', 'invalid', 'expired']):
fails += 1
elif fails >= 3 and ('success' in r or r == '' or r == 'none'):
print(f'FAIL_THEN_SUCCESS\t{ts}\t{p}\tprior_failures={fails}\t{ip}\t{action}')
fails = 0
python
You should see lines like AUTH_CHANGE 2026-10-30T03:11:09Z app-ci oldkey->federated 203.0.113.44 GetToken only when the credential type or auth path changed.
Step 7: Turn the local checks into scheduled detections
In your log platform, create scheduled searches every 5 minutes using the same logic as above. Minimum rules to create:
[
{"name":"cloud-nonhuman-new-ip","condition":"principal_type in [ServicePrincipal,Application,AssumedRole] AND src_ip is first_seen_for principal_id in 7d","severity":"high"},
{"name":"cloud-nonhuman-ip-fanout","condition":"unique(src_ip) by principal_id >= 5 in 1h","severity":"high"},
{"name":"cloud-nonhuman-privileged-action","condition":"principal_type in [ServicePrincipal,Application,AssumedRole] AND action matches /(create|delete|assign|grant|policy|role|secret|key|token)/i","severity":"high"},
{"name":"cloud-nonhuman-fail-then-success","condition":"3 failed auth or access events followed by success for same principal_id in 15m","severity":"medium"}
]
You should see four saved detections with a 5-minute schedule and a 7-day baseline window.
Verify it works
Use a known suspicious sample or replay a filtered subset of logs containing one event from a new IP and one privileged action.
jq 'select(.principal_id == "spn-prod-deploy")' normalized.jsonl | tail -n 20
printf '{"ts":"2026-10-30T12:44:02Z","principal_id":"spn-prod-deploy","principal_type":"ServicePrincipal","auth_type":"client_secret","src_ip":"45.83.12.9","user_agent":"python-requests/2.32","action":"CreateRoleAssignment","result":"Success","target":"/subscriptions/abc/resourceGroups/prod"}\n' >> normalized.jsonl
python3 normalize.py < audit.jsonl >/dev/null 2>&1; python3 - <<'python'
import json
known = set(tuple(line.rstrip('\n').split('\t')) for line in open('known_principal_ip.tsv'))
for line in open('normalized.jsonl'):
e = json.loads(line)
if (e.get('principal_id'), e.get('src_ip')) not in known and e.get('principal_id') == 'spn-prod-deploy':
print('ALERT_NEW_IP', e['principal_id'], e['src_ip'], e['action'])
python
Expected output shape:
ALERT_NEW_IP spn-prod-deploy 45.83.12.9 CreateRoleAssignment
If you are using a SIEM instead of local files, the equivalent proof is one alert with the principal ID, source IP, action, and raw event link present within one scheduler interval.
Common pitfalls
Missing original principal ID during normalization
Mistake: mapping everything to a display name and dropping the immutable principal identifier.
Symptom: one automation account appears as multiple identities, or multiple identities collapse into one noisy bucket.
Fix: keep the raw immutable field as principal_id and store display names separately.
Treating NAT egress changes as compromise
Mistake: alerting on any new source IP without allowlisting CI runners, VPN egress, and cloud NAT ranges.
Symptom: constant NEW_IP alerts after deploys or runner autoscaling.
Fix: add your known egress ranges to known_egress.txt and suppress those before alerting.
Using only success events
Mistake: filtering out denied or failed events to reduce volume.
Symptom: you miss password spray, expired-key retries, and fail-then-success sequences.
Fix: ingest both success and failure audit events and keep the provider error/status field intact.
Matching privileged actions too broadly
Mistake: alerting on every action containing get, list, or read.
Symptom: thousands of low-value alerts from normal inventory and health checks.
Fix: start with mutating verbs only: create|delete|update|set|assign|grant|policy|role|secret|key|token.
Ignoring user agent changes for non-human identities
Mistake: assuming service principals always have stable user agents but never checking them.
Symptom: a stolen key used from curl/8.x or python-requests blends in with normal API activity.
Fix: baseline user_agent per principal and alert on first-seen values, especially generic CLI and HTTP client strings.
This article was written by an AI system and published pending human review. Verify anything you intend to act on.
Have a project in mind?
Get an instant AI price estimate for it, or talk directly to our team.
One email a month on what we learn building with AI