Build an incident timeline from app, identity, and cloud logs

TL;DR — Pull the smallest relevant slice of logs first, convert every timestamp to UTC ISO-8601, then join records on three fields in this order: request/correlation ID, actor identity, and source IP. The most common reason timelines are wrong is mixed timezones or clock skew, so normalize time before you correlate anything. Reading time: ~5 min

Goal

When you finish, you will have a single chronologically sorted timeline.ndjson and timeline.csv built from application logs, identity-provider sign-in/audit logs, and cloud control-plane/audit logs for one incident window, with enough fields to answer who did what, from where, and in what order.

Prerequisites

• Access to the affected app logs: SSH, Kubernetes access, log aggregator export, or local files
• Access to identity logs from your provider's dashboard or API export for sign-in/audit events
• Access to cloud audit logs from your cloud provider's dashboard or CLI export
• jq >= 1.6 — check with jq --version
• python >= 3.11 — check with python3 --version
• GNU coreutils (sort, awk, sed, grep) — check with sort --version
• sqlite3 >= 3.40 optional but useful for ad-hoc joins — check with sqlite3 --version
• The incident time window in UTC if you have it; otherwise the first known bad event time and timezone
• At least one pivot value: user email, user ID, request ID, trace ID, session ID, source IP, hostname, or cloud resource ID

Steps

Step 1: Create a working folder and define the incident window

mkdir -p ~/ir/case-2026-10-30/{raw,normalized,output}
cd ~/ir/case-2026-10-30
export START_UTC="2026-10-30T08:00:00Z"
export END_UTC="2026-10-30T12:00:00Z"
printf '%s\n%s\n' "$START_UTC" "$END_UTC"

You should see the exact UTC start and end timestamps printed on separate lines.

Step 2: Export the three log sources into raw/

Use the literal export path your environment supports. Keep the filenames distinct.

Application logs from local files or a container export:

cp /var/log/myapp/app.log raw/app.log
# or
kubectl logs deploy/myapp -n prod --since=6h --timestamps=true > raw/app.log

Identity logs from your provider's dashboard or API export:

# Save the export as NDJSON or JSON array
# Example resulting file name:
ls -lh raw/idp.json

Cloud audit logs from your provider's dashboard or CLI export:

# Save the export as NDJSON or JSON array
# Example resulting file name:
ls -lh raw/cloud.json

You should have at least one app log file and two JSON exports visible under raw/.

Step 3: Normalize timestamps and fields into a common NDJSON shape

Create a normalizer script that emits one JSON object per line with consistent keys.

cat > normalize.py <<'PY'
import json, sys, re
from datetime import datetime, timezone

ISO_KEYS = ["timestamp","time","ts","createdAt","eventTime","event_time"]
ACTOR_KEYS = ["user","userId","user_id","email","principalEmail","actor"]
IP_KEYS = ["ip","ipAddress","client_ip","sourceIPAddress","source_ip"]
RID_KEYS = ["request_id","requestId","trace_id","traceId","correlationId","correlation_id","session_id"]
ACTION_KEYS = ["action","eventName","operation","event_type","type"]
RESOURCE_KEYS = ["resource","resourceName","target","resource_id"]


def to_iso8601(v):
    if v is None:
        return None
    if isinstance(v, (int, float)):
        if v > 1e12:
            v = v / 1000.0
        return datetime.fromtimestamp(v, tz=timezone.utc).isoformat().replace('+00:00','Z')
    s = str(v).strip()
    if s.endswith('Z'):
        return s
    for fmt in ["%Y-%m-%dT%H:%M:%S.%f%z","%Y-%m-%dT%H:%M:%S%z","%Y-%m-%d %H:%M:%S%z","%Y-%m-%d %H:%M:%S"]:
        try:
            dt = datetime.strptime(s, fmt)
            if dt.tzinfo is None:
                dt = dt.replace(tzinfo=timezone.utc)
            return dt.astimezone(timezone.utc).isoformat().replace('+00:00','Z')
        except ValueError:
            pass
    return s


def pick(d, keys):
    for k in keys:
        if k in d and d[k] not in (None, ""):
            return d[k]
    return None


def emit(source, rec):
    out = {
        "ts": to_iso8601(pick(rec, ISO_KEYS)),
        "source": source,
        "actor": pick(rec, ACTOR_KEYS),
        "ip": pick(rec, IP_KEYS),
        "request_id": pick(rec, RID_KEYS),
        "action": pick(rec, ACTION_KEYS),
        "resource": pick(rec, RESOURCE_KEYS),
        "raw": rec,
    }
    print(json.dumps(out, separators=(",",":"), ensure_ascii=False))

src, path = sys.argv[1], sys.argv[2]
with open(path, 'r', encoding='utf-8') as f:
    first = f.read(1)
    f.seek(0)
    if first == '[':
        for rec in json.load(f):
            emit(src, rec)
    elif path.endswith('.json') or path.endswith('.ndjson'):
        for line in f:
            line = line.strip()
            if line:
                emit(src, json.loads(line))
    else:
        for line in f:
            m = re.match(r'^(\S+)\s+(.*)$', line.strip())
            if not m:
                continue
            ts, msg = m.groups()
            rec = {"timestamp": ts, "message": msg}
            for key, rx in {
                "request_id": r'(?:request_id|trace_id|correlation_id)=([^\s]+)',
                "user": r'(?:user|email)=([^\s]+)',
                "ip": r'(?:ip|client_ip)=([^\s]+)',
                "action": r'(?:action|event)=([^\s]+)'
            }.items():
                mm = re.search(rx, msg)
                if mm:
                    rec[key] = mm.group(1)
            emit(src, rec)
PY
python3 normalize.py app raw/app.log > normalized/app.ndjson
python3 normalize.py idp raw/idp.json > normalized/idp.ndjson
python3 normalize.py cloud raw/cloud.json > normalized/cloud.ndjson
wc -l normalized/*.ndjson

You should see non-zero line counts for each normalized file.

Step 4: Filter to the incident window and sort everything together

jq -c --arg s "$START_UTC" --arg e "$END_UTC" 'select(.ts >= $s and .ts <= $e)' normalized/*.ndjson | sort -t '"' -k4,4 > output/timeline.ndjson
head -3 output/timeline.ndjson | jq .

You should see the first few records with a uniform shape containing ts, source, actor, ip, request_id, and action.

Step 5: Pivot on the best available identifier and expand around it

Start with request ID if you have one; otherwise actor, then IP.

export PIVOT_REQ="req-7f3a2c1b"
export PIVOT_ACTOR="alice@example.com"
export PIVOT_IP="203.0.113.24"

jq -c --arg r "$PIVOT_REQ" 'select(.request_id == $r)' output/timeline.ndjson > output/pivot-request.ndjson
jq -c --arg a "$PIVOT_ACTOR" 'select(.actor == $a)' output/timeline.ndjson > output/pivot-actor.ndjson
jq -c --arg i "$PIVOT_IP" 'select(.ip == $i)' output/timeline.ndjson > output/pivot-ip.ndjson

wc -l output/pivot-*.ndjson

You should see at least one pivot file with a non-zero count; that is your anchor for the timeline.

Step 6: Produce a human-readable CSV for sharing and a quick event flow

jq -r '[.ts,.source,.actor,.ip,.request_id,.action,.resource] | @csv' output/timeline.ndjson > output/timeline.csv
column -s, -t < output/timeline.csv | sed -n '1,10p'

You should see a readable table-like preview with rows in chronological order.

[User] ---- 1. sign-in ----> [IdP]
  ^                             |
  | 4. app request              | 2. token issued
  |                             v
[App]  ---- 3. API call ----> [Cloud Control Plane]

Step 7: Check for clock skew before trusting the sequence

If app events appear before the identity event that should have caused them, compare neighboring records by actor or request ID.

jq -c 'select(.actor == "'$PIVOT_ACTOR'" or .request_id == "'$PIVOT_REQ'") | {ts,source,actor,request_id,action}' output/timeline.ndjson | sed -n '1,20p'

You should see a plausible order; if one source is consistently early or late by the same offset, note the skew in your incident notes and adjust your interpretation.

Verify it works

Run these checks end to end:

test -s output/timeline.ndjson && echo "timeline.ndjson exists"
test -s output/timeline.csv && echo "timeline.csv exists"
jq -r '.source' output/timeline.ndjson | sort | uniq -c
jq -r 'select(.request_id != null) | .request_id' output/timeline.ndjson | head
jq -r '.[0:0]' output/timeline.ndjson 2>/dev/null; echo $?

Expected results:

• timeline.ndjson exists and timeline.csv exists
• uniq -c shows records from app, idp, and cloud
• head prints at least one request ID if your logs contain one
• The final exit code is 0; if jq prints parse error: Invalid numeric literal or similar, one of the inputs was not valid NDJSON/JSON and needs re-exporting or line-by-line normalization

Common pitfalls

Timestamps mixed between local time and UTC

Mistake: app logs are local time while identity/cloud exports are UTC. Symptom: events appear to happen before their cause by exactly your server offset, often +0100, -0500, or similar. Fix: rerun normalization with explicit UTC conversion and only compare ts values ending in Z.

Sorting raw JSON lines instead of normalized ts

Mistake: using sort raw/*.json or sorting on the wrong field. Symptom: the timeline jumps backward because lexical order of original records is not chronological. Fix: sort only output/timeline.ndjson after every record has a single ts field in ISO-8601 UTC.

Joining on IP alone behind NAT or a proxy

Mistake: treating source IP as a unique actor identifier. Symptom: unrelated users collapse into one timeline, especially from office egress IPs, VPNs, or load balancers. Fix: pivot on request_id first, then actor identity, and use IP only as supporting evidence.

Export window too narrow

Mistake: pulling exactly the reported minute of impact. Symptom: you miss the sign-in, role change, token issuance, or resource creation that happened 10-30 minutes earlier. Fix: re-export at least 1 hour before first observed impact and 1 hour after containment.

JSON array vs NDJSON confusion

Mistake: feeding a JSON array export directly into line-oriented tools. Symptom: jq errors or wc -l shows 1 for a file containing many events. Fix: pass the file through normalize.py; it handles both JSON arrays and NDJSON.

Missing correlation IDs in app logs

Mistake: the application never logged request or trace IDs. Symptom: you can see the suspicious actor in identity logs and cloud logs but cannot prove which app request triggered the action. Fix: for future incidents, add request ID logging at ingress and propagate it through app, worker, and outbound API calls; for the current case, correlate on actor plus tight time window plus source IP and document the weaker confidence.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.