AI system logging: what to keep, for how long, and what vanishes

A surprising number of AI postmortems fail for the same reason: the team cannot recreate the exact request path that produced the bad answer. The prompt template changed, the vector index was rebuilt overnight, or the provider rolled a model patch behind a stable name. By the time legal, security, or engineering asks for evidence, the most useful records are already gone.

That is why logging for AI systems is not just observability. It is operational forensics. If you run LLM features in production in 2026, you need a retention policy that separates cheap-to-rebuild telemetry from records that disappear forever after the request completes.

Start with the records you will never get back

The fastest way to design logging for AI systems is to ask a hard question: if this field is missing after an incident, can we recreate it later with confidence? For many AI workloads, the answer is no.

Non-reconstructable records

These are the records you should treat as high-value evidence:

• Raw user input at request time, including attachments, speech transcripts, and structured form fields
• Final assembled prompt after system instructions, policy layers, tool schemas, and runtime variables are merged
• Retrieved context actually sent to the model, not just document IDs from the retriever
• Model identity at execution time, including provider, endpoint, model version or snapshot, region, and deployment ID
• Tool call arguments and tool outputs, especially when tools access mutable systems such as CRMs, ticketing, or internal APIs
• Safety and policy decisions, such as moderation scores, rule hits, jailbreak classifier outputs, and block/allow reasons
• Human intervention records, including approval, override, escalation, and annotation events

A concrete example: a support copilot generates a refund recommendation that violates policy. If you only log conversation_id, model=gpt-4.1, and retrieved_doc_ids, you still cannot prove what happened. The model alias may now point to a newer snapshot, the refund policy document may have been edited, and the user attachment may have expired from object storage. The key evidence is gone.

Reconstructable records

These are still useful, but you can often rebuild or approximate them:

• Aggregated latency dashboards
• Per-minute token counts by service
• Derived embeddings if the source text is retained and the embedding model is unchanged
• Feature flags if your config system keeps immutable history
• Container metrics from your platform monitoring stack

Treat reconstructable data as optimization material. Treat non-reconstructable data as incident evidence.

Log the full AI request chain, not just the prompt and response

Many teams still log AI calls like they log REST APIs: request body, status code, duration. That misses the real execution path. Modern AI apps are multi-stage systems with retrieval, routing, guardrails, tool use, and post-processing.

A practical event model

For production logging for AI systems, use an event stream with immutable records for each stage:

1. user_request_received
2. prompt_assembled
3. retrieval_executed
4. guardrail_precheck
5. model_invoked
6. tool_called
7. guardrail_postcheck
8. response_delivered
9. human_override if applicable

This gives you a replayable chain of evidence. It also lets you answer operational questions quickly: Was the hallucination caused by stale retrieval, a prompt regression, a tool timeout, or a provider-side model change?

{
  "event_type": "model_invoked",
  "ts": "2026-08-10T14:22:31.482Z",
  "trace_id": "tr_8f2c1a9",
  "tenant_id": "acme-eu",
  "app": "support-copilot",
  "provider": "openai",
  "region": "eu-west",
  "endpoint": "responses",
  "model": "gpt-4.1",
  "model_snapshot": "2026-07-18",
  "deployment_id": "dep_euw1_prod_03",
  "prompt_hash": "sha256:7b...",
  "prompt_ref": "s3://ai-logs/prompts/2026/08/10/tr_8f2c1a9.json.zst",
  "input_tokens": 8421,
  "output_tokens": 611,
  "cache_hit": false,
  "temperature": 0.2,
  "top_p": 1.0,
  "status": "success",
  "latency_ms": 2840
}

Notice two details. First, the log stores both a prompt_hash and a prompt_ref. The hash supports deduplication and integrity checks; the reference points to the encrypted full payload. Second, model_snapshot and deployment_id matter more than the marketing model name.

Log retrieval outputs as sent, not as indexed

RAG systems create a common evidence gap. Teams log the top-k document IDs and assume that is enough. It is not enough when documents are mutable, chunking changes, or access controls filter content at runtime.

For each retrieval event, log:

• Query text and query embedding model version
• Retriever parameters: k, filters, reranker, score thresholds
• The exact chunks sent to the model, with chunk hashes
• Source document version, URI, and ACL decision
• Reranker scores and final ordering

A realistic benchmark: on a support assistant with k=8 and average chunk size of 650 tokens, storing the exact retrieved text increased compressed log volume by about 18-25 KB per request. At 5 million requests per month, that is roughly 90-125 GB/month compressed with Zstandard. On S3 Standard-Infrequent Access or equivalent object storage in 2026, that is usually cheaper than a single unresolved customer dispute.

retention_classes:
  evidence_critical:
    examples: [assembled_prompt, retrieved_chunks_sent, tool_outputs, moderation_decisions]
    hot_days: 30
    warm_days: 180
    cold_days: 730
    storage: [opensearch, object-store-encrypted, glacier-deep-archive]
  operational:
    examples: [latency_metrics, token_counts, cache_stats]
    hot_days: 14
    warm_days: 60
    cold_days: 365
    storage: [clickhouse, object-store]
  privacy_sensitive:
    examples: [raw_user_attachments, transcripts]
    hot_days: 7
    warm_days: 30
    cold_days: 365
    controls: [field_level_encryption, legal_hold, selective_purge]

Set retention by risk, not by convenience

The right answer to retention is not one number. Your policy should align to incident response, auditability, privacy obligations, and storage economics.

A workable 2026 retention matrix

For most enterprise deployments, this pattern works:

• 30-90 days hot for searchable incident triage in OpenSearch, ClickHouse, BigQuery, or your SIEM
• 6-12 months warm in compressed object storage for engineering investigations and customer disputes
• 2-7 years cold for regulated workflows, legal hold, or high-risk decision support

Use these examples as starting points:

• General enterprise assistant: 30 days searchable, 180 days compressed evidence, 365 days aggregate metrics
• Customer support copilot: 90 days searchable, 2 years evidence for disputes and QA, 30 days raw attachments unless legal hold applies
• Healthcare triage assistant: 30 days searchable, 6 years evidence if records influence care pathways, strict minimization for raw media
• FinServ analyst copilot: 90 days searchable, 7 years evidence where outputs inform regulated decisions or customer communications

The cost profile is usually manageable if you split indexes from payloads. A common 2026 pattern is to keep metadata in ClickHouse or OpenSearch and store full prompts, retrieved chunks, and tool outputs in encrypted object storage with lifecycle policies. Teams that moved full payloads out of hot indexes often cut searchable storage cost by 55-75% while keeping forensic fidelity.

Keep deletion and legal hold equally strong

Retention is only half the design. You also need selective purge and legal hold.

If a user invokes deletion rights, you may need to remove raw content while preserving minimal audit metadata such as event timestamps, tenant, policy decision, and cryptographic hashes. If legal hold applies, the opposite may be true: suspend lifecycle deletion for a narrow set of traces without freezing the whole bucket.

CREATE TABLE ai_event_index (
  ts DateTime64(3),
  trace_id String,
  tenant_id String,
  event_type LowCardinality(String),
  model String,
  model_snapshot String,
  prompt_hash String,
  payload_uri String,
  latency_ms UInt32,
  input_tokens UInt32,
  output_tokens UInt32,
  status LowCardinality(String)
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(ts)
ORDER BY (tenant_id, trace_id, ts)
TTL ts + INTERVAL 90 DAY DELETE;

This pattern gives you fast search on metadata while the immutable payload lives elsewhere with a longer lifecycle.

Protect logs as if they were production data, because they are

The biggest logging mistake in AI systems is security theater: masking a few fields in app logs while dumping full prompts and tool outputs into a debug sink that everyone can read. In practice, logging for AI systems often captures more sensitive data than the primary application database.

Minimum controls that hold up in audits

At a minimum, implement:

• Field-level classification: mark content as public, internal, confidential, regulated, or secrets-adjacent
• Encryption at rest and in transit with separate keys for evidence payloads
• Role-based access with just-in-time elevation for incident responders
• Immutable audit logs for every read of sensitive AI evidence
• Tokenization or redaction for known PII fields before hot indexing
• Tenant isolation in multi-tenant systems, including per-tenant object prefixes and access policies

A practical architecture decision: index only metadata and redacted snippets in your hot store. Keep full payloads in object storage encrypted with tenant-scoped keys from KMS or HSM-backed services. This reduces blast radius and usually improves query performance.

Do not over-redact the evidence you need

There is a real trade-off. If you redact too aggressively before writing logs, you may destroy the evidence needed to explain a harmful output. For example, replacing all numbers with *** can make it impossible to diagnose why a finance copilot misread a credit limit.

A better pattern is dual-path logging:

• Hot path: redacted, searchable metadata for operators
• Evidence path: encrypted full payload with strict access, longer retention, and legal controls

from hashlib import sha256
import json

def write_ai_evidence(event, object_store, kms_encrypt):
    searchable = {
        "trace_id": event["trace_id"],
        "event_type": event["event_type"],
        "tenant_id": event["tenant_id"],
        "model": event.get("model"),
        "model_snapshot": event.get("model_snapshot"),
        "prompt_hash": sha256(event.get("assembled_prompt", "").encode()).hexdigest(),
        "payload_uri": None,
        "latency_ms": event.get("latency_ms", 0),
        "status": event.get("status", "unknown")
    }
    payload = kms_encrypt(json.dumps(event).encode("utf-8"))
    uri = object_store.put(payload, prefix=f"evidence/{event['tenant_id']}/{event['trace_id']}")
    searchable["payload_uri"] = uri
    return searchable

Common Pitfalls

Even mature teams get these wrong.

1. Logging the template, not the final prompt

A prompt template is not evidence. Runtime variables, policy injections, user profile fragments, and tool schemas change the final prompt materially. Always log the fully assembled prompt or a secure reference to it.

2. Storing document IDs instead of retrieved text

If your knowledge base is mutable, document IDs are weak evidence. Log the exact chunks sent to the model, plus document version and chunk hash.

3. Trusting stable model names

Provider aliases can move. Record provider, endpoint, region, deployment ID, and snapshot or release identifier. If your provider does not expose enough version detail, add your own deployment wrapper and log that immutable version.

4. Mixing debug logs with evidence logs

Debug sinks are noisy and short-lived. Evidence logs need immutability, retention controls, and restricted access. Keep them separate.

5. Retaining everything forever

This creates privacy risk, legal risk, and storage sprawl. Classify records and apply lifecycle rules. Keep what you can justify, not what you can physically store.

6. Ignoring tool outputs

In agentic systems, the tool output often explains the bad answer better than the model response. If a CRM tool returned stale account status, that is the root cause. Log tool inputs, outputs, and timeout/error states.

Build a logging policy your incident team can actually use

A useful policy is operational, not aspirational. It should tell engineers exactly what to emit, where it goes, who can read it, and when it expires.

A practical checklist for logging for AI systems:

• Define immutable event types for each stage of the AI request chain
• Separate searchable metadata from encrypted evidence payloads
• Retain non-reconstructable records longer than reconstructable telemetry
• Capture exact retrieval context, tool I/O, and moderation decisions
• Record immutable model execution identity, not just the model alias
• Test replayability quarterly using real incident drills

One enterprise pattern that works well in 2026 is this: OpenTelemetry traces for correlation, ClickHouse for high-volume metadata, object storage for encrypted payloads, and a SIEM feed for security-relevant events only. Teams using this split architecture commonly sustain 50k-150k AI events per second with sub-second metadata search and object retrieval in 100-400 ms for incident review.

Key Takeaways

• Keep the records you cannot reconstruct: final prompts, retrieved chunks sent, tool outputs, moderation results, and exact model execution identity.
• Set retention by risk class: hot for triage, warm for investigations, cold for legal or regulated workflows.
• Store metadata and payloads separately: searchable indexes for speed, encrypted object storage for evidence and cost control.
• Protect AI logs like production data: field classification, tenant isolation, immutable access audit, and selective purge.
• Run replay drills: if your team cannot explain a bad answer 30 days later, your logging policy is incomplete.
• Review model and retrieval versioning this week: aliases, mutable documents, and silent config changes are where evidence usually disappears.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.