Reduce SIEM ingestion cost with filtering, sampling, and retention tiers

TL;DR — SIEM bills usually explode because everything gets forwarded at the same fidelity and retention, including low-value noise. The highest-leverage fix is to drop or downsample noisy events before they hit the SIEM, keep only high-value security telemetry in hot searchable storage, and send the rest to cheap object storage with a clear rehydrate path. Reading time: ~7 min

What it is and where it sits

In XDR/SIEM pipelines, cost is mostly a function of three things: bytes ingested, how much gets indexed for fast search, and how long it stays in expensive storage. Controlling ingestion cost means changing the pipeline before data lands in the expensive part.

The three levers are:

• Filtering: drop events you know are low-value or duplicate.
• Sampling: keep only a percentage of repetitive events while preserving enough signal for detection and trend analysis.
• Tiered retention: keep recent/high-value data in hot searchable storage; move older or lower-value data to cheaper storage.

Architecture-wise, this sits between your log producers and the SIEM indexer/search tier. In a typical setup:

• apps, hosts, network devices, cloud audit logs, and identity systems emit logs
• an agent, collector, or message bus receives them
• a processing layer parses/enriches/routes them
• the SIEM indexes some subset
• object storage or archive keeps the rest

This often replaces the naive design of "every source forwards everything directly to the SIEM".

[Apps/Hosts] ---- 1. logs/events ----> [Collector]
[Cloud/IdP]  ---- 1. logs/events ----> [Collector]
[Network]    ---- 1. logs/events ----> [Collector]
[Collector]  ---- 2. parse/enrich ----> [Pipeline]
[Pipeline]   ---- 3a. keep full ----> [SIEM Hot Index]
[Pipeline]   ---- 3b. sample/drop --> [SIEM Hot Index]
[Pipeline]   ---- 3c. archive -----> [Object Storage]
[Analyst]    ---- 4. search recent -> [SIEM Hot Index]
[Analyst]    ---- 5. rehydrate ----> [Object Storage]

The key architectural point: the cheapest byte is the one you never ingest into the SIEM index. Compression helps. Better parsing helps. But dropping 70% of known-noise health checks before indexing helps more.

How it actually works

A practical mechanism looks like this:

1. Classify events early by source, event type, severity, and use case.
2. Apply hard filters for events with no security or operational value.
3. Apply deterministic sampling to repetitive low-risk events.
4. Route by class to hot SIEM storage or archive.
5. Set different retention windows by class.
6. Measure what you dropped so you can reverse bad decisions.

End-to-end example: Kubernetes ingress logs flooding the SIEM

Assume your nginx ingress emits one access log per request. Most are 200/304 responses for static assets and health checks. Security-relevant events are rare: 401/403 spikes, admin path hits, odd user agents, request smuggling probes, and bursts from a single IP.

Without controls:

• 500 GB/day of ingress logs go straight to the SIEM
• 95% are routine 200/304 responses
• detections mostly care about 4xx/5xx, auth endpoints, admin paths, and anomaly aggregates

A better pipeline:

1. Collector receives raw logs from Fluent Bit / Vector / Filebeat on the cluster nodes.
2. Parser extracts fields: status, path, method, bytes, upstream time, remote IP, user agent.
3. Filter stage drops obvious noise:
   • /healthz, /readyz, /metrics
   • static assets like .css, .js, .png, .ico
   • internal load balancer health checks by known user agent or source CIDR
4. Sampling stage keeps only 1 in 100 of remaining status < 400 events, but keeps all 4xx/5xx.
5. Routing stage sends:
   • all 401/403/404/429/5xx and admin/auth paths to SIEM hot index for 30 days
   • sampled 2xx/3xx to a low-cost analytics store or short-retention index for 3 days
   • full raw logs to compressed object storage for 180 days
6. Metrics stage emits counters for dropped and sampled events by rule.

Now a brute-force attempt hits /admin/login and returns many 401s from one IP. Those events are unsampled and retained hot, so the SIEM still detects it. Routine /assets/app.js 200 responses are mostly dropped or sampled, so they stop dominating the bill.

The subtle but important part is deterministic sampling. If you sample randomly per event, repeated analysis can become inconsistent. If you sample based on a stable key like hash(client_ip + path + minute) % 100 == 0, the same class of traffic is represented consistently enough for trend work.

Also note what you should not sample:

• authentication success/failure events
• privilege changes
• cloud control-plane audit logs
• endpoint security alerts
• DNS logs if you use them for exfiltration detection and rare-domain analytics
• anything already low-volume and high-signal

When to use it (and when not to)

Use this when your SIEM bill is driven by machine-generated noise, not by genuinely high-value security telemetry.

You probably don’t need this if your problem is actually bad retention defaults, duplicate forwarding, or parsing errors causing event explosion. Check those first.

Concrete checks:

# Count duplicate lines from a local test file before blaming volume on "normal traffic"
sort ingress.log | uniq -c | sort -nr | head

# Estimate top talkers by path from JSON logs
jq -r '.path' ingress.json | sort | uniq -c | sort -nr | head -20

# Find likely duplicate forwarding from two agents on the same host
sudo ss -tpn | grep -E '(:5044|:24224|:9000|:9200)'

If you discover two shippers tailing the same file, fix that before designing a sampling strategy.

Trade-offs

Every cost win here comes with a price.

• Benefit: lower SIEM spend
  Cost: more pipeline logic to own. Someone must version, test, and review drop rules.

• Benefit: faster searches in hot storage
  Cost: analysts may need to rehydrate archived data during investigations, which adds delay.

• Benefit: less noise in detections
  Cost: a bad filter can remove weak early signals. This is especially risky for recon and low-and-slow attacks.

• Benefit: longer overall retention via cheap object storage
  Cost: cold data is slower to query, may need format conversion, and often lacks the same enrichment/indexing.

• Benefit: sampling preserves trend visibility with less volume
  Cost: exact counts become estimates unless you carry a sample rate field and adjust queries.

• Benefit: source-specific routing avoids paying premium index rates for everything
  Cost: more vendor and format coupling in the pipeline. Migrating later is harder if routing logic is embedded in proprietary transforms.

A good rule: drop only what you can explain, sample only what you can statistically tolerate, archive anything you may later need for legal or IR reasons.

In practice

Example 1: Vector pipeline for filtering, sampling, and archiving

[sources.k8s_ingress]
type = "file"
include = ["/var/log/containers/*ingress*.log"]
read_from = "beginning"

[transforms.parse_json]
type = "remap"
inputs = ["k8s_ingress"]
source = '''
. = parse_json!(string!(.message))
.path = string!(.path)
.status = to_int!(.status)
.ua = string!(.user_agent)
.client_ip = string!(.remote_addr)
'''

[transforms.drop_noise]
type = "filter"
inputs = ["parse_json"]
condition = '''
!(starts_with(.path, "/healthz") ||
  starts_with(.path, "/readyz") ||
  starts_with(.path, "/metrics") ||
  match(.path, r'\.(css|js|png|jpg|ico|svg)$') ||
  contains(.ua, "kube-probe"))
'''

[transforms.sample_success]
type = "remap"
inputs = ["drop_noise"]
source = '''
.sample_rate = 1
if .status < 400 {
  key = .client_ip + .path + format_timestamp!(now(), "%Y-%m-%dT%H:%M")
  if to_int!(slice!(sha1(key), 0, 2), 16) % 100 != 0 { abort }
  .sample_rate = 100
}
'''

[sinks.siem]
type = "http"
inputs = ["sample_success"]
uri = "https://siem.example.internal/ingest"
encoding.codec = "json"

[sinks.archive]
type = "aws_s3"
inputs = ["parse_json"]
bucket = "sec-log-archive"
key_prefix = "ingress/date=%F/"
compression = "gzip"
encoding.codec = "json"

This keeps all parsed ingress logs for archive, drops known noise before SIEM ingest, and samples successful requests deterministically. The gotcha: abort drops the event entirely from downstream transforms, so wire your archive sink before the sampling transform if you want full raw retention.

Example 2: Fluent Bit grep + rewrite-tag for hot vs cold routing

[INPUT]
    Name              tail
    Path              /var/log/nginx/access.log
    Tag               nginx.access
    Parser            nginx_json

[FILTER]
    Name              grep
    Match             nginx.access
    Exclude           path ^/healthz$
    Exclude           path ^/readyz$
    Exclude           user_agent kube-probe

[FILTER]
    Name              rewrite_tag
    Match             nginx.access
    Rule              $status ^(401|403|404|429|5\d\d)$ hot.$TAG true
    Rule              $path ^/(admin|login|oauth|api/auth) hot.$TAG true
    Rule              $status ^(2\d\d|3\d\d)$ cold.$TAG true

[OUTPUT]
    Name              es
    Match             hot.nginx.access
    Host              siem-indexer.internal
    Port              9200
    Index             sec-nginx-hot
    Logstash_Format   On

[OUTPUT]
    Name              s3
    Match             cold.nginx.access
    bucket            sec-log-archive
    region            us-east-1
    total_file_size   50M
    upload_timeout    5m
    compression       gzip

This routes security-relevant HTTP events to a searchable index and routine traffic to S3-compatible archive. The gotcha: rewrite_tag can duplicate records if you keep the original tag; check the final boolean on each Rule and verify you are not sending the same event to both outputs unintentionally.

What to monitor after rollout

After changing the pipeline, look at actual event rates and failures, not just the invoice.

# Example shape when an HTTP sink starts rejecting oversized batches
curl -sS -D - -o /dev/null https://siem.example.internal/ingest

HTTP/1.1 413 Payload Too Large
content-type: application/json
retry-after: 30
x-request-id: 7f2b8e1c9d

A common failure mode is that filtering reduced event count but batching increased payload size, causing 413 Payload Too Large or 429 Too Many Requests. Fix by lowering batch size or flush interval in the shipper, not by disabling filters.

Also track counters like:

• events received per source
• events dropped per rule
• events sampled per rule
• archive write failures
• SIEM ingest rejects (400, 413, 429, 5xx)
• rehydrate requests and time-to-query for archived data

⚠️ Dropping or sampling logs can permanently remove evidence. Before enabling a new rule, deploy it in "count-only" mode if your pipeline supports that, or clone the stream to archive first and compare one week of detections and incident workflows.

Further reading

• OpenTelemetry Logs Data Model
• Fluent Bit Manual — Filters and Routing
• Vector Documentation — Transforms and Sinks
• NIST SP 800-92 Guide to Computer Security Log Management
• The "Logging" and "Observability" chapters of Google Site Reliability Engineering

This article was written by an AI system and published pending human review. Verify anything you intend to act on.