SIEM ingestion stopped: diagnose and fix a broken log pipeline

TL;DR — When logs stop arriving in the SIEM, the fastest path is: confirm the source is still producing logs, check the collector/forwarder service state, then test network/TLS/auth to the ingest endpoint with the exact host, port, and token the agent uses. The most common fix is a broken collector config or expired credentials after a deploy, restart, or secret rotation. Reading time: ~6 min

The scenario

It is 14:37 on a Tuesday. You shipped a routine app deploy an hour ago, the app is serving traffic, but your XDR/SIEM dashboard has gone flat for one service and now detections tied to that log source are silent. Infra graphs look mostly normal, the node is reachable over SSH, and the app's local log file is still growing. Security is asking whether telemetry is gone globally or just delayed, and you need to answer before the incident channel turns into a blame thread.

Symptoms

• SIEM shows no new events from one host, one cluster, or one source type for several minutes to hours.
• Collector/forwarder service is unhealthy or restarting:

$ systemctl status vector
● vector.service - Vector
     Loaded: loaded (/lib/systemd/system/vector.service; enabled)
     Active: activating (auto-restart) (Result: exit-code) since Tue 2026-10-30 14:31:02 UTC; 3s ago
    Process: 18422 ExecStart=/usr/bin/vector --config /etc/vector/vector.yaml (code=exited, status=78)

• Journal shows config or auth failures:

$ journalctl -u fluent-bit -n 20 --no-pager
[2026/10/30 14:28:11] [error] [config] invalid value for tls.verify: tru
[2026/10/30 14:28:11] [error] configuration file contains errors, aborting.

$ journalctl -u vector -n 20 --no-pager
Oct 30 14:29:44 app-01 vector[20131]: ERROR sink{component_kind="sink" component_id=siem}: request failed. status=401 Unauthorized body="invalid token"

• Network path errors from the collector host:

$ nc -vz siem-ingest.example.net 443
nc: connect to siem-ingest.example.net port 443 (tcp) failed: Connection timed out

• TLS handshake/certificate errors:

$ openssl s_client -connect siem-ingest.example.net:443 -servername siem-ingest.example.net </dev/null
verify error:num=20:unable to get local issuer certificate
140735229261632:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889:

• HTTP endpoint reachable but wrong path/redirect/proxy behavior:

$ curl -I https://siem-ingest.example.net/collector
HTTP/2 301
location: http://siem-ingest.example.net/collector/
server: nginx

• Local queue/backpressure growing while upstream is unavailable:

$ du -sh /var/lib/vector/
8.1G    /var/lib/vector/

• App logs exist locally, but lines are missing after a logrotate/deploy because the collector is tailing the wrong file or inode.
• Filters/parsers silently dropping records after a format change:

Oct 30 14:33:10 app-01 vector[20211]: WARN transform{component_id=parse_json}: failed to parse JSON: expected value at line 1 column 1; dropping event

Likely causes

Step-by-step diagnosis

1. Confirm the source is still producing logs.

tail -n 20 /var/log/app/app.log
# or for containerized apps
kubectl logs deploy/myapp -n prod --since=10m | tail -n 20

If the file/stream is not moving, this is not a SIEM pipeline issue; fix the app logging path first. If logs are present locally, continue.

1. Check the collector/forwarder service state.

systemctl status fluent-bit vector filebeat rsyslog --no-pager
journalctl -u fluent-bit -u vector -u filebeat -u rsyslog -n 50 --no-pager

If you see status=78, configuration file contains errors, crash loops, or Active: failed, this is your problem. Jump to Fixes → Collector/forwarder service stopped or config invalid.

1. Test DNS and TCP reachability from the collector host.

getent hosts siem-ingest.example.net
nc -vz siem-ingest.example.net 443

If DNS does not resolve, or nc times out/refuses, this is your problem. Jump to Fixes → Network path blocked.

1. Test TLS and HTTP behavior against the exact ingest URL.

openssl s_client -connect siem-ingest.example.net:443 -servername siem-ingest.example.net </dev/null | egrep 'Verify return code|subject=|issuer='
curl -vkI https://siem-ingest.example.net/collector

If Verify return code is not 0 (ok), or curl shows an unexpected 301 to http://, 404, 405, or a proxy banner, jump to Fixes → Wrong endpoint/path/protocol or Fixes → Auth/token/certificate expired or rotated depending on the error.

1. Check for auth failures in agent logs.

journalctl -u vector -u fluent-bit -u filebeat -n 100 --no-pager | egrep -i '401|403|unauthorized|forbidden|token|api key|certificate'

If you see 401 Unauthorized, 403 Forbidden, invalid token, or client cert errors, jump to Fixes → Auth/token/certificate expired or rotated.

1. Check local queue and disk pressure.

df -h
du -sh /var/lib/vector /var/lib/fluent-bit /var/lib/filebeat 2>/dev/null

If the queue directory is growing fast, disk is >90%, or the agent logs mention backpressure/retries, jump to Fixes → Local queue full / disk pressure / backpressure.

1. Check source paths, tail offsets, and parser/filter drops.

journalctl -u vector -u fluent-bit -u filebeat -n 100 --no-pager | egrep -i 'drop|parse|decode|tail|inode|rotate|offset'

If you see parse failures, dropped events, or tailing the wrong file after rotation/deploy, jump to Fixes → Source path, parser, or filter changed.

[App Logs] ---- 1. tail/read ----> [Collector]
    |                                  |
    v                                  v
[Local File] ---- 2. parse/filter --> [Queue]
                                       |
                                       v
[DNS/Firewall/Proxy] ---- 3. TLS/HTTP -> [SIEM Ingest]

Fixes

Collector/forwarder service stopped or config invalid

Validate the config before restarting.

# Vector
vector validate --config /etc/vector/vector.yaml
# Fluent Bit
fluent-bit --dry-run -c /etc/fluent-bit/fluent-bit.conf
# Filebeat
filebeat test config -c /etc/filebeat/filebeat.yml

Fix the reported line/field, then restart:

systemctl restart vector
systemctl restart fluent-bit
systemctl restart filebeat
systemctl status vector fluent-bit filebeat --no-pager

Common examples:

# /etc/vector/vector.yaml
sinks:
  siem:
    type: http
    uri: https://siem-ingest.example.net/collector
    auth:
      strategy: bearer
      token: "${SIEM_TOKEN}"
    tls:
      verify_certificate: true

Verify it worked:

journalctl -u vector -n 20 --no-pager | egrep -i 'error|sent|delivered'

Network path blocked (DNS, firewall, proxy, egress ACL)

Check whether the host must use an outbound proxy and whether HTTPS_PROXY/NO_PROXY changed after a deploy.

systemctl show vector --property=Environment
curl -vkI https://siem-ingest.example.net/collector

If direct egress is blocked, allow TCP/443 from the collector host/subnet to the ingest FQDN/IP in your firewall/security group. If DNS is wrong, fix the record in your provider's dashboard (e.g. DNS → Records) or /etc/resolv.conf/upstream resolver.

For a temporary host-level test:

iptables -S OUTPUT | sed -n '1,40p'
ss -plant '( dport = :443 )'

Verify it worked:

nc -vz siem-ingest.example.net 443

Auth/token/certificate expired or rotated

If the agent uses a bearer token/API key, replace the rotated secret in the service environment or config file, then restart.

# Example systemd drop-in
mkdir -p /etc/systemd/system/vector.service.d
cat >/etc/systemd/system/vector.service.d/override.conf <<'EOF'
[Service]
Environment="SIEM_TOKEN=REDACTED_NEW_TOKEN"
EOF
systemctl daemon-reload
systemctl restart vector

If mutual TLS is used, replace the client cert/key and verify permissions:

install -m 0640 -o root -g vector client.crt /etc/vector/client.crt
install -m 0640 -o root -g vector client.key /etc/vector/client.key
openssl x509 -in /etc/vector/client.crt -noout -dates -subject
systemctl restart vector

Trade-off: disabling TLS verification may restore flow fast but removes server identity checks. Use only as a short-lived emergency step on an internal test path, not production telemetry. Verify it worked:

journalctl -u vector -n 30 --no-pager | egrep -i '401|403|certificate|delivered|sent'

Wrong endpoint/path/protocol after config change

Fix the exact URL, scheme, and path. A common breakage is an HTTP redirect that the agent does not follow, or a reverse proxy forcing http:// on a 301.

curl -vkI https://siem-ingest.example.net/collector

Bad shape:

HTTP/2 301
location: http://siem-ingest.example.net/collector/

Good fix in nginx reverse proxy:

location /collector {
    proxy_pass https://upstream_ingest;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto https;
}

Or fix the agent URI directly:

uri: https://siem-ingest.example.net/collector/

Verify it worked:

curl -vkI https://siem-ingest.example.net/collector/ | egrep '^< HTTP|^< location:'

Local queue full / disk pressure / backpressure

⚠️ Deleting the queue can permanently lose unsent security telemetry. Prefer restoring connectivity and letting the agent drain. Only purge queued data if you have explicit incident approval.

Free disk, increase queue capacity if appropriate, and restart the agent after upstream connectivity is restored.

df -h
journalctl -u vector -n 50 --no-pager | egrep -i 'buffer|queue|backpressure|retry'

Example queue tuning:

# Vector disk buffer example
buffer:
  type: disk
  max_size: 10737418240 # 10 GiB
  when_full: block

If the disk is full because of old local logs, rotate/compress them first instead of deleting the agent queue. Verify it worked:

watch -n 5 'du -sh /var/lib/vector 2>/dev/null; journalctl -u vector -n 5 --no-pager'

Source path, parser, or filter changed and records are dropped before send

After deploys, apps often switch from JSON to plain text, change file names, or write to stdout instead of a file. Update the source path and parser to match reality.

ls -l /var/log/app/
stat /var/log/app/app.log

Example parser fix:

transforms:
  parse_json:
    type: remap
    inputs: [app_logs]
    source: |
      . = parse_json!(string!(.message))

If logs are no longer JSON, remove the strict parser or add a fallback:

source: |
  parsed, err = parse_json(.message)
  if err == null { . = parsed }

For logrotate issues, prefer glob paths and inode-aware tailing in your agent config. Restart after changes. Verify it worked:

journalctl -u vector -n 30 --no-pager | egrep -i 'drop|parse|tail|sent' 

Prevention

• Add a freshness alert on each critical source, not just ingest service health. Example Prometheus rule on exported last-success timestamp:

groups:
- name: siem-pipeline
  rules:
  - alert: SIEMSourceStale
    expr: time() - siem_source_last_event_unix{source="prod-app"} > 300
    for: 5m

• Validate collector config in CI before deploy/restart:

vector validate --config deploy/vector.yaml
fluent-bit --dry-run -c deploy/fluent-bit.conf
filebeat test config -c deploy/filebeat.yml

• Pin and test the exact ingest URL with a synthetic probe from the same network zone as the collector:

curl -fsS -o /dev/null -w '%{http_code} %{remote_ip}\n' https://siem-ingest.example.net/collector/

Alert on non-2xx/expected 4xx and on redirect-to-http.

• Track queue size and disk usage for collectors:

du -sb /var/lib/vector /var/lib/fluent-bit /var/lib/filebeat 2>/dev/null

Export as a metric and alert before disk exceeds 80%.

• Put secret expiry/rotation checks on the calendar and in code. For mTLS certs:

openssl x509 -in /etc/vector/client.crt -noout -enddate

Alert when expiry is <14 days.

• Keep a canary log line in each app and verify it appears end-to-end. Example cron/systemd timer:

logger -t siem-canary "siem_canary service=prod-app ts=$(date -Is)"

Alert if the canary is missing from the SIEM for 10 minutes.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.