Normalise app logs into JSON so your SIEM can parse and alert

TL;DR — If your SIEM is not extracting fields reliably, stop shipping human-formatted log lines and emit one JSON object per line with stable keys: @timestamp, severity, message, service.name, trace.id, user.id, and event.dataset. The single most likely fix is to normalise at the application boundary first, then use your collector only to rename fields and drop noise — not to regex-parse arbitrary text. Reading time: ~5 min

Goal

When you are done, your application will write newline-delimited JSON logs with consistent field names and UTC timestamps, your collector will forward them unchanged or with minimal remapping, and your SIEM will show each event as parsed fields you can filter on immediately instead of one giant message blob.

Prerequisites

• Access to change application logging config or code in the service you own
• Access to the host/container runtime where logs are emitted
• A log collector already available or installable: fluent-bit >= 3.0 — check with fluent-bit --version
• jq >= 1.6 — check with jq --version
• curl and grep
• A sample app log file path or container log stream
• The ingestion endpoint, token, and expected format for your SIEM HTTP/syslog input
• Time sync working on the host — check with timedatectl status

Steps

Step 1: Pick one canonical event shape

Use this exact JSON shape for every application event. Keep keys stable; add fields, do not rename them later.

{"@timestamp":"2026-10-30T14:22:11.381Z","severity":"INFO","message":"HTTP request completed","service":{"name":"billing-api","version":"2.14.3","environment":"prod"},"event":{"dataset":"application","action":"http_request"},"trace":{"id":"4f5b7d9c2e1a4c8b9d0e1f2a3b4c5d6e"},"span":{"id":"9a8b7c6d5e4f3a2b"},"http":{"request":{"method":"GET"},"response":{"status_code":200}},"url":{"path":"/v1/invoices/123"},"client":{"address":"203.0.113.10"},"user":{"id":"u_12345"},"labels":{"request_id":"req-7d7b8d1e"}}

What you should see when this succeeds: one log event equals one JSON object on one line, with no stack trace fragments split across multiple lines unless you intentionally encode them as \\n inside message.

Step 2: Change the app logger to emit NDJSON

If your app is Node.js, use a JSON logger directly. Example with pino:

npm install pino

import pino from 'pino';
const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  base: {
    service: {
      name: 'billing-api',
      version: process.env.APP_VERSION || 'dev',
      environment: process.env.NODE_ENV || 'dev'
    },
    event: { dataset: 'application' }
  },
  timestamp: () => `,"@timestamp":"${new Date().toISOString()}"`,
  formatters: {
    level: (label) => ({ severity: label.toUpperCase() })
  }
});
logger.info({ trace: { id: '4f5b7d9c2e1a4c8b9d0e1f2a3b4c5d6e' }, user: { id: 'u_12345' } }, 'HTTP request completed');

If your app is Python, use python-json-logger:

pip install python-json-logger

import logging, sys
from pythonjsonlogger import jsonlogger

handler = logging.StreamHandler(sys.stdout)
formatter = jsonlogger.JsonFormatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger = logging.getLogger()
logger.handlers = [handler]
logger.setLevel(logging.INFO)
logger.info('HTTP request completed', extra={
    'severity': 'INFO',
    'service': {'name': 'billing-api', 'version': '2.14.3', 'environment': 'prod'},
    'event': {'dataset': 'application', 'action': 'http_request'},
    'trace': {'id': '4f5b7d9c2e1a4c8b9d0e1f2a3b4c5d6e'},
    'user': {'id': 'u_12345'}
})

What you should see when this succeeds: stdout contains compact single-line JSON, not pretty-printed multi-line output.

Step 3: Validate the log lines before shipping them

Run this against a sample file or process output.

tail -n 20 /var/log/myapp/app.log | jq -c . >/dev/null; echo $?

Expected success output:

0

If you want to inspect required fields:

tail -n 3 /var/log/myapp/app.log | jq -c '{ts:."@timestamp",sev:.severity,msg:.message,svc:.service.name,trace:.trace.id,user:.user.id}'

Expected output shape:

{"ts":"2026-10-30T14:22:11.381Z","sev":"INFO","msg":"HTTP request completed","svc":"billing-api","trace":"4f5b7d9c2e1a4c8b9d0e1f2a3b4c5d6e","user":"u_12345"}

What you should see when this succeeds: jq exits 0 and prints objects without parse errors like parse error: Invalid numeric literal.

Step 4: Configure Fluent Bit to pass JSON through and rename only what the SIEM needs

⚠️ If you replace an existing collector config on a shared host, you can interrupt log delivery for other services. Back up the current config first with cp /etc/fluent-bit/fluent-bit.conf /etc/fluent-bit/fluent-bit.conf.bak.

Create this exact config for a file input and HTTP output. Replace the endpoint and token values.

[SERVICE]
    Flush         1
    Daemon        Off
    Log_Level     info

[INPUT]
    Name          tail
    Path          /var/log/myapp/app.log
    Tag           app.billing
    Parser        json
    DB            /var/lib/fluent-bit/tail-billing.db
    Read_from_Head On

[FILTER]
    Name          modify
    Match         app.billing
    Rename        severity log.level
    Rename        message message
    Rename        @timestamp @timestamp

[OUTPUT]
    Name          http
    Match         app.billing
    Host          siem.example.com
    Port          443
    URI           /ingest
    Format        json_lines
    tls           On
    Header        Authorization Bearer REPLACE_WITH_TOKEN
    Header        Content-Type application/x-ndjson

Test the config before restarting:

fluent-bit --dry-run -c /etc/fluent-bit/fluent-bit.conf

Expected success output shape:

Fluent Bit v3.x.x
* dry run OK

What you should see when this succeeds: dry run returns exit code 0 and no parser or section errors.

Step 5: Start the collector and watch for delivery errors

systemctl restart fluent-bit
systemctl --no-pager --full status fluent-bit
journalctl -u fluent-bit -n 50 --no-pager

Expected healthy output shape:

● fluent-bit.service - Fluent Bit
     Active: active (running)
...
[ info] [output:http:http.0] siem.example.com:443, HTTP status=200

Common failing output shapes:

[error] [output:http:http.0] HTTP status=400
[error] [input:tail:tail.0] parser 'json' failed
[ warn] [engine] failed to flush chunk '1-...'

What you should see when this succeeds: the service is active (running) and the output plugin logs HTTP status=200 or 201.

Step 6: Preserve stack traces and request correlation correctly

For exceptions, keep the event as one JSON object and put the stack trace in a field, not as raw multiline text.

{"@timestamp":"2026-10-30T14:25:01.004Z","severity":"ERROR","message":"Unhandled exception","service":{"name":"billing-api","environment":"prod"},"event":{"dataset":"application","action":"exception"},"trace":{"id":"4f5b7d9c2e1a4c8b9d0e1f2a3b4c5d6e"},"error":{"type":"TypeError","message":"Cannot read properties of undefined","stack_trace":"TypeError: Cannot read properties of undefined\n    at createInvoice (/srv/app.js:42:13)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)"},"labels":{"request_id":"req-7d7b8d1e"}}

What you should see when this succeeds: one SIEM event contains the full exception, and searches on trace.id or labels.request_id return both the error and the surrounding request logs.

Verify it works

Run an end-to-end test: write one known event, confirm the collector accepts it, then confirm the SIEM parsed fields.

echo '{"@timestamp":"2026-10-30T14:30:00.000Z","severity":"WARN","message":"SIEM normalisation test","service":{"name":"billing-api","environment":"prod"},"event":{"dataset":"application","action":"test_event"},"trace":{"id":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"},"user":{"id":"u_test"}}' >> /var/log/myapp/app.log
journalctl -u fluent-bit -n 20 --no-pager | grep -E 'HTTP status=200|HTTP status=201'

Expected collector output shape:

[ info] [output:http:http.0] siem.example.com:443, HTTP status=200

Then, in your SIEM search UI, query these exact fields and values:

event.action:"test_event" AND service.name:"billing-api" AND user.id:"u_test"

Expected result: one event appears with separate parsed fields for @timestamp, severity or log.level, service.name, trace.id, and user.id; the raw message is not the only searchable field.

Common pitfalls

Tailing container logs that are already wrapped in JSON

Mistake: pointing Fluent Bit at Docker or CRI logs and also parsing the application payload as if it were the top-level record.

Symptom: the SIEM shows fields nested under log or message, or the collector logs parser 'json' failed.

Fix: parse the container runtime format first, then decode the application JSON from the log field, or emit app logs directly to stdout and use the runtime-aware input plugin.

Using local time instead of UTC ISO-8601

Mistake: writing timestamps like 10/30/2026 2:30:00 PM or local time without offset.

Symptom: events arrive hours off, out of order, or fail time parsing in the SIEM.

Fix: emit @timestamp exactly as UTC ISO-8601, for example 2026-10-30T14:30:00.000Z.

Pretty-printed JSON or raw multiline stack traces

Mistake: enabling pretty output in development and shipping it to production.

Symptom: one exception becomes 20 separate events; searches and alerts miss the real context.

Fix: disable pretty printing in production and store stack traces in error.stack_trace with \n escapes inside one JSON object.

Renaming fields in the collector on every service differently

Mistake: one service sends level, another severity, another logLevel, and the collector rewrites each ad hoc.

Symptom: SIEM detections need per-service exceptions and dashboards break.

Fix: standardise field names in application code first; use the collector only for minimal compatibility mapping.

Shipping sensitive data because the message was never reviewed

Mistake: logging full headers, bearer tokens, session cookies, or raw request bodies.

Symptom: secrets appear in the SIEM and trigger incident response work you created yourself.

Fix: redact before emit; drop keys like authorization, cookie, set-cookie, password, and token fields in application code or a collector filter before output.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.