Malformed JSON from LLMs Under Load: Enforce Schemas and Retry Safely

TL;DR — When JSON breaks only under load, the usual culprit is not "the parser being picky"; it is one of five things: weak output constraints, truncation from token/time limits, broken streaming assembly, concurrency-induced context degradation, or retries that amplify partial outputs. The fastest win is to validate against a strict schema server-side, add one bounded repair pass for parseable near-misses, and cap retries with a hard budget so load spikes do not turn malformed JSON into an outage. Reading time: ~6 min

The scenario

You ship a Tuesday afternoon change that increases parallel request volume: batch summarization, extraction, or tool-call fanout. Ten minutes later, your API error rate jumps from 0.3% to 7%, but only on endpoints that expect structured JSON from the model. Application logs show JSONDecodeError, some responses end mid-object, and a few contain polite prose wrapped around otherwise valid fields. Product is asking why the queue is backing up when the model provider status page still says green.

Symptoms

• Spikes in parser failures under concurrency, often with one of these exact errors:

json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 842 (char 841)

json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 1297 (char 1296)

SyntaxError: Unexpected end of JSON input

pydantic_core._pydantic_core.ValidationError: 3 validation errors for OutputSchema
items.2.score
  Input should be a valid number [type=float_type, input_value="high", input_type=str]

• HTTP 200 from your app, but downstream worker/job marked failed because schema validation rejects the body.
• Streaming responses that end without a closing } or ].
• Increased latency before failure, often paired with upstream timeouts:

upstream request timeout after 30.0s
context deadline exceeded

• Logs showing retries on the same request ID, then a final malformed payload:

request_id=9f2... attempt=1 parse_error=true
request_id=9f2... attempt=2 parse_error=true
request_id=9f2... attempt=3 gave_up retry_budget_exhausted

• User-visible symptoms: empty widgets, "Could not parse response", partial extracted fields, or jobs stuck in processing until timeout.

Likely causes

| Response truncation from token/output-length or request timeout | Very common | ```bash
grep -E 'max_tokens|timeout|finish_reason|length' app.log | tail -n 20
``` |
| Streaming assembly bug concatenates chunks incorrectly or validates before stream completion | Common | ```bash
grep -E 'stream|chunk|Unexpected end of JSON input|unterminated' app.log | tail -n 50
``` |
| Concurrency/load causes prompt degradation or context clipping, increasing malformed outputs | Common | ```bash
grep -E '429|queue_depth|concurrency|semaphore|context_length' app.log | tail -n 50
``` |
| Retry policy is unbounded or retries malformed partials without a repair/validation gate | Common | ```bash
grep -E 'attempt=[2-9]|retry_budget|same request_id' app.log | tail -n 50
``` |
| Your parser/validator is too permissive in one place and too strict in another, causing inconsistent handling | Less common | ```bash
rg -n 'json.loads|orjson|pydantic|ajv|zod|fastjsonschema' .
``` |

## Step-by-step diagnosis
1. Check whether requests actually ask for strict structured output.
```bash
jq '.response_format, .schema // empty, .messages[-1].content' /var/log/app/sample-request.json

If you do not see a schema object, or your prompt says things like Return JSON only without machine-enforced structure, this is your problem. Jump to Fixes → Weak or missing schema enforcement.

1. Check for truncation or timeout indicators before touching parser code.

grep -E 'finish_reason|length|max_tokens|timeout|deadline exceeded|upstream request timeout' /var/log/app/app.log | tail -n 50

This is your problem if you see finish_reason=length, Unexpected end of JSON input, or request deadlines around the same timestamps as parse failures. Jump to Fixes → Truncation from token or timeout limits.

1. If you use streaming, verify that you assemble the full payload before parsing.

grep -E 'stream_start|stream_chunk|stream_end|parse_before_end|Unexpected end of JSON input' /var/log/app/app.log | tail -n 100

This is your problem if parse attempts happen before stream_end, or chunk counts vary suspiciously on failures. Jump to Fixes → Streaming assembly/parsing bug.

1. Check whether failures correlate with concurrency or provider throttling.

paste <(grep 'parse_error=true' /var/log/app/app.log | awk '{print $1" "$2}' | cut -d: -f1-2 | sort | uniq -c) <(grep -E '429|queue_depth|concurrency' /var/log/app/app.log | awk '{print $1" "$2}' | cut -d: -f1-2 | sort | uniq -c) | tail

This is your problem if malformed JSON rises with queue depth, 429s, or when you increased worker count/batch size. Jump to Fixes → Concurrency pressure and context degradation.

1. Inspect retry behavior on a single failing request ID.

grep 'request_id=9f2' /var/log/app/app.log

This is your problem if the same request is retried multiple times without a validation/repair gate, or if each retry reuses already-partial content. Jump to Fixes → Bad retry policy and no repair pass.

1. Finally, compare validation paths across services.

rg -n 'json.loads|orjson.loads|BaseModel|model_validate|JSON.parse|z\.object|ajv\.compile' ./services ./workers

This is your problem if one service accepts loose JSON or stringified numbers while another rejects them later. Jump to Fixes → Inconsistent parser/validator behavior.

Fixes

Weak or missing schema enforcement

Do not rely on prompt wording alone. Validate server-side against a strict schema and reject extra fields/types.

Python with Pydantic v2:

from pydantic import BaseModel, ConfigDict, ValidationError

class Item(BaseModel):
    model_config = ConfigDict(extra='forbid', strict=True)
    id: str
    score: float
    label: str

class OutputSchema(BaseModel):
    model_config = ConfigDict(extra='forbid', strict=True)
    items: list[Item]
    summary: str

raw = model_response_text
obj = OutputSchema.model_validate_json(raw)

Node with Ajv:

npm i ajv ajv-formats

import Ajv from 'ajv';
import addFormats from 'ajv-formats';

const ajv = new Ajv({allErrors: true, strict: true, removeAdditional: false});
addFormats(ajv);
const schema = {
  type: 'object',
  additionalProperties: false,
  required: ['items', 'summary'],
  properties: {
    summary: {type: 'string'},
    items: {
      type: 'array',
      items: {
        type: 'object',
        additionalProperties: false,
        required: ['id', 'score', 'label'],
        properties: {
          id: {type: 'string'},
          score: {type: 'number'},
          label: {type: 'string'}
        }
      }
    }
  }
};
const validate = ajv.compile(schema);

Add a fail-closed path: if validation fails, do not pass the payload downstream.

Verify it worked:

pytest -q tests/test_structured_output.py -k strict_schema

Truncation from token or timeout limits

Raise output budget only enough to fit worst-case valid JSON, and increase upstream timeout if your app cuts the response first. Also reduce verbosity in the prompt; every extra token competes with closing braces.

Example env/config:

export MODEL_TIMEOUT_MS=45000
export MODEL_MAX_OUTPUT_TOKENS=1800
export APP_REQUEST_DEADLINE_MS=50000

If you run nginx in front of the app:

proxy_read_timeout 60s;
proxy_send_timeout 60s;

Reload safely:

nginx -t && sudo systemctl reload nginx

Trade-off: higher token/time limits reduce truncation but increase tail latency and cost. If malformed JSON appears only on very large arrays, cap result size and paginate.

Verify it worked:

grep -E 'finish_reason=length|Unexpected end of JSON input|deadline exceeded' /var/log/app/app.log | tail -n 20

You want no new matches after deploy.

Streaming assembly/parsing bug

Buffer the complete stream, then parse once. Do not parse per chunk unless you are using a framing protocol designed for incremental decoding.

Node example:

let buf = '';
stream.on('data', (chunk) => {
  buf += chunk.toString('utf8');
});
stream.on('end', () => {
  const parsed = JSON.parse(buf);
  handle(parsed);
});

If the provider sends line-delimited events, assemble only the content field and ignore keepalives/metadata.

Add a guard to log final byte length and whether end fired:

stream.on('end', () => logger.info({bytes: Buffer.byteLength(buf), stream_complete: true}, 'model stream complete'));
stream.on('error', (err) => logger.error({err}, 'model stream error'));

Trade-off: buffering increases memory use for large outputs. If outputs are large, switch to NDJSON or a framed schema your consumer can validate incrementally.

Verify it worked:

grep -E 'parse_before_end|stream_complete|Unexpected end of JSON input' /var/log/app/app.log | tail -n 50

Concurrency pressure and context degradation

Throttle parallel generations and bound queue depth. Under load, shorter prompts and fewer simultaneous generations often improve JSON validity more than adding retries.

Python semaphore example:

import asyncio
SEM = asyncio.Semaphore(8)

async def call_model(payload):
    async with SEM:
        return await client.generate(payload)

Worker config example:

export MODEL_CONCURRENCY=8
export JOB_QUEUE_MAX_INFLIGHT=100

If you batch requests, reduce batch size:

export EXTRACTION_BATCH_SIZE=10

Trade-off: lower concurrency reduces throughput but usually improves success rate and p95 latency during incidents because you stop amplifying retries and timeouts.

Verify it worked:

grep -E 'parse_error=true|429|queue_depth=' /var/log/app/app.log | tail -n 100

You want parse errors to drop with stable queue depth.

Bad retry policy and no repair pass

Use one repair pass for near-miss JSON, then one full regeneration at most. Cap total attempts and total wall-clock budget per request.

Python example:

import json, time
from pydantic import ValidationError

MAX_ATTEMPTS = 2
MAX_WALL_MS = 12000

def try_parse_or_repair(raw: str):
    try:
        return OutputSchema.model_validate_json(raw)
    except Exception:
        repaired = repair_json(raw)  # local deterministic repair only
        return OutputSchema.model_validate_json(repaired)

start = time.monotonic()
for attempt in range(1, MAX_ATTEMPTS + 1):
    raw = generate()
    try:
        obj = try_parse_or_repair(raw)
        break
    except Exception:
        if (time.monotonic() - start) * 1000 > MAX_WALL_MS or attempt == MAX_ATTEMPTS:
            raise RuntimeError('retry_budget_exhausted')

Keep repair deterministic and local: strip markdown fences, trim leading prose before first {, trim trailing text after final }, then revalidate. Do not build a recursive "ask the model to fix the model" loop during an incident.

Verify it worked:

grep -E 'retry_budget_exhausted|attempt=3|repair_pass=true' /var/log/app/app.log | tail -n 50

Inconsistent parser/validator behavior

Standardize on one validator per language boundary and one canonical schema. Reject stringified numbers if downstream expects numbers.

CI check example for Node:

npm test -- structured-output

Test fixture:

{"items":[{"id":"a1","score":"0.91","label":"ok"}],"summary":"x"}

Expected result: fail validation.

For Python services, replace mixed json.loads() + ad hoc checks with model_validate_json() at ingress.

Verify it worked:

rg -n 'json.loads\(|JSON.parse\(' ./services ./workers

Review remaining call sites and remove duplicate loose parsing.

Prevention

• Add a structured-output SLI and alert on parse/validation failure rate, not just HTTP 5xx.

# Prometheus-style counters emitted by app
llm_json_parse_fail_total{endpoint="/extract"} 17
llm_schema_validation_fail_total{endpoint="/extract"} 9
llm_retry_exhausted_total{endpoint="/extract"} 3

Alert when parse_fail_total / request_total > 0.01 for 5 minutes.

• Put schema fixtures in CI with malformed edge cases.

pytest -q tests/test_structured_output.py
npm test -- structured-output

Include fixtures for truncated JSON, markdown-fenced JSON, extra fields, and wrong scalar types.

• Pin concurrency and timeout config in code or checked-in env templates.

cat >> .env.example <<'EOF'
MODEL_CONCURRENCY=8
MODEL_TIMEOUT_MS=45000
MODEL_MAX_OUTPUT_TOKENS=1800
APP_REQUEST_DEADLINE_MS=50000
MAX_REPAIR_PASSES=1
MAX_GENERATION_ATTEMPTS=2
EOF

Do not leave these as undocumented runtime tweaks.

• Log finish reason, output byte length, and request attempt on every generation.

{"request_id":"9f2","attempt":1,"finish_reason":"length","output_bytes":16372,"schema_valid":false}

Without these fields, you will waste time guessing whether the issue is truncation or validation.

• Add a load test that validates JSON at target concurrency before deploy.

k6 run --vus 20 --duration 5m scripts/llm-json-load.js

Pass criterion: schema_valid_rate >= 99.5% at your normal p95 concurrency.

• If you stream, add an integration test that kills the upstream connection mid-response and asserts your app returns a controlled error, not partial JSON.

pytest -q tests/test_stream_interrupt.py

That catches the class of bugs where a client disconnect becomes Unexpected end of JSON input in production.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.