Stream model responses over SSE with cancellation and partial failures

TL;DR — Build your SSE endpoint so it sends proper headers, flushes each chunk, detects client disconnects, and emits an explicit in-band error event before closing when the upstream model fails mid-stream. The most common breakage is buffering by your app server or reverse proxy; disable buffering and test with curl -N before debugging application code. Reading time: ~5 min

Goal

When you finish, your app exposes an HTTP endpoint that streams model output over Server-Sent Events, stops upstream work immediately when the client cancels, and reports partial failures as SSE events instead of hanging or returning malformed JSON.

Prerequisites

• Node.js >= 20 — check with:

node --version

• npm >= 10 — check with:

npm --version

• A running reverse proxy only if you use one in front of the app (nginx, Apache, ingress, or a cloud load balancer)
• A model/upstream function that can yield partial tokens or chunks; this article uses a local async generator so you can test without a vendor SDK
• curl with SSE-friendly output support — check with:

curl --version

• If you run nginx, shell access to edit the site config and reload nginx
• A browser or any HTTP client that supports AbortController for cancellation testing

Steps

Step 1: Create a minimal SSE server that streams chunks

Create server.mjs with this exact content:

import http from 'node:http';
import { setTimeout as sleep } from 'node:timers/promises';

async function* fakeModelStream({ signal }) {
  const chunks = ['Hello', ' ', 'from', ' ', 'the', ' ', 'model'];
  for (const chunk of chunks) {
    if (signal.aborted) throw new Error('aborted');
    await sleep(300);
    yield chunk;
  }
}

function writeSse(res, { event, data, id }) {
  if (id) res.write(`id: ${id}\n`);
  if (event) res.write(`event: ${event}\n`);
  const payload = typeof data === 'string' ? data : JSON.stringify(data);
  for (const line of payload.split('\n')) {
    res.write(`data: ${line}\n`);
  }
  res.write('\n');
}

const server = http.createServer(async (req, res) => {
  if (req.url !== '/stream') {
    res.writeHead(404, { 'content-type': 'text/plain; charset=utf-8' });
    res.end('not found');
    return;
  }

  const ac = new AbortController();
  req.on('close', () => ac.abort());
  req.on('aborted', () => ac.abort());

  res.writeHead(200, {
    'content-type': 'text/event-stream; charset=utf-8',
    'cache-control': 'no-cache, no-transform',
    'connection': 'keep-alive',
    'x-accel-buffering': 'no'
  });

  res.write(': connected\n\n');

  try {
    let i = 0;
    for await (const chunk of fakeModelStream({ signal: ac.signal })) {
      i += 1;
      writeSse(res, { event: 'token', id: String(i), data: { delta: chunk } });
    }
    writeSse(res, { event: 'done', data: { finish_reason: 'stop' } });
    res.end();
  } catch (err) {
    if (ac.signal.aborted) {
      res.end();
      return;
    }
    writeSse(res, { event: 'error', data: { message: err.message, retryable: false } });
    res.end();
  }
});

server.listen(3000, () => {
  console.log('listening on http://127.0.0.1:3000');
});

Start it:

node server.mjs

Success looks like:

listening on http://127.0.0.1:3000

Step 2: Test raw SSE framing from the terminal

In another terminal, call the endpoint without output buffering:

curl -N http://127.0.0.1:3000/stream

Success looks like a sequence of SSE frames arriving incrementally, not all at once:

: connected

event: token
id: 1
data: {"delta":"Hello"}

event: token
id: 2
data: {"delta":" "}

event: done
data: {"finish_reason":"stop"}

Step 3: Add explicit partial-failure handling

Replace fakeModelStream in server.mjs with this exact version to simulate a mid-stream upstream failure:

async function* fakeModelStream({ signal }) {
  const chunks = ['Hello', ' ', 'partial', ' ', 'output'];
  for (let idx = 0; idx < chunks.length; idx += 1) {
    if (signal.aborted) throw new Error('aborted');
    await sleep(300);
    if (idx === 3) throw new Error('upstream 502 while streaming');
    yield chunks[idx];
  }
}

Restart the server:

pkill -f 'node server.mjs' || true
node server.mjs

Test again:

curl -N http://127.0.0.1:3000/stream

Success looks like partial tokens, then an error event, then connection close:

event: token
id: 1
data: {"delta":"Hello"}

event: token
id: 2
data: {"delta":" "}

event: token
id: 3
data: {"delta":"partial"}

event: error
data: {"message":"upstream 502 while streaming","retryable":false}

Step 4: Add client-side cancellation

Create client.mjs with this exact content:

const ac = new AbortController();
setTimeout(() => ac.abort(), 1000);

const res = await fetch('http://127.0.0.1:3000/stream', {
  signal: ac.signal,
  headers: { Accept: 'text/event-stream' }
});

const reader = res.body.getReader();
const decoder = new TextDecoder();

try {
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    process.stdout.write(decoder.decode(value, { stream: true }));
  }
} catch (err) {
  console.error('client aborted:', err.name);
}

Run it:

node client.mjs

Success looks like a small amount of streamed output followed by client abort, and your server process stays healthy without logging an unhandled exception:

: connected

event: token
id: 1
data: {"delta":"Hello"}
client aborted: AbortError

Step 5: Disable reverse-proxy buffering if you use nginx

⚠️ Reloading nginx with a broken config can interrupt traffic. Run nginx -t before systemctl reload nginx.

Add this exact location block to your site config:

location /stream {
    proxy_pass http://127.0.0.1:3000;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_buffering off;
    proxy_cache off;
    chunked_transfer_encoding on;
    proxy_read_timeout 3600s;
    add_header X-Accel-Buffering no;
}

Test and reload:

sudo nginx -t && sudo systemctl reload nginx

Success looks like:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Step 6: Return the right status-code behavior

Keep this rule in your handler: if streaming has started, send an SSE error event and close; do not try to switch to HTTP 500 after bytes have been sent. If validation fails before streaming starts, return normal HTTP errors. Use this exact pattern in your route logic:

if (!req.headers.accept?.includes('text/event-stream')) {
  res.writeHead(406, { 'content-type': 'application/json; charset=utf-8' });
  res.end(JSON.stringify({ error: 'send Accept: text/event-stream' }));
  return;
}

Success looks like a clean non-streaming error before any SSE bytes are sent:

curl -i http://127.0.0.1:3000/stream

HTTP/1.1 406 Not Acceptable
content-type: application/json; charset=utf-8

{"error":"send Accept: text/event-stream"}

Verify it works

Run these checks end to end.

Terminal stream test:

curl -N -H 'Accept: text/event-stream' http://127.0.0.1:3000/stream

Expected result: token events arrive one by one every ~300 ms; on simulated failure you get event: error and the connection closes.

Cancellation test:

node client.mjs

Expected result: the client exits with AbortError; the server does not keep generating chunks after disconnect.

Proxy test, if nginx is in front:

curl -N -H 'Accept: text/event-stream' http://YOUR_HOST/stream

Expected result: same incremental behavior through nginx. If everything arrives at once at the end, buffering is still enabled somewhere upstream.

Header test:

curl -I -H 'Accept: text/event-stream' http://127.0.0.1:3000/stream

Expected result includes these headers:

HTTP/1.1 200 OK
content-type: text/event-stream; charset=utf-8
cache-control: no-cache, no-transform
x-accel-buffering: no

Common pitfalls

Proxy buffering is still on

Mistake: leaving default buffering enabled in nginx, ingress, or a cloud proxy.

Symptom: curl -N prints nothing for several seconds, then dumps the whole response at once.

Fix: set proxy_buffering off; for the SSE path and send X-Accel-Buffering: no from the app.

Treating mid-stream failure like a normal HTTP 500

Mistake: calling res.writeHead(500) after token bytes were already sent.

Symptom: client sees truncated output, ERR_INVALID_HTTP_RESPONSE, or a parser error instead of a structured failure event.

Fix: after the first SSE byte, emit event: error with JSON payload and then res.end().

Not handling client disconnects

Mistake: the server never listens for req.close or req.aborted.

Symptom: upstream model generation keeps running after the browser tab closes, wasting tokens and CPU.

Fix: create an AbortController, call ac.abort() on close and aborted, and pass ac.signal into upstream generation.

Using compression on the SSE route

Mistake: applying gzip/br compression middleware globally to /stream.

Symptom: chunks are delayed, coalesced, or never flushed until enough data accumulates.

Fix: exclude the SSE route from compression and keep cache-control: no-transform.

Sending malformed SSE frames

Mistake: writing raw JSON chunks without data: prefixes and blank-line separators.

Symptom: browser EventSource never fires message handlers or only fires once.

Fix: write each event as event: <name>\ndata: <payload>\n\n; split multi-line payloads into multiple data: lines.

Redirects on the stream endpoint

Mistake: exposing /stream behind an HTTP-to-HTTPS or slash-normalization redirect.

Symptom: clients fail intermittently; curl -I shows a redirect instead of 200 OK:

HTTP/1.1 301 Moved Permanently
Location: https://example.com/stream/

Fix: point clients at the final URL directly and avoid redirect hops on long-lived streaming endpoints.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.