Node service memory keeps growing until OOM kill: leak diagnosis runbook

TL;DR — If a Node service survives traffic spikes but its memory only ever trends upward and eventually dies with OOMKilled, start by separating V8 heap growth from native/external memory growth. The most common real fix is removing unbounded in-process retention (Maps, caches, listeners, queued jobs, per-request buffers), confirmed with two heap snapshots or process.memoryUsage() plus --trace-gc. Reading time: ~6 min

The scenario

It is Tuesday at 3:40 PM. Your Node service has been up since the morning deploy, latency is still acceptable, and health checks are green, but one pod keeps restarting every couple of hours. The dashboard shows memory climbing in a clean staircase until the container is OOMKilled, while CPU stays mostly flat. You roll back and the pattern slows down but does not disappear, which means you now need to prove whether this is a real leak, a cache with no bounds, or memory outside the V8 heap.

Symptoms

• Container or host restarts with OOM events:

Last State:     Terminated
Reason:         OOMKilled
Exit Code:      137

• Process manager logs show signal-based death:

PM2        | App [api:0] exited with code [0] via signal [SIGKILL]
systemd[1] | api.service: Main process exited, code=killed, status=9/KILL

• Node fatal heap exhaustion when V8 heap, not container limit, is hit:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb0a8e0 node::Abort()
 2: 0xa1b04e node::FatalError(char const*, char const*)

• Kubernetes events show memory pressure:

kubectl describe pod api-7d8c9f6f5b-2xk9n
...
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
Events:
  Warning  OOMKilling  2m10s  kubelet  Memory cgroup out of memory: Killed process 12345 (node)

• RSS grows while heapUsed stays relatively flat:

rss=1280MB heapUsed=180MB heapTotal=240MB external=820MB arrayBuffers=790MB

• GC runs more often with little recovery:

[12345:0x7f...]  812345 ms: Mark-sweep 412.3 (430.1) -> 410.9 (431.3) MB, 38.7 / 0.1 ms  allocation failure; scavenge might not succeed

• User-visible impact: intermittent 502/504 during restarts, dropped websocket clients, queue consumers lagging after each restart.

Likely causes

| EventEmitter/listener leak or timers/intervals never cleared | Common | ```bash
NODE_OPTIONS='--trace-warnings' node app.js
``` |
| External/native memory growth from Buffers, streams, compression, image libs, DB drivers | Common | ```bash
node -e 'setInterval(()=>{const m=process.memoryUsage();console.log({rss:m.rss,heapUsed:m.heapUsed,external:m.external,arrayBuffers:m.arrayBuffers})},5000)'
``` |
| Heap fragmentation / high-water behavior mistaken for leak | Sometimes | ```bash
node --trace-gc app.js
``` |
| Running with too-low container memory or too-high `--max-old-space-size` for the limit | Sometimes | ```bash
cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /sys/fs/cgroup/memory/memory.limit_in_bytes
``` |
| Actual native addon leak or runtime bug | Less common | ```bash
npx clinic heapprofiler -- node app.js
``` |

## Step-by-step diagnosis
1. Check whether the process is killed by the OS/container or by V8.
```bash
kubectl describe pod <pod> | sed -n '/Last State:/,/Events:/p'
# or
journalctl -u <service> -n 100 --no-pager
# or
docker inspect <container> --format '{{.State.OOMKilled}} {{.State.ExitCode}}'

If you see OOMKilled, Exit Code: 137, or status=9/KILL, jump to step 2. If logs show Reached heap limit Allocation failed - JavaScript heap out of memory, jump to step 3.

1. Compare container memory limit to Node's actual usage.

cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /sys/fs/cgroup/memory/memory.limit_in_bytes
ps -o pid,rss,vsz,cmd -p $(pgrep -n node)

If the cgroup limit is close to observed RSS and much lower than what the service normally needs, jump to ### Memory limit mismatch. If RSS keeps climbing over time regardless of traffic shape, continue.

1. Log memory buckets every 10 seconds under representative load.

node -e 'setInterval(()=>{const m=process.memoryUsage();console.log(new Date().toISOString(),{rss:(m.rss/1048576).toFixed(1)+"MB",heapUsed:(m.heapUsed/1048576).toFixed(1)+"MB",heapTotal:(m.heapTotal/1048576).toFixed(1)+"MB",external:(m.external/1048576).toFixed(1)+"MB",arrayBuffers:(m.arrayBuffers/1048576).toFixed(1)+"MB"})},10000)'

If heapUsed trends upward with each interval and never returns after GC, jump to ### Unbounded in-process retention. If rss and external or arrayBuffers grow while heapUsed stays mostly flat, jump to ### External/native memory growth.

1. Turn on GC tracing in a non-production replica or canary.

node --trace-gc --trace-gc-verbose app.js 2>&1 | tee /tmp/gc.log

If you see repeated major GC with tiny drops, such as 412.3 -> 410.9 MB, the process is retaining memory; continue to step 5. If memory stabilizes after warm-up and only RSS remains high, jump to ### Heap fragmentation / high-water behavior.

1. Capture two heap snapshots several minutes apart.

⚠️ Heap snapshots pause the process and can briefly spike memory. Do this on a canary, one replica behind a load balancer drain, or a reproduced staging workload.

node --heapsnapshot-signal=SIGUSR2 app.js
kill -USR2 $(pgrep -n node)
sleep 300
kill -USR2 $(pgrep -n node)
ls -lh *.heapsnapshot

Open the snapshots in Chromium DevTools: DevTools -> Memory -> Load profile.... If the diff shows growing retained size under Map, Array, request objects, queue payloads, or your own module paths, jump to ### Unbounded in-process retention. If it shows many listeners or closures retained by timers, jump to ### EventEmitter/listener or timer leak.

1. Check for listener warnings and runaway intervals.

NODE_OPTIONS='--trace-warnings' node app.js 2>&1 | tee /tmp/warnings.log

If you see MaxListenersExceededWarning: Possible EventEmitter memory leak detected, jump to ### EventEmitter/listener or timer leak.

1. If external memory is growing, inspect Buffer-heavy paths.

grep -R "Buffer\.alloc\|Buffer\.from\|concat(\|zlib\|sharp\|canvas\|pdf\|xlsx\|archiver\|multer\|memoryStorage" -n src package.json

If the growth aligns with uploads, downloads, compression, image/PDF processing, or DB result buffering, jump to ### External/native memory growth.

1. If none of the above explains it, profile with a production-safe tool on a replica.

npx clinic heapprofiler --on-port 'autocannon -d 60 http://127.0.0.1:$PORT/health' -- node app.js

If the report points into a native addon or a specific dependency, jump to ### Native addon or runtime bug.

Fixes

Unbounded in-process retention

Bound every cache and queue. Replace ad-hoc globals like this:

const cache = new Map();
app.get('/user/:id', async (req, res) => {
  if (cache.has(req.params.id)) return res.json(cache.get(req.params.id));
  const user = await db.getUser(req.params.id);
  cache.set(req.params.id, user);
  res.json(user);
});

with a bounded TTL/LRU cache:

npm install lru-cache

import { LRUCache } from 'lru-cache';
const cache = new LRUCache({ max: 5000, ttl: 5 * 60 * 1000 });

For queues, stop accumulating work in arrays; use a broker or cap in-memory backlog:

if (pending.length > 10000) throw new Error('backlog limit exceeded');

Verify it worked:

watch -n 10 'echo; date; pmap -x $(pgrep -n node) | tail -1'

RSS should plateau instead of climbing forever.

EventEmitter/listener or timer leak

Fix duplicate listener registration inside request handlers or reconnect loops. Bad pattern:

app.get('/stream', (req, res) => {
  bus.on('event', data => res.write(JSON.stringify(data)));
});

Use cleanup and one-time registration:

app.get('/stream', (req, res) => {
  const onEvent = data => res.write(JSON.stringify(data));
  bus.on('event', onEvent);
  req.on('close', () => bus.off('event', onEvent));
});

For intervals:

const t = setInterval(sync, 10000);
process.on('SIGTERM', () => clearInterval(t));

Do not silence the symptom with emitter.setMaxListeners(0) unless you have proven the listener count is intended. Verify it worked:

NODE_OPTIONS='--trace-warnings' node app.js

The MaxListenersExceededWarning should disappear.

External/native memory growth

Stop buffering whole payloads in memory. For uploads in Express, avoid memory storage:

import multer from 'multer';
const upload = multer({ storage: multer.diskStorage({ destination: '/tmp/uploads' }) });

For proxying/downloading, stream instead of await res.arrayBuffer() or Buffer.concat():

import { pipeline } from 'node:stream/promises';
const upstream = await fetch(url);
await pipeline(upstream.body, fs.createWriteStream('/tmp/file.bin'));

For image/PDF/compression libraries, process one item at a time and destroy streams explicitly. If a dependency version is implicated, pin or upgrade it:

npm ls sharp pg sqlite3 canvas
npm install <package>@latest

Verify it worked:

node -e 'setInterval(()=>{const m=process.memoryUsage();console.log({rss:m.rss,external:m.external,arrayBuffers:m.arrayBuffers})},5000)'

external and arrayBuffers should stop trending upward under steady load.

Heap fragmentation / high-water behavior

If memory rises during warm-up and then stabilizes, this may be allocator behavior rather than a leak. Confirm with a longer run and forced idle period. On glibc-based images, try jemalloc if your base image supports it, or switch to an image that uses a less fragmentation-prone allocator. Also reduce bursty allocation patterns by streaming large responses and avoiding giant temporary arrays. Verify it worked:

node --trace-gc app.js

After warm-up, major GC should recover to a stable band.

Memory limit mismatch

Set the container limit high enough for RSS, not just V8 heap. If you set --max-old-space-size, leave headroom for native memory, stacks, code space, and buffers. Kubernetes example:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

Node example:

export NODE_OPTIONS="--max-old-space-size=768"

On a 1 GiB container, 768 MB old space is usually safer than 1024 because RSS includes more than old space. Verify it worked:

kubectl top pod <pod>

Peak memory should stay below the limit with headroom.

Native addon or runtime bug

Isolate the dependency and reproduce on the smallest script possible. Then upgrade Node and the package, or replace the package if the issue is open and unfixed.

node -v
npm outdated
npm install node@latest # use your version manager instead of this on real systems

If you can reproduce with a 20-line script, file an upstream issue with heap profiles and exact versions. Verify it worked:

npm ls <suspect-package>
node repro.js

The minimal repro should stop growing memory.

Prevention

• Add per-process memory bucket logging, not just container RSS:

setInterval(() => {
  const m = process.memoryUsage();
  console.log(JSON.stringify({ msg: 'mem', rss: m.rss, heapUsed: m.heapUsed, external: m.external, arrayBuffers: m.arrayBuffers }));
}, 60000);

• Alert on slope, not only absolute usage. Example PromQL for a 30-minute upward trend:

deriv(process_resident_memory_bytes{job="api"}[30m]) > 1048576

• Add a soak test in CI for long-lived behavior:

docker compose up -d api
k6 run --duration 30m script.js
docker stats --no-stream

Fail the job if RSS grows monotonically past an allowed band.

• Enable on-demand heap snapshots in non-prod and canaries:

export NODE_OPTIONS="--heapsnapshot-signal=SIGUSR2"

Document where snapshots are written and how to drain a replica before capturing.

• Ban unbounded caches in code review. Standardize on one cache library and require max plus ttl in code:

new LRUCache({ max: 1000, ttl: 300000 })

• Pin Node and base image versions. Memory behavior changes across Node/V8 and libc allocators; upgrade deliberately and compare a 1-hour soak graph before promoting.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.