The Node.js Memory Leak That Is Not Your Code: Bisecting a Runtime Stream Regression

Your Node service is leaking. Memory climbs steadily under load, never comes back down, and eventually the pod gets OOM-killed and restarts. You have been staring at your own code for two days. You have audited every event listener, every closure, every cache without an eviction policy. You added --max-old-space-size to buy time. You are now fairly sure you are a bad programmer.

You are probably not. Sometimes the leak is in the runtime itself. Node 20.11.1 shipped exactly this: a stream-subsystem regression that leaked memory under sustained high-throughput logging, and it did not exist in 20.10.0. If your logging library is Winston with a few custom transports and you produce a lot of logs, you were one of the people who spent a weekend blaming code that was fine.

This article is about the specific bug, but more usefully it is about the method: how to prove a memory leak lives in the runtime and not in your code, so you stop auditing closures and start bisecting versions. That skill outlives any single Node release.

The symptom: steady growth under sustained log volume

There is no thrown error here, which is what makes it so expensive to diagnose. The only signal is a resident-memory line that climbs and never recovers:

# RSS over 30 minutes of steady traffic, Node 20.11.1
t+0m    412 MB
t+10m   690 MB
t+20m   980 MB
t+30m  1240 MB   -> eventually OOMKilled, exit 137

The upstream issue (nodejs/node#51936) reported the trigger precisely: Winston with multiple custom transports, many logger instances, and a large volume of logs. The reporter confirmed it does not happen on v20.10.0 and that reverting one specific stream-subsystem commit in the v20.11.1 source resolved it. That is the shape of a runtime regression: version-specific, load-dependent, invisible until you have been running for a while.

Why stream-heavy logging is the canary

Logging libraries like Winston are, under the hood, stream plumbing. Each transport (console, file, HTTP, your custom one) is a writable stream, and the logger pipes records into all of them. A single request might write several log lines; a busy service writes thousands per second. That means the stream write path is one of the hottest code paths in the whole process.

Now think about what a leak in that path does. If each stream write retains a few bytes that should have been freed, most code paths would never notice. But a path executed thousands of times per second accumulates those bytes fast, and it never idles long enough for the pattern to look like anything but a monotonic climb. So stream-heavy logging is not specially cursed, it is just the highest-frequency user of the subsystem that regressed, which makes it the first place a stream leak becomes visible. If you run a low-log service on the same Node version you might never see it.

Step 1: Prove it is memory, and that it does not recover

Before touching versions, confirm you have a genuine leak and not just a large-but-bounded heap. Force a garbage collection and check whether memory comes back. Start the process with --expose-gc and log process.memoryUsage() before and after a manual global.gc(), or simply watch RSS over a long window under steady load. A leak keeps climbing after GC; a healthy heap plateaus. If it plateaus, stop here, you do not have a leak.

Step 2: Diff two heap snapshots to see who owns the retained memory

This is the step that ends the "is it me or the runtime" argument. Start your process with the inspector open:

node --inspect your-app.js
# then open chrome://inspect in Chrome and click "inspect"

In the DevTools Memory tab, take a heap snapshot. Let the service run under load for several minutes. Take a second snapshot. Then use the Comparison view between the two snapshots and sort by Delta. This shows you which object types grew between the two points in time.

Read the result carefully, because this is the whole diagnosis:

• If the growing objects are yours (your request contexts, your cache entries, your closures holding references), the leak is in your code. Fix it there and stop reading.
• If the growing objects are internal (stream buffers, Writable state, arrays inside Node's own modules) and your own object counts are flat, the leak is below your code. That points at the runtime or a native dependency.

For the 20.11.1 case, the retained growth sits in stream internals, not in Winston's or your objects. That is the fingerprint that tells you to stop auditing your closures.

Step 3: Bisect the Node version to isolate the regression

Once the snapshot says "not my code", find the exact version where the behaviour changed. You do not need to build Node from source for this; you bisect across released versions first, which is much faster.

Install a version manager and pin candidates

# with nvm
nvm install 20.10.0
nvm install 20.11.1

# run your reproduction load against each
nvm use 20.10.0 && node --inspect your-app.js   # memory flat
nvm use 20.11.1 && node --inspect your-app.js   # memory climbs

The moment you have a "good" version and a "bad" version with the same reproduction, you have proven a runtime regression and you have a workaround in the same breath: run the good version. If the two versions are several minor releases apart, halve the gap and test the middle version, exactly like git bisect but across release tags. For a truly precise answer, the upstream maintainers can bisect commits between two tags, which is how #51936 was narrowed to a single stream commit.

Pin the good version so it survives a deploy

Do not just nvm use on your laptop and forget. Pin it where the deploy will honour it, so an autoscaled instance or a teammate's machine does not silently pull the bad version:

// package.json
{
  "engines": {
    "node": "20.10.0"
  }
}

FROM node:20.10.0-slim

Add a short comment next to the pin explaining why, with a link to the issue, so nobody "upgrades to latest" in six months and reintroduces the leak without understanding what they undid.

Verification: memory plateaus on the pinned version

Run the exact same load test that produced the climbing graph, on the pinned good version, for at least as long as it took the leak to become obvious before. You are looking for a flat line after warm-up:

# RSS over 30 minutes, Node 20.10.0, same load
t+0m    398 MB
t+10m   430 MB
t+20m   428 MB
t+30m   431 MB   -> plateau, no OOM

A plateau under the same load that previously climbed is your proof. Keep the heap-snapshot comparison from Step 2 in the incident notes; it is the evidence that this was a runtime bug and not something your team introduced, which matters when the pin later looks like an unexplained constraint.

What people get wrong

"Just raise --max-old-space-size." Raising the heap limit does not fix a leak, it delays the crash. You still hit the ceiling, just later, and now the process holds more memory the whole time, so you can fit fewer instances per box. It is a legitimate stopgap to survive a night, never a fix.

"Restart the process periodically to clear it." A cron that restarts your service every few hours to dodge a leak is a confession, not a solution. It papers over a bug you have not identified and it makes a real future leak invisible because the periodic restart masks it. If you genuinely cannot fix the root cause yet, fine, but treat the restart as an alarm you owe yourself a fix for, not a design.

"It must be my code, the runtime is well-tested." Node is well-tested and it still ships stream and memory regressions in point releases, because sustained-load leaks are exactly the kind of bug unit tests do not catch. Assuming the runtime is infallible is what costs you the two days. Snapshot-diff first; let the evidence decide.

When it is still broken

If pinning the Node version does not stop the growth, widen the search:

• Suspect a native addon. Anything with a C++ binding (some database drivers, image processors, crypto libs) can leak below the JS heap where snapshots do not see it. Watch whether RSS climbs while the JS heap stays flat; that gap is native memory. Diff which addon versions changed.
• Reduce the reproduction. Strip the app to the smallest thing that still leaks: a loop that writes to your transports with no HTTP layer. A minimal reproduction is what gets a runtime bug fixed upstream, and it often reveals the culprit yourself.
• Check your transports for unbounded buffering. A custom transport that writes to a slow sink (a remote HTTP endpoint) without honouring backpressure will buffer records in memory under load. That is your bug, not the runtime, and it looks identical on the RSS graph. The general shape of this problem, retrying and buffering under load, is the same one covered in what actually helps a worker memory leak.
• Look for the same class of bug in async I/O. Stream and socket leaks in the runtime are a recurring genre; the TLS variant is documented in the asyncio SSL memory leak nobody warns you about, and the diagnostic method transfers directly.

The habit worth building is this: when memory climbs and your code review comes up empty, do not assume you missed something. Diff two heap snapshots, read whose objects are growing, and if the answer is "the runtime's", bisect the version. The pin buys you a stable service today, and the snapshot evidence protects you from someone quietly upgrading into the same leak tomorrow.