[BUG]: express.middleware spans retained unbounded via AsyncLocalStorage frame capture → heap OOM

[dwinrick-lever]

Tracer Version(s)

5.110.0 (also confirmed present on master / v5.111.0)

Node.js Version(s)

24.16.0 (also reproduces on 18.20.8)

Bug Report

Under sustained traffic, an Express app instrumented by dd-trace accumulates express.middleware spans in the JS heap without bound, eventually crashing with FATAL ERROR: … JavaScript heap out of memory. In a production service the post-GC live heap climbed ~linearly to the --max-old-space-size ceiling and OOM'd roughly daily; ~93% of retained DatadogSpan objects were express.middleware.

Root cause (traced through the source): the active span is stored with AsyncLocalStorage.enterWith({ ...store, span }) (not run()) — packages/dd-trace/src/plugins/tracing.js:268-269, packages/dd-trace/src/plugins/plugin.js:105-107, and per-middleware at packages/datadog-plugin-router/src/index.js:41. Because the HTTP request event is emitted synchronously on the connection's async-context frame, this writes { ...store, span } into that frame. Any never-released async resource created during request handling (a retained AsyncResource, an event listener added to a long-lived emitter and never removed, a subscription, a cache entry, etc.) captures that frame (AsyncContextFrame / legacy kResourceStore) and pins the entire middleware span chain for the life of that resource — independent of the HTTP socket. The span is finished correctly, but the captured store snapshot still references it, so it is never collected. Real handlers create such resources constantly (DB clients, pub/sub subscriptions, timers, listeners), so retained spans grow ~N-per-request unbounded.

Retainer chain from a heap snapshot (taken after the load client closed its sockets + GC), proving it is not socket- or timer-anchored:

AsyncResource [Symbol(context_frame)] → AsyncContextFrame → <array> → Object{ ...store, span } → DatadogSpan → (parent chain, ~20-deep middleware)

Reproduction Code

Full, self-contained reproducer: https://github.com/dwinrick-lever/dd-trace-express-middleware-span-leak — npm install && ./reproduce.sh

Express + ~20 no-op middlewares + dd-trace.init(); sustained requests where each creates one never-destroyed async resource (stand-in for a real subscription/listener). Measured:

config	retained per request	outcome
dd-trace ON	36,730–36,748 B/req (constant every batch, dead-linear heap climb, no plateau)	OOM
dd-trace OFF (same load)	90 B/req	bounded/flat 10+ min

• ~407× amplification by dd-trace. Constant bytes/request with no plateau ⇒ strictly unbounded.
• Crashes at both --max-old-space-size=256 (~18 s) and 512 (~22 s): the larger cap crashes later but still crashes — a bounded working set would plateau below 512 MB and never crash there.
• DD_TRACE_MIDDLEWARE_TRACING_ENABLED=false (or tracer.use('express', { middleware: false })) cuts retained spans ~96% and stops the growth.
• Reproduces on Node 24 (AsyncContextFrame default) and Node 18 (legacy async_hooks); --no-async-context-frame does not help.

Error Logs

<--- Last few GCs --->
[1:0x...] Mark-Compact ... allocation failure; scavenge might not succeed
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Tracer Config

Default: require('dd-trace').init() with the express integration enabled; server uses HTTP keep-alive (Node default). No unusual config required to trigger it.
Workaround: DD_TRACE_MIDDLEWARE_TRACING_ENABLED=false or tracer.use('express', { middleware: false }).

Operating System

Linux (mechanism is not OS-specific).

Bundling

No Bundling

[bvizy-reshop]

@BridgeAR Any update when a fix can be released? The underlying issue is creating a memory leak in our application as well