Skip to content

fix(compression): terminate idle workers instead of dropping them - #12542

Open
pacocartones wants to merge 2 commits into
diegosouzapw:release/v3.8.51from
pacocartones:fix/compression-terminate-idle-workers
Open

fix(compression): terminate idle workers instead of dropping them#12542
pacocartones wants to merge 2 commits into
diegosouzapw:release/v3.8.51from
pacocartones:fix/compression-terminate-idle-workers

Conversation

@pacocartones

Copy link
Copy Markdown
Contributor

Summary

  • CompressionWorkerPool.finish() (open-sse/services/compression/compressionWorkerPool.ts:188) evicted an idle worker with this.remove(slot, false), and remove() (:198-203) only called worker.terminate() if (terminate). The idle path therefore deleted the slot from the pool set but never terminated the thread; compressionWorker.ts:13 keeps a parentPort.on("message") listener, so the worker's event loop and V8 isolate stayed alive forever, and dispatch() (:156) spawned a replacement because workers.size < size. The OMNI_COMPRESSION_WORKERS cap held in bookkeeping only: every idle cycle leaked one live thread. This is the mechanism confirmed on fix(backend): memory spike causing OOM (~1.5GB → ~16GB) when routing requests via combo #11804 (request-driven thread growth, ~90 threads parked in ep_poll, RSS far outside the main heap) and the one-line patch the reporter measured on a live instance: 57 leaked workers / 72 threads / 5.77 GB climbing before, 0 leaked / 15 threads / 846 MB flat after.
  • remove() now always terminates the thread. The terminate parameter is removed rather than flipped at the idle call site: close() and fail() already passed true, so the parameter only existed to express the buggy case. A short comment on remove() records why dropping the slot alone is not enough. close(), fail(), timeouts and the queue/dispatch logic are otherwise unchanged; terminate() failures are still swallowed as before.
  • Out of scope, on purpose: the callLogArtifactWorker hypothesis raised mid-thread (later marked unconfirmed by its author), the six bundled copies of the pool in the Next.js server chunks (they pick this change up at build time), and any change to the idle timeout or pool size defaults.

Related Issues

Validation

  • Change type: other (compression worker pool, open-sse/)
  • Focused tests and category gates from the golden path: tests/unit/compression/compression-worker.test.ts + tests/unit/compression/compression-worker-file-resolution.test.ts 17/17 (4 suites), node scripts/check/check-complexity-ratchets.mjs --base-ref origin/release/v3.8.51 OK (0 violations, base 0), npm run check:changelog-integrity OK, npm run typecheck:core exit 0
  • npm run lint — eslint run on the two touched source files only (--suppressions-location config/quality/eslint-suppressions.json), exit 0; the repository-wide command was not run
  • Reconciled with the current active release base release/v3.8.51 (1a0375fba); focused checks rerun afterward
  • Production-code changes include a new or updated automated test in this PR
  • SonarQube is temporarily opt-in while the private project has no quota; it is not a PR gate.

Red/green: the new case fails on the base with idle eviction must terminate the worker thread: 0 !== 1 and passes with the change (three consecutive runs, 8/8 in the file).

Tests Added Or Updated

  • tests/unit/compression/compression-worker.test.ts (1 new case, "terminates an idle worker instead of only dropping it from the pool"): spies on Worker.prototype.terminate and Worker.prototype.postMessage around a local { size: 1, idleMs: 50 } pool, runs one job, waits past the idle window, and asserts that exactly one worker was spawned, that terminate() was called once, and that the process does not retain the worker's MessagePort (process.getActiveResourcesInfo()). The finally block restores both prototypes, closes the pool, and terminates any worker the pool forgot, so a future regression fails the assertion instead of keeping the test runner alive. The seven existing cases are untouched.
  • No stryker.conf.json change: open-sse/services/compression/ is not in the mutate list.

Coverage Notes

  • open-sse/services/compression/compressionWorkerPool.ts: the idle path of finish() and the unconditional terminate() in remove() are exercised by the new case; close() by the new case's finally and the existing timeout case; fail() by the existing "fails open" case. No touched file lost coverage.

Reviewer Notes

  • Behavioural change is limited to idle eviction: the thread is now terminated at the moment it is removed from the set, which is what close() and fail() already did. Nothing about job routing changes, so a request arriving after eviction still gets a freshly spawned worker exactly as before.
  • Anyone patching an installed 3.8.50/3.8.51 bundle by hand should note that the minified idle call site appears in five files (four Next.js runtime chunks plus open-sse/mcp-server/server.js), as documented on the issue; a normal build from this branch covers all of them.
  • No migrations, feature flags, or env changes.

`CompressionWorkerPool.finish()` evicted an idle worker with
`remove(slot, false)`, which deleted the slot from the pool set but never
called `worker.terminate()`. The worker keeps a `parentPort` listener, so
its event loop and V8 isolate stayed alive forever, and `dispatch()` then
spawned a replacement because `workers.size < size`. The size cap held in
bookkeeping only: every idle cycle leaked one live thread.

`remove()` now always terminates the thread; the `terminate` parameter is
gone because every remaining caller wanted it. The regression test spies
on `Worker.prototype.terminate` around a `{ size: 1, idleMs: 50 }` pool
and checks the idle eviction terminates the thread and does not retain
its MessagePort.

Closes diegosouzapw#11804
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(backend): memory spike causing OOM (~1.5GB → ~16GB) when routing requests via combo

1 participant