Reap orphaned Chrome processes to stop exhausting the PID limit

[moufmouf]

The problem

The staging monitoring pod was reporting test failures that had nothing to do with the test. Playwright was surfacing:

Error: browserType.launch: Target page, context or browser has been closed
[pid=650176][err] FATAL:zygote_host_impl_linux.cc(184)] Check failed: process.IsValid().
                  Failed to launch zygote process

That's Chromium failing to fork(). The pod had been up 19 days:

pids.current  18560
pids.max      18597     <- 37 PIDs left
ps -eo stat   18515 Z   <- all parented to PID 1

Chromium leaves helper processes behind on every run. They get re-parented to PID 1, which was npm run start — npm is not an init and never calls wait(), so the orphans stayed as zombies forever, each holding a PID slot.

Note that the pod stayed Running 1/1 the entire time. It degraded silently over weeks.

The fix

Dockerfile — install tini and make it the entrypoint, so PID 1 actually reaps orphans.

src/index.ts — the timeout path was orphaning browsers at the source. exec("npm run test") spawns sh -c ..., so childProcess.kill() killed only that shell and cut npm → playwright → chrome loose. Switched to spawn with detached: true so each run gets its own process group, and the kill now takes the whole tree via process.kill(-pid, 'SIGKILL').

Moving from exec to spawn also means streaming stdout/stderr as they arrive rather than buffering the whole run.

Verification

npm run typecheck passes. Staging was restarted to clear the backlog (zombies can't be reaped without an init — only a restart clears them), and after one run on the current image:

pids.current  47      (was 18560)
2 Z                   <- 2 zombies per run, the leak in miniature
1 passed (6.7s)

2 zombies/run × 288 runs/day at the default 300s interval is how it reached 18,515 in ~19 days. The restart buys a few weeks; this image change is the actual fix.

Not addressed here

helm/values.yaml has resources: {}, so the pod is unlimited. A memory limit would have turned this into a loud OOMKill weeks earlier instead of a silently degrading pod. Worth a follow-up.

🤖 Generated with Claude Code