ProcPool never evicts a dead idle process from warmedProcQueue — a later job can be dispatched into a corpse

[jisang1213]

Summary

ProcPool never evicts an idle process from warmedProcQueue when it dies before being claimed by a real job — e.g. when the ping/pong watchdog (SupervisedProc) kills an unresponsive idle worker. A later launchJob() call can then dequeue and dispatch a real job straight into the already-dead process.

Confirmed present, byte-for-byte identical, in the latest published 1.7.0 as well as 1.6.3/1.6.4. Node.js SDK (@livekit/agents), dist/ipc/proc_pool.cjs.

Root cause

In ProcPool.procWatchTask() (src/ipc/proc_pool.ts / compiled dist/ipc/proc_pool.cjs):

await proc.initialize();
await this.warmedProcQueue.put({ proc, unlock: procUnlock });
procUnlockTransferred = true;
unlock();
initReleased = true;
} catch {
}
await proc.join();

Once a freshly-initialized idle process is queued into warmedProcQueue, procWatchTask just blocks on proc.join(). If the process dies here — e.g. SupervisedProc's ping/pong watchdog fires (pingTimeout, default 60s) because the idle worker never sends a pong — nothing removes the now-dead { proc, unlock } entry from warmedProcQueue.items. The numIdleProcesses slot it held also stays effectively "spent" until whoever eventually dequeues it releases it (see ProcPool.launchJob(): entry.unlock() is only called once something dequeues the entry).

Meanwhile ProcPool.launchJob() has no liveness check on dequeue:

async launchJob(info) {
  let proc;
  if (this.procMutex) {
    const entry = await this.warmedProcQueue.get();
    proc = entry.proc;
    entry.unlock();
  } else { ... }
  await proc.launchJob(info);
}

If a real job arrives and dequeues the stale entry, it gets dispatched into a process that was already killed (or is in the process of dying) — SupervisedProc.launchJob()'s only guard is this.proc?.connected, which on a Node ChildProcess can still read true in the window between kill() being called and the 'exit' event actually firing.

Net effect: an idle worker that goes briefly unresponsive can silently poison the pool for an unbounded amount of time — until some later job happens to be dispatched to it, at which point that job (a real user's call, in our case) experiences severe delays / hangs / timeouts, not a clean failure.

Reproduction

Isolated harness stubbing JobProcExecutor (no real child process, no network) to exercise ProcPool's control flow directly:

1. numIdleProcesses: 1 — pool warms exactly one idle worker.
2. Confirm it reaches warmedProcQueue (queued, procUnlockTransferred semantics as above).
3. Simulate the watchdog killing it (resolve the stubbed proc.join() promise) before anything claims it via launchJob().
4. Assert: the dead entry is still reachable via warmedProcQueue.items — i.e. a subsequent launchJob() would dequeue and dispatch into it.

Result on unpatched 1.6.3/1.7.0: dead entry stays in the queue indefinitely (assertion fails — bug confirmed).

Happy to share the harness script if useful — kept it out of this report to keep things minimal, since it doesn't depend on real LiveKit infra and can be dropped straight into a repo checkout.

Suggested fix

When proc.join() resolves inside procWatchTask, check whether the entry this task itself queued is still sitting unclaimed in warmedProcQueue.items (i.e. nobody ever called entry.unlock() for it via launchJob()). If so:

• Remove it from warmedProcQueue.items.
• Call its procUnlock() so the freed numIdleProcesses slot lets a fresh, healthy replacement spin up immediately, instead of the pool silently running short a worker until the next job stumbles into the corpse.

This is a small, local change confined to procWatchTask — doesn't touch launchJob(), Queue, or the mutex classes. Happy to open a PR with this if it'd help — wanted to raise the issue first in case there's context I'm missing about why eviction isn't done today (e.g. some invariant elsewhere I'm not seeing).

Environment

• @livekit/agents: 1.6.3, 1.6.4, and 1.7.0 (confirmed via a direct diff of the published dist/ipc/proc_pool.cjs across versions — file is byte-identical across all three for the relevant code path)
• Self-hosted worker (Fly.io), not LiveKit Cloud-managed hosting — noting this since it may be relevant to how often an idle process goes unresponsive long enough to trigger the watchdog (shared/burstable vCPU contention is a plausible contributing factor there), but the eviction gap itself is host-independent — it's in the pool bookkeeping, not the environment.