Chroma-mcp orphan leak across worker generations: live orphan's registry entry is overwritten without a kill (682 orphans / 21.7 GB swap)

[eslonaguiar]

Summary

Every time the worker dies without a graceful shutdown (SIGKILL, crash, OOM kill), its chroma-mcp subprocess tree survives — and the next worker generation silently discards the only record of it. One leaked uvx → uv → python chroma-mcp tree per ungraceful worker death, unbounded across restarts.

On an 8 GB macOS machine (Apple Silicon, macOS 25.5.0, claude-mem 13.11.0) this accumulated 682 orphaned processes (341 uvx wrappers + 341 python instances) in ~15 hours, filling 21.7 GB of swap and driving the load average to 293 until macOS started showing the "system has run out of application memory" dialog. Memory pressure makes workers die more often, which leaks more orphans — a self-reinforcing spiral.

Likely a major contributor to the process-accumulation reports in #3270 and related to the leak scenarios described in #3298.

Root cause

The in-process lifecycle in ChromaMcpManager is well defended (disposeCurrentSubprocess(), tree-kill on failed connect, onclose sweep — #2313). The hole is at the worker-process-death boundary, and it takes three steps:

1. Worker dies ungracefully. The chroma-mcp child is not tied to the worker's lifetime (no process group ownership), so it survives and re-parents to init/launchd. supervisor.json still records its PID under the chroma-mcp id.
2. Next worker boots. ProcessRegistry.initialize() loads the persisted entries and calls pruneDeadEntries() — which only removes entries whose PID is dead (if (isPidAlive(info.pid)) continue;). The orphan's entry is alive, so it is kept... but nothing acts on it.
3. New worker connects. connectInternal() → registerManagedProcess() → registry.register('chroma-mcp', …) overwrites the old entry with the new child's PID. The orphan's PID — which the registry knew about moments earlier — is forgotten without ever being signaled.

The writer lock does not help: it records the worker's PID, not the chroma child's, so isChromaWriterLockLive() correctly reports the lock stale (owner dead) and the new worker proceeds — right past the still-running previous chroma-mcp.

Reproduction (macOS/Linux)

# 1. Let the worker connect chroma-mcp, note the subprocess PID
#    (supervisor.json → processes["chroma-mcp"].pid)
# 2. Simulate a crash:
kill -9 <worker-pid>
# 3. Observe: chroma-mcp tree survives with PPID 1, entry still in supervisor.json
# 4. Let hooks (or `worker-service.cjs start`) bring up a new worker and
#    trigger any chroma call
# 5. Observe: a second chroma-mcp tree is now running; the registry entry
#    was overwritten; the orphan from step 3 lives forever

Repeat 2–5 and the orphan count grows by one each cycle. Under real-world conditions (OOM pressure, version-switch restarts, #3298-style thrash) this happens continuously.

Evidence from the affected machine

• 341 orphaned uv tool uvx --python 3.13 … --from chroma-mcp==0.2.6 chroma-mcp --client-type persistent --data-dir ~/.claude-mem/chroma wrappers, PPID 1, plus their 341 python children.
• All spawned across ~15 h, matching worker restart churn.
• vm.swapusage: 21.7 GB used of 22.5 GB; load average 293 on an 8-core machine.
• Force-quit dialog attributed 32.71 GB to the Claude app (sum of the orphan trees).

Proposed fix

Before spawning a replacement subprocess, consult the persisted registry entry for chroma-mcp:

• entry PID dead → just unregister it;
• entry PID alive and its command line verifiably a chroma-mcp launcher (ps -p <pid> -o command= — guards against PID reuse) → killProcessTree(pid) then unregister;
• anything unexpected → best-effort, never block the connect path.

I have this implemented and validated (96/96 in the chroma/supervisor test suites, plus a live kill-9 reproduction on the affected machine where the new worker logged Reaping orphaned chroma-mcp left by a previous worker generation {pid=…} and the orphan was gone). PR incoming, will link it here.

Note: the same boot-time pattern may deserve consideration for stale-but-alive sdk entries too (#3270 mentions 58+ accumulated claude-mem processes), but this issue and the accompanying PR intentionally scope to the chroma-mcp leak.

---

🇧🇷 Versão em português (resumo)

Resumo

Toda vez que o worker morre sem shutdown gracioso (SIGKILL, crash, OOM), o subprocesso chroma-mcp sobrevive — e a geração seguinte do worker descarta silenciosamente o único registro dele. Resultado: uma árvore uvx → uv → python chroma-mcp vazada por morte abrupta do worker, sem limite.

Num Mac de 8 GB (macOS 25.5.0, claude-mem 13.11.0) isso acumulou 682 processos órfãos em ~15 horas (341 wrappers uvx + 341 instâncias python), enchendo 21,7 GB de swap, com load average de 293 e o alerta do macOS de "memória de aplicativos esgotada". A pressão de memória mata mais workers, que vazam mais órfãos — uma espiral que se retroalimenta.

Causa raiz

O ciclo de vida dentro do processo do worker é bem defendido (#2313). O buraco está na fronteira entre gerações, em três passos:

1. Worker morre abruptamente. O filho chroma-mcp não está atrelado à vida do worker, sobrevive e reparenta para o init/launchd. O supervisor.json ainda registra o PID dele.
2. Próximo worker inicia. pruneDeadEntries() só remove entradas de PIDs mortos — a entrada do órfão (vivo) fica no registro, mas nada age sobre ela.
3. Novo worker conecta. registerManagedProcess() sobrescreve a entrada com o PID do filho novo. O PID do órfão — que o registro conhecia segundos antes — é esquecido sem nunca receber sinal.

Correção proposta

Antes de cada spawn, consultar a entrada persistida do chroma-mcp: PID morto → só remove do registro; PID vivo e comprovadamente um chroma-mcp (checagem da linha de comando via ps, contra reuso de PID) → tree-kill e remove. Tudo best-effort: nunca bloqueia a conexão.

Implementado e validado (96/96 testes + reprodução real com kill -9 na máquina afetada). PR: #3302.

[qihan-bot]

Same failure mode on Linux (Ubuntu 24.04, plugin 13.11.0 — current latest), with forensics confirming the overwrite-without-kill mechanism described here:

• Found 105 chroma-mcp processes (52 uv tool uvx … chroma-mcp --client-type persistent wrapper+python pairs), totaling ~10.3 GB RSS / ~3,400 threads. All but the newest pair were orphaned (re-parented to PID 1 or the systemd --user manager).
• Timeline: during a memory-pressure burst, a new pair appeared every 10–20 s for ~14 minutes — a worker restart storm. supervisor.json ended up recording only the last generation (its pid/startedAt match the newest surviving pair); every earlier generation kept running untracked. Note supervisor.json already records pgid, so a kill-before-respawn has everything it needs.
• Knock-on damage: the leaked RSS pushed the machine into allocation failures — concurrent Claude Code CLI processes started dying with SIGABRT (exit code -6) and shells got EAGAIN on fork, which caused more worker restarts → more leaked generations. It's a feedback loop: the leak manufactures the memory pressure that triggers the next leak.

Workaround we're running until this is fixed — a systemd user timer (every 30 min) that reaps only orphaned chroma-mcp processes (parent = PID 1 or systemd --user, older than 5 min; two passes so the python child of a killed uvx wrapper is caught as well). Chroma data is persisted on disk, so killing idle orphaned servers is safe. Freed ~10 GB immediately and the SIGABRT crashes stopped.

~/.local/bin/chroma-orphan-reaper + systemd units

#!/usr/bin/env bash
# chroma-orphan-reaper — kill orphaned claude-mem chroma-mcp processes.
set -u

USER_MGR=$(pgrep -u "$(id -u)" -o -f '^/usr/lib/systemd/systemd --user$' 2>/dev/null || true)

reap_pass() {
    local killed=0
    for pid in $(pgrep -u "$(id -u)" -f 'chroma-mcp --client-type'); do
        # Only real chroma-mcp processes: the `uv tool uvx ... chroma-mcp`
        # wrapper (exe=uv) or the python server it spawns — not shells that
        # merely mention the string in their cmdline.
        exe=$(readlink "/proc/$pid/exe" 2>/dev/null) || continue
        case "$exe" in *python*|*/uv) ;; *) continue ;; esac

        ppid=$(ps -o ppid= -p "$pid" 2>/dev/null | tr -d ' ') || continue
        [ -n "$ppid" ] || continue
        if [ "$ppid" = "1" ] || { [ -n "$USER_MGR" ] && [ "$ppid" = "$USER_MGR" ]; }; then
            age=$(ps -o etimes= -p "$pid" 2>/dev/null | tr -d ' ')
            if [ "${age:-0}" -gt 300 ]; then
                kill "$pid" 2>/dev/null && killed=$((killed + 1))
            fi
        fi
    done
    echo "$killed"
}

k1=$(reap_pass)
# Second pass: children of just-killed wrappers re-parent to PID 1 / user mgr
sleep 3
k2=$(reap_pass)

total=$((k1 + k2))
if [ "$total" -gt 0 ]; then
    echo "$(date -Is) reaped $total orphaned chroma-mcp process(es)"
fi

# ~/.config/systemd/user/chroma-reaper.service
[Unit]
Description=Reap orphaned claude-mem chroma-mcp processes

[Service]
Type=oneshot
ExecStart=%h/.local/bin/chroma-orphan-reaper

# ~/.config/systemd/user/chroma-reaper.timer
[Unit]
Description=Reap orphaned chroma-mcp every 30 minutes

[Timer]
OnBootSec=10min
OnUnitActiveSec=30min

[Install]
WantedBy=timers.target

Enable with: systemctl --user daemon-reload && systemctl --user enable --now chroma-reaper.timer

Suggested fixes (any one of these breaks the loop):

1. Supervisor: kill-before-respawn — before spawning a new chroma-mcp, kill the previously recorded pgid from supervisor.json.
2. Worker startup reconciliation — on worker start, sweep for chroma-mcp processes (matching the uvx cmdline against the plugin's data dir) that don't match supervisor.json, and kill them.
3. Orphan self-termination — spawn chroma-mcp with PR_SET_PDEATHSIG (Linux) or make the wrapper exit when stdin hits EOF, so a dead worker can't strand its child.