fix(daemon): hard-exit backstop when a graceful shutdown wedges - #74
Merged
Conversation
Incident follow-up #2. When cos's `claude --resume` froze on its exit screen, the daemon was left alive+orphaned (ppid=1) needing kill -9. Cause: cleanShutdown awaits server.close() then process.exit() with NO overall deadline. close()'s child-exit wait is internally bounded (~2s), but the outer promise can still hang indefinitely — socketServer.close()'s callback never fires for a lingering/ untracked socket, or eventWriter.flush() stalls — so the daemon never reaches process.exit() and lingers forever. Add a hard deadline (default 5s, PTY_SHUTDOWN_DEADLINE_MS-overridable) armed by cleanShutdown: if the graceful close() hasn't completed by the deadline, the daemon force-exits regardless AND SIGKILLs its child (PtyServer.forceKillChild) so a frozen child isn't left orphaned to init still alive. cleanShutdown is now idempotent too — SIGTERM/SIGINT/onExit/spawner-watchdog can overlap; only the first arms the deadline and drives close(), the rest get the same in-flight promise. Same class as #69/#72 but the frozen-child / stuck-close case. tests/shutdown-backstop.test.ts: a child that traps SIGHUP wedges the graceful path -> backstop force-exits the daemon and reaps the child (timing-independent proof: only the backstop's SIGKILL can kill a SIGHUP-trapping child; the graceful path only ever sends SIGHUP). A normal session still shuts down promptly, well under the deadline. All existing lifecycle tests (spawner-watchdog, kill-wait, exit-event-race, exit-signal, rm-kill-ephemeral, up-down) intact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up #2 from the cos-restart incident (guardrail #73 was #1). Guarantees a daemon can't get wedged alive + orphaned needing
kill -9.The wedge
When cos's
claude --resumefroze on its exit screen, the daemon was left alive, orphaned (ppid=1) — reclaiming it neededkill -9on both the daemon and the frozen claude.Root cause in the shutdown path (
server.tsentry point):close()'s child-exit wait is internally bounded (~2s), but the outer promise can hang indefinitely:socketServer.close(cb)'s callback never fires for a lingering/untracked socket, oreventWriter.flush()stalls on a frozen child.If the promise never resolves,
process.exit()is never reached → the daemon lingers forever.Fix
cleanShutdownnow arms a hard deadline (default 5s,PTY_SHUTDOWN_DEADLINE_MS-overridable). If gracefulclose()hasn't completed by the deadline, the daemon:process.exit(code)), andPtyServer.forceKillChild) so a frozen child isn't left orphaned to init still alive, andcleanShutdownis also now idempotent — SIGTERM / SIGINT / the child'sonExit/ the spawner watchdog can overlap; only the first arms the deadline and drivesclose(), the rest share the in-flight promise. (Aclose()rejection is covered too — the deadline is the net.)The normal fast path is untouched: on a clean shutdown
close()resolves in well under the deadline, the timer is cleared, and the daemon exits promptly as before.Tests —
tests/shutdown-backstop.test.tstrap "" HUPwedges the graceful path (SIGHUP is ignored, sochildExiteddrags). The backstop force-exits the daemon and the child ends up dead. This is a timing-independent proof: the graceful path only ever sends the child SIGHUP, so a SIGHUP-trapping child would survive as an orphan — only the backstop's SIGKILL can reap it. (deadline set to 300ms via env to force the path deterministically.)sleepsession SIGTERM'd exits promptly, nowhere near the deadline.Full suite: 1262 passed; the one red (
scrollback-fidelity) is a pre-existing load flake unrelated to this change — it passes 8/8 in isolation (confirmed twice); it starves under 96-file concurrent load. Flagging separately, not fixing in this PR (unrelated area). All shutdown/lifecycle-adjacent tests (spawner-watchdog, kill-wait, exit-event-race, exit-signal, rm-kill-ephemeral, up-down, events-emit — 58 tests) pass. Build/typecheck clean.Queue
Next (item #3, per your call): scrub/explicitly-set bus-identity env (
ST_AGENT/ST_ROOT) on restart/spawn — the deeper root cause (restart re-execs under the operator's shell env → wrong bus identity). Separate PR.