Skip to content

fix(daemon): hard-exit backstop when a graceful shutdown wedges - #74

Merged
myobie merged 1 commit into
mainfrom
feat/daemon-shutdown-backstop
Jul 12, 2026
Merged

fix(daemon): hard-exit backstop when a graceful shutdown wedges#74
myobie merged 1 commit into
mainfrom
feat/daemon-shutdown-backstop

Conversation

@myobie

@myobie myobie commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

Follow-up #2 from the cos-restart incident (guardrail #73 was #1). Guarantees a daemon can't get wedged alive + orphaned needing kill -9.

The wedge

When cos's claude --resume froze on its exit screen, the daemon was left alive, orphaned (ppid=1) — reclaiming it needed kill -9 on both the daemon and the frozen claude.

Root cause in the shutdown path (server.ts entry point):

cleanShutdown(code) -> server.close() -> process.exit(code)   // no overall deadline

close()'s child-exit wait is internally bounded (~2s), but the outer promise can hang indefinitely:

  • socketServer.close(cb)'s callback never fires for a lingering/untracked socket, or
  • eventWriter.flush() stalls on a frozen child.

If the promise never resolves, process.exit() is never reached → the daemon lingers forever.

Fix

cleanShutdown now arms a hard deadline (default 5s, PTY_SHUTDOWN_DEADLINE_MS-overridable). If graceful close() hasn't completed by the deadline, the daemon:

  1. force-exits regardless (process.exit(code)), and
  2. SIGKILLs its child (PtyServer.forceKillChild) so a frozen child isn't left orphaned to init still alive, and
  3. best-effort clears the pid/sock so nothing points at the vanished daemon.

cleanShutdown is also now idempotent — SIGTERM / SIGINT / the child's onExit / the spawner watchdog can overlap; only the first arms the deadline and drives close(), the rest share the in-flight promise. (A close() rejection is covered too — the deadline is the net.)

The normal fast path is untouched: on a clean shutdown close() resolves in well under the deadline, the timer is cleared, and the daemon exits promptly as before.

Tests — tests/shutdown-backstop.test.ts

  • Backstop fires + reaps: a child that trap "" HUP wedges the graceful path (SIGHUP is ignored, so childExited drags). The backstop force-exits the daemon and the child ends up dead. This is a timing-independent proof: the graceful path only ever sends the child SIGHUP, so a SIGHUP-trapping child would survive as an orphan — only the backstop's SIGKILL can reap it. (deadline set to 300ms via env to force the path deterministically.)
  • Normal shutdown unaffected: a plain sleep session SIGTERM'd exits promptly, nowhere near the deadline.

Full suite: 1262 passed; the one red (scrollback-fidelity) is a pre-existing load flake unrelated to this change — it passes 8/8 in isolation (confirmed twice); it starves under 96-file concurrent load. Flagging separately, not fixing in this PR (unrelated area). All shutdown/lifecycle-adjacent tests (spawner-watchdog, kill-wait, exit-event-race, exit-signal, rm-kill-ephemeral, up-down, events-emit — 58 tests) pass. Build/typecheck clean.

Queue

Next (item #3, per your call): scrub/explicitly-set bus-identity env (ST_AGENT/ST_ROOT) on restart/spawn — the deeper root cause (restart re-execs under the operator's shell env → wrong bus identity). Separate PR.

Incident follow-up #2. When cos's `claude --resume` froze on its exit screen,
the daemon was left alive+orphaned (ppid=1) needing kill -9. Cause: cleanShutdown
awaits server.close() then process.exit() with NO overall deadline. close()'s
child-exit wait is internally bounded (~2s), but the outer promise can still hang
indefinitely — socketServer.close()'s callback never fires for a lingering/
untracked socket, or eventWriter.flush() stalls — so the daemon never reaches
process.exit() and lingers forever.

Add a hard deadline (default 5s, PTY_SHUTDOWN_DEADLINE_MS-overridable) armed by
cleanShutdown: if the graceful close() hasn't completed by the deadline, the
daemon force-exits regardless AND SIGKILLs its child (PtyServer.forceKillChild)
so a frozen child isn't left orphaned to init still alive. cleanShutdown is now
idempotent too — SIGTERM/SIGINT/onExit/spawner-watchdog can overlap; only the
first arms the deadline and drives close(), the rest get the same in-flight
promise. Same class as #69/#72 but the frozen-child / stuck-close case.

tests/shutdown-backstop.test.ts: a child that traps SIGHUP wedges the graceful
path -> backstop force-exits the daemon and reaps the child (timing-independent
proof: only the backstop's SIGKILL can kill a SIGHUP-trapping child; the graceful
path only ever sends SIGHUP). A normal session still shuts down promptly, well
under the deadline. All existing lifecycle tests (spawner-watchdog, kill-wait,
exit-event-race, exit-signal, rm-kill-ephemeral, up-down) intact.
@myobie
myobie merged commit 1e40d53 into main Jul 12, 2026
1 check passed
@myobie
myobie deleted the feat/daemon-shutdown-backstop branch July 12, 2026 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant