Skip to content
This repository was archived by the owner on Jul 24, 2026. It is now read-only.

convoy up/reconcile: recover missing/unhealthy dings + fix the respawn primitive (manifest-replay, not pty restart) (#82) - #94

Merged
myobie merged 1 commit into
mainfrom
ding-recovery-respawn-fix
Jul 21, 2026
Merged

convoy up/reconcile: recover missing/unhealthy dings + fix the respawn primitive (manifest-replay, not pty restart) (#82)#94
myobie merged 1 commit into
mainfrom
ding-recovery-respawn-fix

Conversation

@myobie

@myobie myobie commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

What

Two linked recovery gaps, both surfaced by the mass-ding-kill (14 dings died; agents needed a full restart). Fixes issue #82 and the reconcile-recreates-missing/unhealthy-ding gap.

1. host.respawn used pty restart -y — unusable for a headless supervisor (VERIFIED empirically in an isolated PTY_ROOT):

  • pty's stateful-agent guard makes pty restart exit 1 on a role=agent session unless --force, so a dead permanent AGENT was never respawned — reconcile only bumped results.failed. This is the root of Supervisor detects provider death but never replays the manifest — no recovery #82.
  • Otherwise pty restart tries to ATTACH after the respawn, which hangs a non-TTY host.
  • Replaced with manifest-replay via spawnDaemon: read the session's pty.toml def and re-spawn the verbatim command with the durable ST_ROOT/PTY_ROOT env — the primitive convoy reload/spawnFromPtyFile already use, and that convoy-rust independently adopted (pty rm + pty up).

2. reconcile was session-centric, so a ding whose process was killed AND whose record was GC'd is absent from the session list (nothing to respawn), and a killed-but-registered ding lost its strategy=permanent tag (pty kill strips it) so the permanent-respawn branch skipped it. Either way a LIVE agent was left with a DEAD ding. Added an agent-centric DING-HEALTH pass (convoy up AND up --once): for each live agent whose manifest declares a ding, ensure the ding process is alive; replay it from the manifest if missing or dead. Health today = process-alive; a richer ding health signal (the Rust ding) plugs in at the same check.

New shared host primitives: spawnManifestSession / readManifestDef / freeSession. New pure reconcile.dingHealthPlan.

Verification

  • Full suite 256/256 green (10 new dingHealthPlan unit cases).
  • Live end-to-end proof against the real pty daemon: dead ding, GC'd/missing ding, and role=agent harness respawn all recover.

⚠️ Merge note (rename base)

This branch is based on the pre-rename @myobie/* main (0816121) — the shared clone it was built on still has @myobie/pty sibling deps, so I could verify here but could not rebase onto the renamed main (@compoundingtech/pty wouldn't resolve on this clone). The only reconciliation at merge is the import specifier @myobie/pty/client -> @compoundingtech/pty/client in host.ts, reconcile.ts, reconcile.test.ts. @cos owns the fleet reinstall + sibling FFs, so please integrate during that pass (or tell me when the clone is renamed and I'll rebase + re-verify).

The fix is already live on the fleet via the shared-clone working tree (on the @myobie base).

…n primitive (manifest-replay, not pty restart) (#82)

Two linked recovery gaps, both surfaced by tonight's mass-ding-kill (14 dings
died; agents needed a full restart):

1) host.respawn used 'pty restart -y', which is unusable for a headless
   supervisor (VERIFIED empirically in an isolated PTY_ROOT):
   - pty's stateful-agent guard makes 'pty restart' exit 1 on a role=agent
     session unless --force, so a dead permanent AGENT was never respawned --
     reconcile only bumped results.failed. This is the root of issue #82.
   - otherwise 'pty restart' tries to ATTACH after the respawn, which hangs a
     non-TTY host.
   Replace it with manifest-replay via spawnDaemon: read the session's pty.toml
   def and re-spawn the verbatim command with the durable ST_ROOT/PTY_ROOT env
   -- the primitive convoy reload/spawnFromPtyFile already use and that
   convoy-rust independently adopted (pty rm + pty up).

2) reconcile was session-centric, so a ding whose process was killed AND whose
   record was GC'd is absent from the session list (nothing to respawn), and a
   killed-but-registered ding lost its strategy=permanent tag (pty kill strips
   it) so the permanent-respawn branch skipped it -- either way a LIVE agent was
   left with a DEAD ding until a full restart. Add an agent-centric DING-HEALTH
   pass (convoy up AND up --once): for each live agent whose manifest declares a
   ding, ensure the ding process is alive; replay it from the manifest if
   missing or dead. Health today = process-alive; a richer ding health signal
   (the Rust ding) plugs in at the same check.

New host primitives: spawnManifestSession / readManifestDef / freeSession,
shared by spawnFromPtyFile + respawn + the ding-health pass. New pure
reconcile.dingHealthPlan (unit-tested, 10 cases). Live end-to-end proof against
the real pty daemon: dead ding, GC'd/missing ding, and role=agent harness
respawn all recover. Full suite 256/256 green.
@myobie
myobie force-pushed the ding-recovery-respawn-fix branch from 2caa0c6 to 5711423 Compare July 21, 2026 10:23
@myobie
myobie merged commit 2d86cb3 into main Jul 21, 2026
1 check passed
@myobie
myobie deleted the ding-recovery-respawn-fix branch July 21, 2026 10:39
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant