This repository was archived by the owner on Jul 24, 2026. It is now read-only.
convoy up/reconcile: recover missing/unhealthy dings + fix the respawn primitive (manifest-replay, not pty restart) (#82) - #94
Merged
Conversation
…n primitive (manifest-replay, not pty restart) (#82) Two linked recovery gaps, both surfaced by tonight's mass-ding-kill (14 dings died; agents needed a full restart): 1) host.respawn used 'pty restart -y', which is unusable for a headless supervisor (VERIFIED empirically in an isolated PTY_ROOT): - pty's stateful-agent guard makes 'pty restart' exit 1 on a role=agent session unless --force, so a dead permanent AGENT was never respawned -- reconcile only bumped results.failed. This is the root of issue #82. - otherwise 'pty restart' tries to ATTACH after the respawn, which hangs a non-TTY host. Replace it with manifest-replay via spawnDaemon: read the session's pty.toml def and re-spawn the verbatim command with the durable ST_ROOT/PTY_ROOT env -- the primitive convoy reload/spawnFromPtyFile already use and that convoy-rust independently adopted (pty rm + pty up). 2) reconcile was session-centric, so a ding whose process was killed AND whose record was GC'd is absent from the session list (nothing to respawn), and a killed-but-registered ding lost its strategy=permanent tag (pty kill strips it) so the permanent-respawn branch skipped it -- either way a LIVE agent was left with a DEAD ding until a full restart. Add an agent-centric DING-HEALTH pass (convoy up AND up --once): for each live agent whose manifest declares a ding, ensure the ding process is alive; replay it from the manifest if missing or dead. Health today = process-alive; a richer ding health signal (the Rust ding) plugs in at the same check. New host primitives: spawnManifestSession / readManifestDef / freeSession, shared by spawnFromPtyFile + respawn + the ding-health pass. New pure reconcile.dingHealthPlan (unit-tested, 10 cases). Live end-to-end proof against the real pty daemon: dead ding, GC'd/missing ding, and role=agent harness respawn all recover. Full suite 256/256 green.
myobie
force-pushed
the
ding-recovery-respawn-fix
branch
from
July 21, 2026 10:23
2caa0c6 to
5711423
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two linked recovery gaps, both surfaced by the mass-ding-kill (14 dings died; agents needed a full restart). Fixes issue #82 and the reconcile-recreates-missing/unhealthy-ding gap.
1.
host.respawnusedpty restart -y— unusable for a headless supervisor (VERIFIED empirically in an isolated PTY_ROOT):pty restartexit 1 on arole=agentsession unless--force, so a dead permanent AGENT was never respawned — reconcile only bumpedresults.failed. This is the root of Supervisor detects provider death but never replays the manifest — no recovery #82.pty restarttries to ATTACH after the respawn, which hangs a non-TTY host.spawnDaemon: read the session's pty.toml def and re-spawn the verbatim command with the durable ST_ROOT/PTY_ROOT env — the primitiveconvoy reload/spawnFromPtyFilealready use, and that convoy-rust independently adopted (pty rm + pty up).2. reconcile was session-centric, so a ding whose process was killed AND whose record was GC'd is absent from the session list (nothing to respawn), and a killed-but-registered ding lost its
strategy=permanenttag (pty killstrips it) so the permanent-respawn branch skipped it. Either way a LIVE agent was left with a DEAD ding. Added an agent-centric DING-HEALTH pass (convoy upANDup --once): for each live agent whose manifest declares a ding, ensure the ding process is alive; replay it from the manifest if missing or dead. Health today = process-alive; a richer ding health signal (the Rust ding) plugs in at the same check.New shared host primitives:
spawnManifestSession/readManifestDef/freeSession. New purereconcile.dingHealthPlan.Verification
dingHealthPlanunit cases).This branch is based on the pre-rename
@myobie/*main (0816121) — the shared clone it was built on still has@myobie/ptysibling deps, so I could verify here but could not rebase onto the renamed main (@compoundingtech/ptywouldn't resolve on this clone). The only reconciliation at merge is the import specifier@myobie/pty/client->@compoundingtech/pty/clientinhost.ts,reconcile.ts,reconcile.test.ts. @cos owns the fleet reinstall + sibling FFs, so please integrate during that pass (or tell me when the clone is renamed and I'll rebase + re-verify).The fix is already live on the fleet via the shared-clone working tree (on the @myobie base).