Worker exits when an expectsReply escalation times out; lane-runner does not relaunch it and registry/lane state stay 'running'

[HenryLach]

Summary

A worker's escalate_to_supervisor call with expectsReply: true blocks the worker polling for a reply keyed to the escalation id. When the wait times out with no reply, the worker session exits. Nothing relaunches it: the lane-runner does not respawn the worker, and the runtime registry and lane state stay running with frozen timestamps. Supervisor steers then fail (send_agent_message has no live process to deliver to), and the supervisor has no tool that re-engages the lane short of orch_resume — which, in the observed run, was itself blocked (see the companion issue on replacement-supervisor resume).

Observed in penster batch henrylach-20260905T210935 (TP-2047, single wave, single lane), 2026-09-06.

Sequence

1. The worker hit a plan-gate REVISE at the round-2 cap and, per the operator's standing hold rule, entered a hard hold and escalated to the supervisor for a ruling: escalation id 1788657826267-4ee58, expectsReply: true.
2. The supervisor session that should have replied was itself wedged on a provider-side 400 (thinking-block mismatch) and was being replaced; no reply arrived inside the worker's wait window.
3. The worker's poll timed out and the process (pid 59452) exited. Get-Process returned nothing.
4. .pi/runtime/<batch>/registry.json and lanes/lane-1.json remained status: running, updatedAt 1788657826268 (the moment after the escalation call), for the next 17+ minutes. batch-state.json stayed phase: executing.
5. The replacement supervisor's send_agent_message refused (no live consumer); a manual inbox file drop was inert (no process to drain it; an escalation-wait expects a replyTo-keyed reply, not a fresh steer). orch_pause was accepted but inert (nothing running to honor it).
6. Recovery required a manual batch-state.json phase edit plus a fresh supervisor session plus orch_resume (precedence-4 re-execute in the existing worktree). Committed work survived; the hold-time commits (hold(TP-2047):) were intact.

Why this matters

The hard hold is the operator-sanctioned posture for a cap escalation (#627). A worker that holds correctly and then dies because the ruling took longer than its poll timeout converts a governance pause into a dead lane with stale running state — the worst of both: no work, no signal, and a registry that lies about it.

Expected

• A timed-out expectsReply escalation must not terminate the worker. Options: the worker re-enters its hold loop and re-polls (bounded by the stall limit, which the hold should suspend per Cap-ruling hold: runtime must block .DONE and merge while an operator ruling is pending (.PENDING-RULING) #627); or the lane-runner treats the exit as a hold-safe exit and preserves a held marker rather than nothing.
• If the worker process does exit for any reason while the task is non-terminal, the lane-runner (or the engine monitor) must reconcile the registry/lane state to exited within one poll interval and surface a supervisor alert (worker-exited-while-held or similar), instead of leaving running frozen.
• send_agent_message to a dead worker should return a distinct, actionable error naming the dead pid and last-seen timestamp, not a generic refusal.

Relation to other issues

• Cap-ruling hold: runtime must block .DONE and merge while an operator ruling is pending (.PENDING-RULING) #627 (held state): a first-class held state with no iteration/stall progression would make the wait indefinite by design and remove the timeout-exit path entirely.
• Replacement supervisor cannot orch_resume the batch it inherits: activation imports phase 'executing' into memory and the in-memory guard overrides disk eligibility #631 (companion, filed together): once the worker was dead, the replacement supervisor could not resume the batch it inherited.
• supervisor_takeover destroys the paused lane: task marked skipped, batch completed 0/1, worktree+branch removed, uncommitted work lost #628 / orch_retry_task resets the task record but not the v2 segment record, so orch_resume skips the wave (single-wave batch cannot be re-driven after a finalize-gate refusal) #629: the recovery deliberately avoided supervisor_takeover, orch_retry_task; orch_resume was the correct primitive once reachable.

[HenryLach]

Tier-1 fix landed (commit 27118a70; local-deployed). escalate_to_supervisor is fire-and-forget, so a correctly-holding worker ends its turn and exits — and the lane relaunched it with the "you exited prematurely — work continuously" nag, the exact prompt that pushes a holding worker toward self-release.

Now: a clean exit after an unanswered escalation is a hold exit — not a stall, relaunched with a hold-resume prompt (act on a delivered ruling; never proceed past the hold / self-approve / .DONE), bounded to 3 relaunches, then the task fails with an explicit Hold unresolved reason + supervisor alert carrying the escalation id, instead of a frozen running lane. A ruling delivered as a steer, or consumed by the exit-intercept, releases the hold (reply watermark by message timestamp). send_agent_message to a dead-pid agent now returns your requested distinct error (pid, last registry update, resume guidance).

Engine-dead case (yours): covered by #631 — takeover summary names the dead worker, ownership gate lets the replacement resume, reconciliation re-executes in the existing worktree.

Kept open for the design follow-up you and #627 describe: a first-class held state with an in-tool waitForReply (hold-aware stall monitor, reply correlation, hold blocks completion). Sage's design review was firm that the wait needs that durable hold record rather than a bolt-on, so it goes with #627.

[HenryLach]

For agents reading this issue — what you will observe after the Tier-1 fix (PR #632, v0.30.6)

Worker

• After escalate_to_supervisor, holding is now safe: ending your turn without progress is recorded as a Hold exit (STATUS log), not a stall. You will be relaunched with a ⏸️ YOU ARE ON HOLD awaiting a supervisor ruling prompt (not the "you exited prematurely — work continuously" nag). If a supervisor reply arrives as a steering message at session start, act on it. If not: do not proceed past the hold, self-approve, write .DONE, or re-send the escalation; optionally notify_supervisor(replyTo=<escalation id>) with a one-line status, then end your turn. The runtime relaunches you up to 3 times.
• After 3 unanswered relaunches the task fails with Hold unresolved: escalation <id> … and STATUS ⏸️ Held — ruling outstanding. Your committed work is preserved in the worktree.

Supervisor

• The Hold unresolved alert (category task-failure, context.messageId = escalation id) is a governance signal: rule on the escalation, then orch_retry_task + orch_resume(force=true); the relaunched worker receives your reply as a steer.
• Your reply releases the hold whether delivered as a steer mid-session or consumed by the exit-intercept (reply watermark by message timestamp).
• send_agent_message to a worker whose process is gone now returns Agent … is DEAD: its process (PID …) no longer exists … with the last registry update and the instruction not to hand-edit registry.json — orch_resume(force=true) reconciles it (ownership gate first, see Replacement supervisor cannot orch_resume the batch it inherits: activation imports phase 'executing' into memory and the in-memory guard overrides disk eligibility #631).

Engine-dead case (this issue's actual incident) is #631's path. Still open here: first-class held state + in-tool waitForReply (with #627).

[HenryLach]

Follow-up from the first PR #632 batch (penster 20260906T194514) — acknowledgement contract

The batch asked the right question: what happens when the outstanding item is an operator ruling that takes hours, and does a supervisor acknowledgement count as a reply? Honest answer for the build that ran: yes, it counted as the ruling — any steer after the escalation released the hold, so an "acknowledged, pending" would have turned the worker's next idle exit into an ordinary stall and reverted its relaunch prompt to the "work continuously" nag. And with no acknowledgement the three relaunches burn in minutes. Neither fits an hours-long ruling. (The batch didn't hit it because the worker never exited.)

Fixed (commit 2de3f24d, pushed to PR #632, local-deployed — build 6300ce063342). The contract is explicit, by send_agent_message type, no content heuristics:

Supervisor sends	Meaning	Effect on the holding worker
type="info"	Acknowledgement ("received, ruling pending, ~2h")	stays on hold; relaunch budget resets (STATUS: Hold acknowledged)
type="steer" (default)	The ruling / instruction	hold released; worker acts on it

Applied on both reply paths (delivered steer and exit-intercept). The worker's hold-resume prompt now also tells it to keep working on anything that does not depend on the ruling (other remediable findings, unaffected checkboxes). Primer + tool guidance updated.

Also from the feedback:

• Build marker: engine.json now records taskplaneVersion + taskplaneBuild (sha256 prefix of the loaded sources) and the takeover summary shows it. Until the version bumps, tool-presence detection remains fine; from now on engine.json is the authoritative answer.
• Hot-fix audit: the primer now says hand-remediation under an operator ruling (hot-fix commits, ratification files, manual repairs) is a recovery action — log it with log_recovery_action.

Behavioural test: ack keeps the hold past the 3-relaunch budget; a later steer releases it.

[HenryLach]

From penster feedback #3 (batches 20260906T194514 / 221045, PR #632 build):

Worked as designed: three unanswered hold relaunches → Hold unresolved → task failed, batch paused, worktree preserved; then engine.json exitedAt verified → orch_retry_task → orch_resume(force=true) relaunched the worker into the hold with the operator's amendment. First live confirmation of the #630/#629/#631 chain end to end.

Fixed from the report (commit 61eafece, deployed build bedb2c594fcc):

• Item 5 — 60 s exit-intercept window too short for a supervisor inside a tool call. New taskRunner.worker.exit_intercept_timeout_sec (default 60, 15..1800); the host safety race follows it (+60 s). Resume's re-execute/reconnect paths now forward the worker env (they previously dropped model/thinking/tools too). A "supervisor busy" heartbeat that defers the count is a good idea — noted for the Cap-ruling hold: runtime must block .DONE and merge while an operator ruling is pending (.PENDING-RULING) #627 hold design, where the counter is suspended outright.
• Item 4 — phantom ✅ Complete on a REVISE'd step (uncommitted STATUS edit, never authored by the worker): the lane-runner's checkbox heuristic flipped it every relaunch. Step completion is now review-gated; STATUS logs Step completion withheld.

Item 3 (intermittent hold prompt) is explained on #627 — by design of the Tier-1 bridge (steer releases; hold state is volatile across retry+resume), and the reason the durable hold record is the real fix.

[HenryLach]

Partial fix shipped in v0.30.6 (the minimal / Tier-1 cut described above, live-verified on Penster). This issue stays open for its Tier-2 scope, which is the next design pass together with #627 (held state, durable hold record, typed rulings, takeover state machine, full coverage gate, engine lease/generation).