Replacement supervisor cannot orch_resume the batch it inherits: activation imports phase 'executing' into memory and the in-memory guard overrides disk eligibility

[HenryLach]

Summary

When a supervisor session dies or is replaced while its batch is executing, the replacement session cannot resume the batch it inherits. Supervisor activation copies the persisted phase ("executing") from batch-state.json into in-memory orchBatchState, and doOrchResume checks that in-memory phase before consulting disk — refusing with "A batch is currently executing … Cannot resume." Yet checkResumeEligibility on the persisted state explicitly treats executing as resumable ("was executing when the orchestrator disconnected"). The two disagree, and the in-memory check wins, so the designed disconnect-recovery path is unreachable from the session that is supposed to perform it.

Observed in penster batch henrylach-20260905T210935 (TP-2047), 2026-09-06, after the original supervisor session wedged on a provider-side 400 and a new session took over the lock.

Sequence

1. Supervisor session A owned the batch; its engine/monitor loop ran in-process. Session A wedged (every provider request failed with the same 400 on a thinking block in the latest assistant message).
2. The operator started supervisor session B. Lock takeover (stale heartbeat / dead pid path) ran buildTakeoverSummary and populated orchBatchState from batch-state.json, including phase = "executing" (extension.ts ~L4246 and ~L4292).
3. No engine loop exists in session B for this batch. Evidence: registry.json updatedAt frozen at 1788657826268 for 17+ minutes; the dead worker pid (59452) never reconciled; orch_pause accepted (set pauseSignal) but nothing honored it; batch-state.json stayed phase: executing, updatedAt 1788656981285.
4. orch_resume (with and without force) refused: in-memory phase executing. orch_retry_task / orch_skip_task / orch_force_merge carry the same active-phase guard. orch_stop is not in the supervisor toolset; orch_abort cleans up state (not resumable); supervisor_takeover is unsafe (supervisor_takeover destroys the paused lane: task marked skipped, batch completed 0/1, worktree+branch removed, uncommitted work lost #628).
5. Recovery: back up state; edit one field in batch-state.json (phase: executing → paused); start a third supervisor session so activation imports paused; orch_resume then reconciled the dead worker as precedence-4 re-execute in the existing worktree and the lane relaunched cleanly.

Cause (code pointers, feat/review-boundary-supervisor-notifications @ a32cde3)

• extensions/taskplane/extension.ts ~L2919 doOrchResume: refuses when orchBatchState.phase ∈ {launching, executing, merging, planning} — before loadBatchState.
• extensions/taskplane/extension.ts ~L4246 / ~L4292 (lock takeover, stale/no-lockfile/corrupt and live-takeover branches): orchBatchState.phase = batchState.phase — imports an ACTIVE phase into a session that has no engine for it.
• extensions/taskplane/resume.ts L396 checkResumeEligibility: executing → eligible ("orchestrator disconnected"). This is the intended semantics; the in-memory guard defeats it.

Expected

• On lock takeover, if no engine for the batch runs in the taking-over process, the imported phase must not be an active one. Either import paused/disconnected (a phase meaning "persisted state says executing, no live engine here"), or set a flag engineAttached=false that the active-phase guards consult.
• doOrchResume should treat "active phase in memory but no engine attached in this process" as resumable, deferring to checkResumeEligibility on the persisted state — which already encodes the disconnect case.
• The supervisor toolset should carry a non-destructive stop (orch_stop or equivalent) that moves an orphaned batch to paused without cleanup, so the only out is not a hand edit of batch-state.json.
• Test: session A persists executing and dies (or its lock heartbeat goes stale); session B takes over; orch_resume succeeds and reconciles dead workers via the existing precedence rules.

Notes

• The stale heartbeat / dead-pid detection in the lock takeover already knows session A is gone; that knowledge should flow into the phase decision.
• The registry did not need repair: resume's liveness check is !isTerminalStatus(manifest.status) && isProcessAlive(manifest.pid), so a dead pid marked running is correctly treated as dead. Worth documenting so operators do not hand-edit registry.json unnecessarily.

Relation to other issues

• Worker exits when an expectsReply escalation times out; lane-runner does not relaunch it and registry/lane state stay 'running' #630 (companion, filed together): the worker itself died because an expectsReply: true escalation timed out while the supervisor was being replaced, and nothing relaunched it.
• orch_retry_task resets the task record but not the v2 segment record, so orch_resume skips the wave (single-wave batch cannot be re-driven after a finalize-gate refusal) #629: the retry+resume path failing after a finalize-gate refusal; a different reconciliation defect, same area.
• Cap-ruling hold: runtime must block .DONE and merge while an operator ruling is pending (.PENDING-RULING) #627: a first-class held state would have removed the timeout in the companion issue; this issue stands regardless — any supervisor replacement during executing hits it.

[HenryLach]

Fixed (commit dd068ea5; local-deployed). This became the largest piece of the bundle because Sage's design review established the rule I then had to make hold everywhere: recovery mutations require exclusive engine ownership established by confirmed shutdown, not inferred inactivity — the engine is a forked child that outlives its supervisor, so a dead supervisor pid proves nothing.

What landed

• Engine identity .pi/runtime/<batchId>/engine.json (pid, supervisor pid, start/exit), published before the engine starts (batch id preallocated; the engine adopts it and refuses to resume any other batch). Exit marking is pid-scoped. An orphaned engine winds itself down as paused.
• One ownership gate (decideRecoveryOwnership, pure + tested) for orch_resume, orch_retry_task, orch_skip_task, orch_force_merge, orch_pause (inherited), orch_abort, orch_integrate and a fresh /orch: refused while an engine is attached to this session (running or still exiting); refused while the target's recorded engine is alive elsewhere; refused when no identity is recorded; allowed once verified dead/exited. force never bypasses it. Evaluated against the batch actually acted on (persisted, or reconstructed; integration looks up ownership by the selected branch), never a cached phase.
• Your three asks: (1) inherited active phase no longer blocks — it is resumable once the engine is verified gone; (2) orch_pause on an inherited batch performs an administrative pause (persists paused) — the non-destructive stop, no hand-edit; (3) the takeover summary reports engine liveness and dead-but-"running" registry agents with the note that no registry edit is needed.
• Cleanup is bound to the batch it belongs to (state deletion, lane-branch sweep, CI post-PR cleanup); abort verifies the local engine has actually exited before persisting/deleting; agents alive before a lane re-executes are terminated with verification.
• Legacy path: batches with no identity (pre-0.30.6) fail closed once; orch_confirm_engine_shutdown(note, batchId?) / /orch-confirm-engine-shutdown [--batch <id>] <note> records the operator's out-of-band verification (audited).

Accepted Tier-2 residuals (lease/generation follow-up): two simultaneous replacement sessions can both authorize against one verified-dead target; markEngineExited is read-check-write, not CAS (pid matching covers the sequential case only).

[HenryLach]

For agents reading this issue — operational summary of the fix (PR #632, v0.30.6)

The rule (applies to orch_resume, orch_retry_task, orch_skip_task, orch_force_merge, orch_pause on an inherited batch, orch_abort, orch_integrate, and a fresh /orch): a recovery mutation runs only when the target batch's engine is verified gone. force never bypasses it.

Engine state (from .pi/runtime/<batchId>/engine.json)	Tools do
attached to this session (running, or still exiting)	refuse — wait / pause it; a terminal cached phase does not mean the process has exited
alive in another process	refuse, naming the PID; wait for it (an orphaned engine pauses itself when its supervisor disconnects) or terminate it
none recorded (pre-0.30.6 batch, or engine never published)	refuse — unknown ownership is not confirmed shutdown
dead / exited (incl. operator-confirmed)	proceed; normal persisted-state eligibility applies

What you will observe as a replacement supervisor

1. The takeover summary has an Engine: line (alive PID / dead / exited / none) and lists dead agents still marked running — do not hand-edit registry.json; resume reconciles them.
2. Engine dead/exited → orch_resume(force=true) works (no more Cannot resume on an inherited executing). Dead workers re-execute in their existing worktrees; any still-alive agent is terminated with verification first.
3. Engine alive → the refusal tells you the PID and what to do. Do not supervisor_takeover/orch_abort your way past it.
4. No identity → verify out-of-band (Get-CimInstance Win32_Process | Where-Object { $_.CommandLine -match "engine-worker" } / pgrep -af engine-worker), then run orch_confirm_engine_shutdown(note, batchId?) (or /orch-confirm-engine-shutdown [--batch <id>] <note>) — audited; refuses if a real identity exists. Use batchId when the refusal names a batch that has no batch-state.json (meta-only runtime dir).
5. orch_pause on an inherited batch whose engine is verified gone performs an administrative pause (persists paused) — the non-destructive stop; never hand-edit batch-state.json.

Primer Pattern 9 ("You Inherited an executing Batch") documents the same flow. Not fixed here (Tier 2): two simultaneous replacement sessions racing one dead target; markEngineExited is not compare-and-swap.

[HenryLach]

Partial fix shipped in v0.30.6 (the minimal / Tier-1 cut described above, live-verified on Penster). This issue stays open for its Tier-2 scope, which is the next design pass together with #627 (held state, durable hold record, typed rulings, takeover state machine, full coverage gate, engine lease/generation).