Wisp chain fragility: single crash permanently breaks self-pouring patrol loop

[alexsiri7]

Problem

The refinery patrol (and similar wisp-based loops) relies on each iteration pouring the next wisp before burning the current one. If the session crashes at the wrong moment — after starting work but before pouring the next wisp — the entire loop breaks permanently.

Failure scenario

1. Refinery is processing work in wisp N
2. Quota exhaustion (or any crash) kills the session
3. Wisp N is still assigned but the session is dead
4. No new wisp N+1 was poured
5. When the session eventually restarts, it may find the old wisp but the chain is interrupted
6. If the old wisp gets cleaned up (burn, timeout), there's no next iteration

Impact

The gascity refinery patrol loop broke during quota exhaustion. 6 work beads sat unprocessed for hours until manual intervention poured a fresh wisp.

Suggested fixes

Option A: External wisp watchdog

An order (not wisp-driven) that periodically checks: "does each always-on agent have an active wisp assigned?" If not, pour one. This is the NDI (nondeterministic idempotence) approach — multiple observers, convergent.

Option B: Reconciler-aware wisp management

The reconciler could detect named sessions with no active wisp and trigger formula dispatch.

Option C: Crash-safe pour-before-burn

Make the formula runner pour the next wisp at the START of each iteration (not the end), so crashes after pour are safe. The old wisp stays around as a no-op if the new one takes over.

Option A is most aligned with the existing architecture (NDI, no framework intelligence).

[rileywhite]

Investigation notes

I looked into this with the intent to fix, but the behavior appears to be addressed by the convergence system in internal/convergence/.

The recoverCurrentActiveWisp function (manual.go:447) handles the exact failure scenario described: when the active wisp is missing (crashed session, cleaned up bead), it reconstructs the chain by:

1. Looking up the lastProcessedWisp to determine the expected next iteration
2. Searching for the next iteration's bead by idempotency key
3. Falling back to scanning children for the best open or closed candidate

This is called from both the reconcile path (reconcile.go:390) and the manual handler path (manual.go:278, 323), and the controller runs startup recovery via convergenceStartupReconcile on boot.

Existing test coverage:

• TestReconcile_Active_MissingActiveWisp_ReconstructsChain — active wisp cleaned up after crash, recovery rebuilds the chain
• TestReconcile_Active_MissingActiveWisp_ReplaysRecoveredClosedReplacement — replacement wisp already closed, recovery replays it
• TestStopHandler_MissingActiveWisp_RecoversReplacementBeforeForceClose — recovery during stop handling

This shifts patrol loops from the fragile self-pouring pattern (agent pours its own next wisp) to externally managed iteration lifecycle (convergence reconciler manages the chain). The crash window described in this issue — "after starting work but before pouring the next wisp" — is no longer fatal because the convergence reconciler reconstructs the chain on the next tick.

This issue may still be relevant for patrol formulas that haven't migrated to the convergence system. If any self-pouring patrol loops remain outside convergence management, they would still have this fragility. A maintainer would know whether all patrol formulas have been migrated.

[julianknutsen]

Bug confirmed — accepted for implementation

Classification: confirmed bug · P1 · core workflow broken

Summary

Investigation confirmed the reported wisp chain fragility. The failure scenario described — session crash between starting work and pouring the next wisp permanently breaks the patrol loop — is reproducible on both the reported build (v0.13.4) and current main (d601161).

Root cause

The convergence reconciler (convergenceStartupReconcile at cmd/gc/convergence_tick.go:297) filters by Type: "convergence", but patrol wisps created via bd mol wisp are type "task". This makes patrol wisps invisible to crash recovery. No alternative recovery mechanism exists for self-poured patrol chains.

Affected locations:

• cmd/gc/convergence_tick.go:297 — type filter excludes patrol wisps
• internal/molecule/molecule.go:652 — default type is "task"
• cmd/gc/session_reconciler.go — no wisp chain awareness

Acknowledging @rileywhite's analysis

The contributor investigation correctly identified the convergence system's recovery capabilities and flagged the open question: whether all patrol formulas have migrated. The answer is they have not — all three gastown patrol formulas (refinery, witness, deacon) still use the self-pouring bd mol wisp pattern.

Proposed fix

Migrate patrol formulas to convergence-managed loops, reusing the existing tested recovery infrastructure rather than duplicating logic with a new watchdog.

---

Automated investigation by bugflow · investigation packet · run ga-q8c75r