Epic: session-model migration — spawn-per-task → long-lived pull-loop sessions

[lwgray]

What is this system, briefly

Marcus is an open-source system that takes a plain-English project description, decomposes it into tasks on a shared kanban board, and coordinates independent AI coding agents that pull those tasks, build them, and report back — communicating only through the board (decisions, artifacts, progress), never with each other.

This epic tracks the session-model migration: replacing how agents come to exist and how Marcus tracks them, plus the stabilization work sequenced around it.

Why (the user-visible motivation)

Today Marcus spawns one short-lived agent process per task (a claude CLI in a tmux pane, in its own git worktree, killed after one task). Analysis of all 22 open build-blockers (epic #701) showed 4 of the 6 run-killing bugs live partly or wholly in that spawning/babysitting layer — worktree explosions (#628), watchdogs killing healthy runs (#703), infinite respawn loops (#667), merge-recovery wedges (#700). Meanwhile agent platforms now provide process management natively.

The change: Marcus stops spawning anything. Long-lived agent sessions (started by a thin launcher outside Marcus's core) register once, then loop: pull a task → work → report → pull the next. Marcus tracks agents at exactly one level — the lease (task_id, expires_at, last_progress_at) — and stays deliberately blind to the OS process. Several of the worst bugs then get deleted instead of fixed.

The plan (six phases, strict order)

Target calendar (set 2026-07-06): Phase 1 remainder → Jul 19 · Phase 3 landed (launcher + pull-loop + D11) → Aug 2 · Phase 4 → Aug 30 · Phase 5 → Sep 13 · Phase 6 gate (5 unattended Shape-B passes) → Sep 27. Dates are targets, reviewed weekly against actual weekend output.

Phase 1 — Board-protocol run-killers (re-architecture-safe; start immediately)

• bug: integration-verification task enters infinite agent spawn loop — lease too short + stale-completion rejection + no spawn-controller backoff = runaway when LLM calls fail #667 — circuit breaker: N silent recoveries → autonomous BLOCKED → merged via PR fix(#667): circuit breaker — mark task BLOCKED after N consecutive silent recoveries #707 (incl. provider-portable BLOCKED per Codex review); spawn-backoff half documented as deleted-by-Phase-3
• Merge conflict leaves dirty index; rebase recovery fails with 'uncommitted changes' — task stuck blocked despite completed work #700 — bridge fix: clean the git index before the recovery rebase (small, surgical; merge execution migrates later)
• bug: integration verification dead-ends into project gridlock when the agent never declares verifications #676 — escape hatch for the verification-declare reject loop + JSON-parse crash guard
• feat: add request_task_redo MCP tool so integration agents can send failed work back without violating Invariant #2 #627 — request_task_redo MCP tool (recovery enabler)
• bug: integration-verification task never claims when an upstream dependency stays in BLOCKED or partial-done state #629 — recovery/best-effort claim path for the terminal integration task
• _collect_task_artifacts uses wrong kanban_client.get_attachments contract (same bug as #623) #624 — fix get_attachments contract (same shape as merged fix(context): Codex P1 + P2 on #622 release PR #623)

Phase 2 — The boundary ADR (gates Phases 3 and 5 only)

• Draft + decide the ADR covering the 10 open decisions → ADR 0012, PR docs: ADR 0012 — session-model migration (all 10 decisions + 5 obligations) #708 (Simon: e836842e, b90e948a, 9548fed8, 40687aa6): merge-execution home (launcher vs integrator task) · session isolation model (shared main / worktree-per-session / clone-per-session) · lease timing model · provenance enforcement strength (hard vs soft log_decision gate) · session-seam definition · pool-sizing authority · session↔project cardinality · recovery of long partial work · internal-subagent accounting · fate of "a run"
• The ADR must explicitly address the 5 review-found gaps → ADR 0012 §Inherited obligations (O1–O5, each assigned a home) (see below)

Phase 3 — Session MVP: launcher + pull-loop (deletes bugs instead of fixing them)

PRE-3 gates (must complete before any Phase-3 code):

• Keep/delete call-site inventory (advisory-panel review, Codex seat): grep-verified list, checked into this epic as a comment, of every durable-core touchpoint on (a) worktrees/<agent_id> path construction (e.g. task.py:3242 merge-recovery), (b) direct active_leases mutation outside the lease manager, (c) run_id as a schema/join key in cost_tracking ("core goes runless" deletes the ledger's join key — worker_ingester.py requires it on every token_events row), (d) experiment-scoped registration coupling in agent.py, (e) BOTH spawn_agents.py copies (experiments/ vs dev-tools/experiments/runners/) — each site mapped to a numbered migration change or explicitly marked "survives unchanged + why"
• Implement ADR D11 fencing (PR docs: ADR 0012 amendment — D11 false-recovery fencing (the two-robots-one-chore rule) #710): lease_epoch on claim, epoch carried on reports, stale-epoch → preserve + reconciliation card (never discard), timeout ≥ 2× checkpoint cadence as config invariant. Verification scenario: falsely recover a live session; both report; exactly one completion, zero discarded work, one reconciliation card
• Pull-loop worker prompt (delete "EXACTLY ONE task then exit"; honor retry_after; add an explicit project-drained stop signal)
• Thin launcher/pool outside core (operator- or board-depth-controlled session count; replaces the per-task spawn loop)
• Registration lifecycle: idempotent re-register + persist/rehydrate agent_status / agent_project_map (gap 1 — today a Marcus restart orphans every session)
• Lease retune: durations for session-seam tasks, raise/remove max_renewals(10) / stuck_task_threshold_renewals(5) (gap 4), recovery cadence for silent deep work, session-aware stale-completion guard (gap 3 — a falsely-recovered live session must not have its work discarded)
• Re-home cost/audit binding (gap 2 — attribution currently keyed off the spawn-registry file + per-agent cwd, both being deleted): record session→agent/run/project at register/claim time
• Session context-management obligation (gap 5 — unbounded context growth across tasks in one process)
• Remove merge EXECUTION from report_task_progress per the ADR decision; keep merge EVIDENCE
• On landing: close bug: ephemeral subtask spawning explosion — todo-completion-2 created 279 worktrees for a 9-task project #628 and Spawn-thrash detector tears down healthy-but-slow runs (resets only on task completion) #703 with migration notes; retire bug: integration-verification task enters infinite agent spawn loop — lease too short + stale-completion rejection + no spawn-controller backoff = runaway when LLM calls fail #667's spawn-backoff half

Phase 4 — Verification moat (survives everything; the product)

• bug: composition/integration verify that the project BUILDS, not that it BEHAVES — and the checks assume web (curl 2xx / DOM), not pipelines/CLIs/libraries #677 — gate proves the app BEHAVES, not builds; non-web stacks
• bug: verification-command generator infers tech stack from prose, not the scaffold's actual dependency manifest #643 — verification commands grounded in the real dependency manifest
• bug: composition+verify pipeline ships unplayable apps when spec-coverage gap-fill creates redundant tasks (orphan source trees pass smoke gate) #636 — smoke-gate ownership hard cutover (Invariant Task Assigned 4 times #2 v2)
• bug: composition step doesn't detect orphan source trees — agent-built files that no task imports get ignored #638 — orphan source-tree detection
• Detect when the composition agent shadows a domain export (imported-but-never-used) — the detection half of #691 #693 — shadowed-export detection (detection half of Composition task reimplements domain logic instead of wiring it (presentation renderer shadowed in test109) #691)

Phase 5 — Decompose at the session seam (waits on ADR seam definition)

• Implement the seam test — cut only where the interface between two chunks can be written as a contract smaller than either chunk: retune should_decompose thresholds, remove the enterprise force-decompose rule, gate the second-level subtask split, reshape contract-first to genuine seams
• Re-triage the graph-quality bugs at the new granularity before fixing (some shrink or vanish): Contract consistency gate: compare type structure, not type names (false-positive fallback) #669 → contract_first systematically falls back at standard+ via cross-contract type-consistency check #615, bug: gap-fill rollup (#611) misroutes criteria onto semantically unrelated tasks — keyboard handler landed on Vite-setup task #665, bug: spec_coverage augmenter inserts redundant gap-fill tasks for umbrella phrases already covered by other tasks #637, Decomposer emits near-duplicate tasks at enterprise complexity (no fuzzy dedup pass) #620, Design-ghost auto-completion is non-deterministic (status=todo sometimes instead of done) #619, Functional requirements with empty descriptions fall through to infrastructure stub #617 (bug: trivial-spec /marcus runs ship wrong-language output and over-decompose (constraint-drop in 5 prompts + domain floor for prototype mode) #649's over-decomposition half is fixed by design here)

Phase 6 — The gate

• Shape B benchmark ("order-processing platform": contract card → 4 component cards → integrate+verify) completes end-to-end, unattended, 5 consecutive times, behaviorally verified
• Snake/one-card apps retired as coordination benchmarks (kept as regression tests)
• Then measurement (board vs native orchestration) begins — not before.

Interim rule: no large experiments until Phase 3 lands — the shed bugs still bite live runs.

The 5 review-found gaps (must not be lost)

An adversarial code-review pass over the migration plan found five failure modes that only appear AFTER going session-based:

1. Registration doesn't survive restart — agent_status/agent_project_map init empty and are never reloaded (src/marcus_mcp/server.py:200-201; leases ARE reloaded, agent state is not — src/marcus_mcp/tools/task.py:3874-3876); with register-once sessions, a Marcus restart orphans every session (_scope_tasks_to_project raises at task.py:5150-5155).
2. Cost attribution loses its input — the resolver reads the spawn-registry file written by spawn_agents.py and disambiguates agents by cwd == per-agent worktree (src/cost_tracking/worker_ingester.py:26, 312-316); both are being deleted.
3. Stale-completion guard discards live sessions — the guard (task.py:3865-3960) assumes agents die on lease recovery; a long-silent session that trips recovery then reports 100% gets its entire coarse task thrown away.
4. Renewal ceiling trips healthy long tasks — max_renewals=10, stuck_task_threshold_renewals=5 (src/core/assignment_lease.py:211,219) flag a perfectly-heartbeating long task as stuck.
5. Unbounded context growth — no code path manages context accumulation across many tasks in one process (never needed under 1-process-per-task).

Where to look in the code first

File	Why
dev-tools/experiments/runners/spawn_agents.py	The spawn-per-task runner being replaced (worker prompt 854-972, spawn loop 1849-2061, worktrees 1374-1447, pane tracking 1598-1634)
dev-tools/experiments/runners/spawn_controller.py	compute_spawn_count, StallWatchdog, SpawnThrashDetector — retired in Phase 3
src/marcus_mcp/tools/task.py	request_next_task no-task branch (2745-2793), stale-completion guard (3865-3960), merge execution (3395-3571, 4447)
src/core/assignment_lease.py	Lease timers (206-300), renewal ceiling (211/219), recovery cadence (1425-1533), worktree coupling (690-998)
src/marcus_mcp/tools/agent.py	register_agent ephemerality assumptions (16-171)
src/marcus_mcp/coordinator/decomposer.py	should_decompose thresholds (18-215) — Phase 5
src/cost_tracking/worker_ingester.py	Attribution binding to re-home (114-130, 269-325)

How to verify it works

1. Phases 1: each fix lands with a regression test; pytest -m unit green; no new hangs in a small live run.
2. Phase 3: start Marcus + launcher with 3 sessions; kill and restart Marcus mid-run → sessions keep working (gap 1); kill a session mid-task → lease expires → another session claims (single-level liveness); token_events rows attribute to the right agent (gap 2).
3. Phase 6 gate: 5 consecutive unattended Shape-B runs, each ending with the behavioral gate passing and zero manual rescues.

Glossary

Term	Meaning
session	a long-lived agent CLI process (claude/codex/gemini/human) that registers once and loops pulling tasks
launcher / pool	thin ops layer OUTSIDE Marcus core that starts N sessions; Marcus spawns nothing
lease	Marcus's only tracking of an agent: (task_id, expires_at, last_progress_at); heartbeat = liveness
seam	a boundary where the interface between two chunks can be written as a contract smaller than either chunk — the only place decomposition cuts
Shape B benchmark	the smallest workload exercising every between-card mechanic: contract → 4 components → integrate+verify
shed / keep / split	verdicts from the layer triage of epic #701: delete with the runner / durable-core fix / half each

Related

• Epic Build-blocker fix plan (2-week triage): what stops Marcus from building projects #701 (build-blocker fix plan — this epic re-sequences it; see the comment there)
• Deleted-instead-of-fixed on Phase 3 landing: bug: ephemeral subtask spawning explosion — todo-completion-2 created 279 worktrees for a 9-task project #628, Spawn-thrash detector tears down healthy-but-slow runs (resets only on task completion) #703, bug: integration-verification task enters infinite agent spawn loop — lease too short + stale-completion rejection + no spawn-controller backoff = runaway when LLM calls fail #667 (spawn-backoff half)
• Simon: decision 9479b614 (migration change set), 2f062a90 (drop product-wedge; realign to core), fc0737eb (original Build-blocker fix plan (2-week triage): what stops Marcus from building projects #701 plan)

[linear-code]

PM-319