fix(#667): circuit breaker — mark task BLOCKED after N consecutive silent recoveries#707
Conversation
…coveries Fix 3a from issue #667 (the board-policy half; the spawn-backoff half 3b is intentionally NOT built — the spawn controller is deleted by the session-model migration, epic #706). A "silent" recovery is a lease recovery where the agent never reported progress (renewal_count == 0, the `updates=0` signature of the snake_game runaway: 81 registrations over 3+ hours). After max_silent_recoveries (default 3) consecutive silent recoveries, recover_expired_lease now marks the task BLOCKED instead of returning it to TODO: BLOCKED tasks are never offered by request_next_task, so the reclaim/re-assign loop physically cannot continue — regardless of who supplies agents (Invariant #1: the guard lives in Marcus, not the spawner). - A recovery WITH progress resets the streak (slow-but-alive is not the silent-failure pattern — the #703 lesson at board level) - BLOCKED path performs the same coordination cleanup as normal recovery (lease, persistence, server callback) — no #485-class leaks - Diagnostic kanban comment + task_blocked_circuit_breaker history event; human resets to TODO to retry (streak starts fresh) 8 new tests; full lease module 74/74; mypy + pre-commit clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6181d728bf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| await self.kanban_client.update_task( | ||
| lease.task_id, | ||
| {"status": TaskStatus.BLOCKED, "assigned_to": None}, |
There was a problem hiding this comment.
When the circuit breaker trips on GitHub- or Linear-backed boards, this update_task call does not actually move the card to BLOCKED: checked GitHubKanban.update_task, which only opens/closes the issue for status, and LinearKanban.update_task, which ignores status entirely; both providers put their blocked-state logic in move_task_to_column. After the next project refresh, request_next_task can see the task as still open/TODO, so the silent-recovery loop is not stopped for those providers.
Useful? React with 👍 / 👎.
…ders Codex P2 on PR #707: GitHub- and Linear-backed boards don't persist status through update_task (GitHub only opens/closes the issue; Linear ignores status) — their blocked-state mechanics live in move_task_to_column. The breaker now also calls move_task_to_column(task_id, 'blocked') (GitHub 'blocked' label / Linear 'Blocked' state / SQLite direct status match) so a project refresh can't resurrect the task as TODO and resume the loop. +2 tests: move called with 'blocked' on trip; client lacking the method degrades gracefully (update_task path still lands). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Codex P2 (BLOCKED persistence on GitHub/Linear) addressed in the latest commit: the breaker now also calls |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
…04s (#709) The anthropics/claude-code-action default model (Claude Sonnet 4, claude-sonnet-4-20250514) has been retired; every claude-review run now fails with API 404 not_found_error before reviewing anything (first observed on PR #707). Pin a current model explicitly in both workflows. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
What is this system, briefly
Marcus is an AI-agent coordinator: it decomposes a project into tasks on a shared kanban board; agents claim tasks under a time-limited lease and must report progress to keep it. If a lease expires, Marcus recovers the task — resets it to TODO so another agent can claim it.
Why (the user-visible problem)
When every agent fails the same way before reporting any progress (e.g. an LLM/API outage), recovery becomes an infinite loop: claim → silent failure → lease expiry → back to TODO → a fresh agent claims → identical failure. The recorded incident (issue #667) ran 3+ hours, 81 agent registrations, 55 agent IDs on one integration task, burning cost the entire time with nothing on the board indicating anything was wrong. Nothing counted how many times the task had been through the wash.
What changed
Fix 3a from #667 — a circuit breaker in the recovery path (
src/core/assignment_lease.py):recover_expired_leasenow tracks a per-task consecutive silent-recovery streak (silent = the expired lease hasrenewal_count == 0, i.e. the agent never reported progress — theupdates=0signature from the incident log).max_silent_recoveries(new constructor param, default 3), the task is marked BLOCKED instead of TODO:request_next_tasknever offers BLOCKED tasks, so the loop physically cannot continue — regardless of who supplies agents (per Invariant Duplicate Tasks - Planka #1 the guard must live in Marcus, not in any particular spawner).task_blocked_circuit_breakerevent inlease_history.Deliberately NOT built: Fix 3b (spawn-controller backoff). That half throttles the spawner, and the spawner is deleted by the session-model migration (epic #706, Phase 3). Building it would be discarded work.
Design notes:
Worked example
Incident replay: agents 2_12, 1_15, 2_16 each claim the integration task, hit the Anthropic 400 credit error, never report progress, and expire. Under this PR: recovery 1 → TODO (streak 1); recovery 2 → TODO (streak 2); recovery 3 → BLOCKED + comment "3 consecutive agents … no progress … likely external failure." Total waste: ~3 lease windows (~20 min) instead of 3+ hours.
Test plan
pytest tests/unit/core/test_assignment_lease.py::TestSilentRecoveryCircuitBreaker— 8 new tests: below-threshold TODO resets, 3rd-silent → BLOCKED + ownership cleared, progressful recovery resets the streak, diagnostic comment posted exactly once, coordination cleanup on the BLOCKED path (lease/persistence/callback), threshold configurable, task model flips BLOCKED, history event recordedmypy src/core/assignment_lease.pyclean; pre-commit all hooks passWhere to look in the code first
src/core/assignment_lease.py(recover_expired_lease)src/core/assignment_lease.py(_block_task_after_silent_recoveries)tests/unit/core/test_assignment_lease.py(TestSilentRecoveryCircuitBreaker)Glossary
renewal_count == 0)Related
🤖 Generated with Claude Code