Skip to content

Commit c4ce538

Browse files
HenryLachHenryLach
authored
docs: autonomous supervisor specification (draft) (#276)
Priority #1 spec for true agent supervision. Three-layer architecture: - Layer 1: Engine deterministic recovery (known patterns) - Layer 2: Supervisor LLM for novel/ambiguous situations - Layer 3: Feedback loop (supervisor files issues → patterns become code) Key open question: how to wake the supervisor without a user message (pi's model is request-response). Needs investigation into pi's extension event system.
1 parent 8650312 commit c4ce538

1 file changed

Lines changed: 163 additions & 0 deletions

File tree

‎docs/specifications/taskplane/autonomous-supervisor.md‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
# Autonomous Supervisor Specification
2+
3+
**Status:** Draft
4+
**Priority:** #1
5+
**Created:** 2026-03-27
6+
7+
## Problem Statement
8+
9+
Taskplane's supervisor agent is reactive — it only acts when the user sends a message. Between messages, it's dormant. This means:
10+
11+
1. **Failures go undetected** until the user checks in and nudges the supervisor
12+
2. **Recovery is manual** — the user must prompt the supervisor to investigate and fix issues
13+
3. **Long-running batches cannot be unattended** — someone must watch and intervene
14+
15+
In every observed failure, the supervisor *was capable* of diagnosing and recovering the batch when prompted. The issue isn't intelligence — it's that the supervisor sleeps between user messages.
16+
17+
## Design Principles
18+
19+
1. **The supervisor must be a true autonomous agent** — it monitors, detects, and acts without human intervention
20+
2. **Deterministic code handles known failure patterns** — the engine should programmatically recover from documented edge cases (`.DONE` race, stale sessions, transient errors)
21+
3. **The supervisor handles novel/ambiguous situations** — when deterministic recovery can't resolve an issue, the supervisor LLM investigates and decides
22+
4. **Feedback loop reduces incident frequency over time** — the supervisor creates GitHub issues for recurring failure patterns, driving deterministic fixes into the engine
23+
5. **The user is informed, not required** — the supervisor notifies the user of incidents and actions taken, but doesn't block on user input
24+
25+
## Architecture
26+
27+
### Layer 1: Engine Deterministic Recovery
28+
29+
The engine (running in the forked child process) handles known failure patterns programmatically:
30+
31+
- **`.DONE` race condition** — ✅ Shipped in v0.21.3 (git branch check)
32+
- **Stale tmux sessions** — detect and kill orphaned sessions from prior runs
33+
- **Transient spawn failures** — retry with backoff (already exists)
34+
- **Merge conflicts** — automatic retry with fresh worktree (partially exists)
35+
- **Context pressure** — wrap-up signal and kill (already exists)
36+
37+
These are deterministic, fast, and don't require LLM reasoning. New patterns are added as they're discovered.
38+
39+
### Layer 2: Supervisor Autonomous Monitoring
40+
41+
The supervisor must have a **background monitoring loop** that runs independently of user interaction:
42+
43+
#### Option A: Engine-Driven Supervisor Triggers
44+
45+
The engine (Layer 1) detects situations it can't handle deterministically and sends a structured event to the supervisor:
46+
47+
```
48+
{ type: "supervisor-intervention-needed",
49+
reason: "merge_failed_unknown",
50+
context: { waveIndex: 1, laneId: "lane-1", error: "..." },
51+
suggestedActions: ["investigate merge log", "retry merge", "skip task"] }
52+
```
53+
54+
The supervisor receives this event and acts autonomously — no user message required.
55+
56+
**Pros:** Engine controls when LLM reasoning is needed. Efficient — supervisor only wakes for real issues.
57+
**Cons:** Requires a mechanism for the engine to trigger supervisor action.
58+
59+
#### Option B: Supervisor Polling Loop
60+
61+
The supervisor has a timer-based loop that periodically:
62+
1. Calls `orch_status()` to check batch state
63+
2. Reads the events file for errors/warnings
64+
3. Takes action if issues are detected
65+
66+
**Pros:** Simple conceptually. Supervisor is always watching.
67+
**Cons:** Requires pi to support timer-based self-prompting (not currently available). Burns LLM tokens on polling even when nothing is wrong.
68+
69+
#### Option C: Hybrid — Engine Events + Supervisor Watchdog
70+
71+
The engine handles known patterns (Layer 1) and emits events for everything else. A lightweight watchdog (not LLM-based) monitors for:
72+
- Batch stalled (no progress for N minutes)
73+
- Unexpected engine process death
74+
- Wave transition failures
75+
76+
When the watchdog detects an issue, it triggers the supervisor via a mechanism TBD (synthetic user message, tool invocation, or pi extension event).
77+
78+
**Pros:** Efficient — LLM only engaged when needed. Engine handles the fast path.
79+
**Cons:** More complex. Requires a triggering mechanism.
80+
81+
### Recommended: Option C (Hybrid)
82+
83+
The hybrid approach matches how production systems work:
84+
- Automated recovery for known issues (Layer 1)
85+
- Monitoring infrastructure for detection (watchdog)
86+
- Human-level reasoning for novel issues (supervisor LLM)
87+
88+
### Layer 3: Feedback Loop
89+
90+
When the supervisor resolves an incident, it should:
91+
92+
1. **Log the incident** — what happened, what it did, outcome
93+
2. **Classify the pattern** — is this a known type? Is it recurring?
94+
3. **Create a GitHub issue** — if the pattern should be handled deterministically in Layer 1
95+
4. **Tag the issue** — with severity, component, and reproduction steps
96+
97+
Over time, the feedback loop converts Layer 2 (LLM-handled) incidents into Layer 1 (deterministic) fixes. The supervisor should need to intervene less and less.
98+
99+
## Open Questions
100+
101+
### Q1: How does the supervisor "wake up" without a user message?
102+
103+
Pi's agent model is request-response. The supervisor runs as the main pi session. Options:
104+
- **Synthetic user message** — the engine/watchdog injects a message into the conversation
105+
- **Extension event** — pi fires an event that triggers a handler which invokes LLM reasoning
106+
- **Self-scheduling** — the supervisor's response includes a "check back in N seconds" signal
107+
- **Separate process** — the supervisor runs as its own pi process, not the user's session
108+
109+
This is the key technical blocker. Needs investigation into what pi supports.
110+
111+
### Q2: How does the supervisor take corrective action?
112+
113+
The supervisor already has tools: `orch_status`, `orch_resume`, `orch_pause`, `orch_abort`, `orch_integrate`, `orch_start`. For most recovery scenarios, these are sufficient. Additional tools may be needed:
114+
- `orch_retry_task` — retry a specific failed task
115+
- `orch_skip_task` — skip a task and unblock dependents
116+
- `orch_manual_fix` — apply a specific fix (edit batch state, force merge, etc.)
117+
118+
### Q3: What's the token budget for autonomous supervision?
119+
120+
Autonomous monitoring costs tokens. Need to balance:
121+
- Frequency of status checks
122+
- Depth of investigation on failure
123+
- Cost ceiling per batch
124+
125+
### Q4: How does the user stay informed?
126+
127+
The supervisor should notify the user of:
128+
- Incidents detected and actions taken (summary, not verbose)
129+
- Decisions that need human judgment (escalation)
130+
- Batch completion with incident report
131+
132+
Notification channels: pi chat (when user returns), dashboard, terminal notification.
133+
134+
## Implementation Phases
135+
136+
### Phase 1: Engine Deterministic Recovery (in progress)
137+
- ✅ `.DONE` branch reconciliation (v0.21.3)
138+
- [ ] Orphan tmux session cleanup on task failure (#242)
139+
- [ ] Stale worktree detection and cleanup
140+
- [ ] Merge retry with fresh worktree
141+
142+
### Phase 2: Supervisor Triggering Mechanism
143+
- [ ] Investigate pi's extension event system for supervisor wake-up
144+
- [ ] Prototype: engine emits event → supervisor handler → LLM reasoning
145+
- [ ] Define the supervisor intervention protocol (event schema, response contract)
146+
147+
### Phase 3: Autonomous Supervisor Loop
148+
- [ ] Supervisor monitors batch lifecycle events
149+
- [ ] Supervisor investigates and recovers from failures
150+
- [ ] Supervisor escalates to user when it can't resolve
151+
- [ ] Incident logging and classification
152+
153+
### Phase 4: Feedback Loop
154+
- [ ] Supervisor creates GitHub issues for recurring patterns
155+
- [ ] Issue template for "incident → deterministic fix" proposals
156+
- [ ] Metrics: incidents per batch, auto-recovered vs escalated
157+
158+
## Success Criteria
159+
160+
1. A batch with a recoverable failure (`.DONE` race, merge conflict, transient error) completes without user intervention
161+
2. The supervisor notifies the user of what happened and what it did
162+
3. After N batches, recurring patterns are filed as issues and subsequently fixed in engine code
163+
4. Incident rate per batch decreases measurably over time

0 commit comments

Comments
 (0)