Skip to content

Commit 60c679d

Browse files
committed
fix(tasks): confirm stopped jobs before recovery wake
1 parent daa55a0 commit 60c679d

12 files changed

Lines changed: 568 additions & 44 deletions

docs/background-orchestration.md

Lines changed: 18 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -180,9 +180,10 @@ on the local job board.
180180

181181
After a full OpenCode or plugin restart, persisted running background-task
182182
history is rehydrated into the local job board and immediately reconciled against
183-
live host session status. A missing or idle child is therefore surfaced as
184-
`stopped, unreconciled`, while a busy child remains running; status lookup
185-
failures remain uncertain rather than being treated as completion.
183+
live host session status. A missing or idle child is a stop candidate: after a
184+
5s confirmation grace it is surfaced as `stopped, unreconciled`, while a busy
185+
child remains running; status lookup failures remain uncertain rather than being
186+
treated as completion.
186187

187188
Specialist outputs are inputs, not final truth. The orchestrator reconciles them
188189
against each other and the original user goal.
@@ -453,17 +454,22 @@ liveness authority. After a tracked task launches, the plugin periodically
453454
checks that single map for every board job still marked `running`, while normal
454455
session events remain the fast path.
455456

456-
`busy` and `retry` confirm that a job is live. An explicit `idle` state or an
457-
absent session in an otherwise valid map records `stopped, unreconciled` rather
458-
than `completed`: it means execution ended before a native terminal task result
459-
was delivered, not that the task succeeded. Stopped sessions are never reusable
460-
and stay visible to the parent for recovery. A later live `busy` observation can
461-
revive them, and only explicit terminal task output proves completion, error, or
462-
cancellation.
457+
`busy` and `retry` confirm that a job is live and reset any pending stop
458+
confirmation. An explicit `idle` state or an absent session in an otherwise
459+
valid map is not immediately terminal: the first observation starts a 5s
460+
confirmation grace and keeps the job `running, status uncertain`. Repeat
461+
non-busy evidence after that grace records `stopped, unreconciled` rather than
462+
`completed`: it means execution ended before a native terminal task result was
463+
delivered, not that the task succeeded. Stopped sessions are never reusable and
464+
stay visible to the parent for recovery. A later live `busy` observation can
465+
revive an unreconciled stopped job. After the parent has been woken and the stop
466+
acknowledged, stale busy cannot flip the job back to running. Only explicit
467+
terminal task output proves completion, error, or cancellation.
463468

464469
Malformed status entries and failed status requests are surfaced as `status
465-
uncertain`; they never prove that a job stopped or completed. Each observation
466-
is generation-aware, so a delayed response cannot modify a relaunched task.
470+
uncertain`; they never prove that a job stopped or completed and do not confirm
471+
a pending stop. Each observation is generation-aware, so a delayed response
472+
cannot modify a relaunched task.
467473

468474
### Opt-in Wall-clock Supervisor
469475

src/hooks/task-session-manager/codemap.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ Manages V2 background job-board state for task execution and injected completion
99
The directory follows a **Facade + Strategy** pattern where `index.ts` acts as the facade that composes and orchestrates behavior across specialized strategy modules:
1010

1111
- **index.ts**: Main facade that wires hooks into OpenCode's lifecycle and coordinates between the job board, pending calls, task context tracking, and explicit user waits. Implements the plugin hook interface (`tool.execute.before`, `tool.execute.after`, `experimental.chat.messages.transform`, `event`) and exposes `beginUserWait()` to the `wait_for_user` tool.
12+
- **stop-confirmation.ts**: Shared 5s grace for idle/absent runtime observations. Transient non-busy evidence stays provisional; confirmed durable stop evidence calls `markStopped` and can wake the parent. Busy/retry/live-busy reset the clock.
1213
- **input-wait-tracker.ts**: Provides the single `hasInputWait()` seam used by idle reconciliation and continuation evaluation. It combines local question/permission waits with the process-global explicit user-wait latch.
1314
- **continuation-attempt-gate.ts**: Owns process-global continuation epochs, reservations, and explicit user waits across hook recreation. The wait is encoded as an `attempts` sentinel so pre-upgrade #856 hooks sharing the store also fail closed. Distinct external user-message identity rearms both states.
1415
- **continuation-model-selection.ts**: Normalizes current-session and chat-hook model shapes before forwarding runtime model and variant choices to idle continuation prompts.
@@ -56,8 +57,8 @@ All modules depend on `BackgroundJobBoard` from `src/utils/background-job-board.
5657

5758
5. **Lifecycle Events (`event`)**
5859
- `session.created`: Adds new task IDs to pending managed set
59-
- `session.idle` / `session.status` (idle): Reconciles injected terminal jobs for the parent session (backstop path), then can run the opt-in continuation evaluator in the same idle cycle under its existing guards
60-
- `session.status` (busy): Marks sessions as running from live session state
60+
- `session.idle` / `session.status` (idle): Reconciles injected terminal jobs for the parent session (backstop path), then can run the opt-in continuation evaluator in the same idle cycle under its existing guards. Child idle is a stop candidate: the first observation stays provisional, and only a confirmed idle/absent after the 5s grace marks `stopped`
61+
- `session.status` (busy): Marks sessions as running from live session state and resets pending stop confirmation
6162
- `session.deleted`: Clears job state, child jobs, and pending call records for the session
6263

6364
6. **Human-in-the-loop Waits**
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
import { describe, expect, mock, test } from 'bun:test';
2+
import { BackgroundJobBoard } from '../../utils';
3+
import { createIdleReconciler } from './idle-reconciliation';
4+
5+
async function flushChildIdleReconcile(): Promise<void> {
6+
await new Promise((resolve) => setTimeout(resolve, 5));
7+
}
8+
9+
function createHarness(options?: { stopConfirmationGraceMs?: number }) {
10+
const board = new BackgroundJobBoard();
11+
const terminalListener = mock(() => {});
12+
board.addTerminalStateListener(terminalListener);
13+
const contextFilesForPrompt = mock(() => []);
14+
const prune = mock(() => {});
15+
const reconciler = createIdleReconciler({
16+
backgroundJobBoard: board,
17+
reconcileInjectedTerminalJobs: mock(() => {}),
18+
idleReconcileDelayMs: 0,
19+
stopConfirmationGraceMs: options?.stopConfirmationGraceMs ?? 0,
20+
hasInputWait: () => false,
21+
getIdleSessionToken: () => Symbol('idle'),
22+
isCurrentIdleSessionToken: () => true,
23+
taskContextTracker: {
24+
pendingManagedTaskIds: new Set(['child-1']),
25+
contextFilesForPrompt,
26+
prune,
27+
},
28+
});
29+
board.registerLaunch({
30+
taskID: 'child-1',
31+
parentSessionID: 'parent-1',
32+
agent: 'fixer',
33+
description: 'fix idle race',
34+
now: 0,
35+
});
36+
return { board, reconciler, terminalListener, contextFilesForPrompt, prune };
37+
}
38+
39+
async function observeIdle(
40+
reconciler: ReturnType<typeof createIdleReconciler>,
41+
idleObservedAt: number,
42+
generation: number,
43+
): Promise<void> {
44+
reconciler.scheduleChildIdleReconciliation(
45+
'child-1',
46+
idleObservedAt,
47+
generation,
48+
);
49+
await flushChildIdleReconcile();
50+
}
51+
52+
describe('idle reconciliation stop confirmation', () => {
53+
test('idle then busy inside grace remains running with no terminal listener', async () => {
54+
const { board, reconciler, terminalListener } = createHarness({
55+
stopConfirmationGraceMs: 60_000,
56+
});
57+
const generation = board.get('child-1')?.generation ?? 1;
58+
59+
await observeIdle(reconciler, 10, generation);
60+
expect(board.get('child-1')).toMatchObject({ state: 'running' });
61+
expect(terminalListener).not.toHaveBeenCalled();
62+
63+
board.markRunningFromLiveSession('child-1', 15);
64+
await observeIdle(reconciler, 16, generation);
65+
66+
expect(board.get('child-1')).toMatchObject({
67+
state: 'running',
68+
stopConfirmationStartedAt: 17,
69+
});
70+
expect(terminalListener).not.toHaveBeenCalled();
71+
});
72+
73+
test('repeated idle beyond confirmation grace becomes stopped exactly once', async () => {
74+
const { board, reconciler, terminalListener, contextFilesForPrompt, prune } =
75+
createHarness();
76+
const generation = board.get('child-1')?.generation ?? 1;
77+
78+
await observeIdle(reconciler, 10, generation);
79+
expect(board.get('child-1')).toMatchObject({ state: 'running' });
80+
expect(terminalListener).not.toHaveBeenCalled();
81+
82+
await observeIdle(reconciler, 20, generation);
83+
expect(board.get('child-1')).toMatchObject({
84+
state: 'stopped',
85+
terminalUnreconciled: true,
86+
});
87+
expect(terminalListener).toHaveBeenCalledTimes(1);
88+
expect(contextFilesForPrompt).toHaveBeenCalledTimes(1);
89+
expect(prune).toHaveBeenCalledTimes(1);
90+
91+
await observeIdle(reconciler, 30, generation);
92+
expect(board.get('child-1')).toMatchObject({ state: 'stopped' });
93+
expect(terminalListener).toHaveBeenCalledTimes(1);
94+
});
95+
96+
test('a busy observation resets pending stop confirmation', async () => {
97+
const { board, reconciler, terminalListener } = createHarness();
98+
const generation = board.get('child-1')?.generation ?? 1;
99+
100+
await observeIdle(reconciler, 10, generation);
101+
expect(board.get('child-1')?.stopConfirmationStartedAt).toBe(11);
102+
103+
board.markRunningFromLiveSession('child-1', 15);
104+
expect(board.get('child-1')).toMatchObject({
105+
state: 'running',
106+
stopConfirmationStartedAt: undefined,
107+
});
108+
109+
await observeIdle(reconciler, 20, generation);
110+
expect(board.get('child-1')).toMatchObject({ state: 'running' });
111+
expect(board.get('child-1')?.stopConfirmationStartedAt).toBe(21);
112+
expect(terminalListener).not.toHaveBeenCalled();
113+
});
114+
});

src/hooks/task-session-manager/idle-reconciliation.ts

Lines changed: 23 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,18 @@
11
import type { BackgroundJobStore, ContextFile } from '../../utils';
22
import { log } from '../../utils/logger';
33
import type { RevivedRunTracker } from './revived-run-tracker';
4+
import {
5+
observeNonBusyRuntime,
6+
STOP_CONFIRMATION_GRACE_MS,
7+
} from './stop-confirmation';
48

59
export function createIdleReconciler(options: {
610
backgroundJobBoard: BackgroundJobStore;
711
reconcileInjectedTerminalJobs: (parentSessionID: string) => void;
812
/** Called when a deferred inline error is terminalized at idle. */
913
onErrorTerminalize?: (sessionID: string) => void;
1014
idleReconcileDelayMs: number;
15+
stopConfirmationGraceMs?: number;
1116
isFallbackInProgress?: (sessionID: string) => boolean;
1217
hasInputWait: (sessionID: string) => boolean;
1318
getIdleSessionToken: (sessionID: string) => symbol;
@@ -84,19 +89,29 @@ export function createIdleReconciler(options: {
8489
if (terminalPublished) return;
8590
}
8691

87-
// Idle is a quiescent runner observation, not proof that the background
88-
// task ended. Keep the job live so a late terminal task result can win.
92+
const updated = observeNonBusyRuntime({
93+
backgroundJobBoard: options.backgroundJobBoard,
94+
taskID: sessionID,
95+
observedAt: idleObservedAt,
96+
generation: observedGeneration,
97+
graceMs: options.stopConfirmationGraceMs ?? STOP_CONFIRMATION_GRACE_MS,
98+
lastStatusError:
99+
'Runtime session is idle; task termination is unconfirmed.',
100+
taskContextTracker: options.taskContextTracker,
101+
});
102+
if (updated?.state === 'stopped') {
103+
log('[task-session-manager] confirmed runtime-stopped job from idle', {
104+
sessionID,
105+
alias: updated.alias,
106+
parentSessionID: updated.parentSessionID,
107+
});
108+
return;
109+
}
89110
log('[task-session-manager] observed quiescent job from idle', {
90111
sessionID,
91112
alias: job.alias,
92113
parentSessionID: job.parentSessionID,
93114
});
94-
options.backgroundJobBoard.markStatusUncertain(
95-
sessionID,
96-
'Runtime session is idle; task termination is unconfirmed.',
97-
observedGeneration,
98-
idleObservedAt,
99-
);
100115
}, options.idleReconcileDelayMs).unref?.();
101116
childIdleReconcileTimers.set(sessionID, timer);
102117
}

src/hooks/task-session-manager/runtime-status-reconciliation.test.ts

Lines changed: 132 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { createRuntimeStatusReconciler } from './runtime-status-reconciliation';
55
function createReconciler(
66
status: () => Promise<unknown>,
77
statusTimeoutMs?: number,
8+
stopConfirmationGraceMs?: number,
89
) {
910
const board = new BackgroundJobBoard();
1011
const contextFilesForPrompt = mock(() => []);
@@ -16,6 +17,7 @@ function createReconciler(
1617
} as never,
1718
backgroundJobBoard: board,
1819
statusTimeoutMs,
20+
stopConfirmationGraceMs,
1921
taskContextTracker: {
2022
pendingManagedTaskIds: new Set(['child-1']),
2123
contextFilesForPrompt,
@@ -256,13 +258,139 @@ describe('runtime status reconciliation', () => {
256258
});
257259
});
258260

259-
test('allows runtime busy to revive an acknowledged stopped job', () => {
261+
test('idle then busy inside grace remains running with no terminal listener', async () => {
262+
let liveStatus: unknown = { data: { 'child-1': { type: 'idle' } } };
263+
const { board, reconciler } = createReconciler(
264+
async () => liveStatus,
265+
undefined,
266+
60_000,
267+
);
268+
const listener = mock(() => {});
269+
board.addTerminalStateListener(listener);
270+
271+
await reconciler.reconcile();
272+
expect(board.get('child-1')).toMatchObject({
273+
state: 'running',
274+
statusUncertain: true,
275+
});
276+
expect(listener).not.toHaveBeenCalled();
277+
278+
liveStatus = { data: { 'child-1': { type: 'busy' } } };
279+
await reconciler.reconcile();
280+
281+
expect(board.get('child-1')).toMatchObject({
282+
state: 'running',
283+
statusUncertain: false,
284+
stopConfirmationStartedAt: undefined,
285+
});
286+
expect(listener).not.toHaveBeenCalled();
287+
});
288+
289+
test('repeated idle beyond confirmation grace becomes stopped exactly once', async () => {
290+
const { board, reconciler, contextFilesForPrompt, prune } = createReconciler(
291+
async () => ({ data: { 'child-1': { type: 'idle' } } }),
292+
undefined,
293+
0,
294+
);
295+
const listener = mock(() => {});
296+
board.addTerminalStateListener(listener);
297+
298+
await reconciler.reconcile();
299+
expect(board.get('child-1')).toMatchObject({ state: 'running' });
300+
expect(listener).not.toHaveBeenCalled();
301+
302+
await reconciler.reconcile();
303+
expect(board.get('child-1')).toMatchObject({
304+
state: 'stopped',
305+
terminalUnreconciled: true,
306+
});
307+
expect(listener).toHaveBeenCalledTimes(1);
308+
expect(contextFilesForPrompt).toHaveBeenCalledTimes(1);
309+
expect(prune).toHaveBeenCalledTimes(1);
310+
311+
await reconciler.reconcile();
312+
expect(board.get('child-1')).toMatchObject({ state: 'stopped' });
313+
expect(listener).toHaveBeenCalledTimes(1);
314+
});
315+
316+
test('a busy observation resets pending stop confirmation', async () => {
317+
let liveStatus: unknown = { data: { 'child-1': { type: 'idle' } } };
318+
const { board, reconciler } = createReconciler(
319+
async () => liveStatus,
320+
undefined,
321+
0,
322+
);
323+
const listener = mock(() => {});
324+
board.addTerminalStateListener(listener);
325+
326+
await reconciler.reconcile();
327+
expect(board.get('child-1')?.stopConfirmationStartedAt).toBeDefined();
328+
329+
liveStatus = { data: { 'child-1': { type: 'busy' } } };
330+
await reconciler.reconcile();
331+
expect(board.get('child-1')).toMatchObject({
332+
state: 'running',
333+
stopConfirmationStartedAt: undefined,
334+
});
335+
336+
await new Promise((resolve) => setTimeout(resolve, 2));
337+
liveStatus = { data: { 'child-1': { type: 'idle' } } };
338+
await reconciler.reconcile();
339+
expect(board.get('child-1')).toMatchObject({ state: 'running' });
340+
expect(board.get('child-1')?.stopConfirmationStartedAt).toBeDefined();
341+
expect(listener).not.toHaveBeenCalled();
342+
});
343+
344+
test('status lookup failure does not confirm a stop or wake the parent', async () => {
345+
let liveStatus: () => Promise<unknown> = async () => ({
346+
data: { 'child-1': { type: 'idle' } },
347+
});
348+
const { board, reconciler } = createReconciler(
349+
() => liveStatus(),
350+
undefined,
351+
0,
352+
);
353+
const listener = mock(() => {});
354+
board.addTerminalStateListener(listener);
355+
356+
await reconciler.reconcile();
357+
expect(board.get('child-1')?.stopConfirmationStartedAt).toBeDefined();
358+
359+
liveStatus = async () => {
360+
throw new Error('server restarting');
361+
};
362+
await reconciler.reconcile();
363+
364+
expect(board.get('child-1')).toMatchObject({
365+
state: 'running',
366+
statusUncertain: true,
367+
lastStatusError: 'Runtime status lookup failed: server restarting',
368+
});
369+
expect(board.get('child-1')?.stopConfirmationStartedAt).toBeDefined();
370+
expect(listener).not.toHaveBeenCalled();
371+
});
372+
373+
test('does not let stale busy revive a confirmed stopped job after terminal wake', () => {
374+
const { board } = createReconciler(async () => ({ data: {} }));
375+
const generation = board.get('child-1')?.generation;
376+
board.markStopped('child-1', 'no result', 150, generation, 150);
377+
board.markReconciled('child-1', 160);
378+
379+
board.markRunningFromLiveSession('child-1', 200, generation);
380+
381+
expect(board.get('child-1')).toMatchObject({
382+
state: 'stopped',
383+
terminalUnreconciled: false,
384+
lastLiveBusyAt: 200,
385+
});
386+
});
387+
388+
test('later live busy can still revive an unreconciled stopped job', () => {
260389
const { board } = createReconciler(async () => ({ data: {} }));
261390
const generation = board.get('child-1')?.generation;
262-
board.markStopped('child-1', 'no result', 1, generation);
263-
board.markReconciled('child-1');
391+
board.markStopped('child-1', 'no result', 150, generation, 150);
264392

265-
board.markRunningFromLiveSession('child-1', 2, generation);
393+
board.markRunningFromLiveSession('child-1', 200, generation);
266394

267395
expect(board.get('child-1')).toMatchObject({
268396
state: 'running',

0 commit comments

Comments
 (0)