Tasks prematurely marked NOT_NEEDED via 'overlay: tree did not exercise' before agent gets a chance to run them

[pedapudi]

Summary

In a v18 validation run (session 8df93041-5ab8-4c03-ba79-a8c155f74417, post-#319 fix landed at 1de1800), turn 1's draft_slides and review_presentation tasks were marked NOT_NEEDED at 22:28:35 and 22:30:36 — only ~2 minutes after research_panels completed, and before any agent attempted to run them. The slideshow was never produced for turn 1 as the user requested.

The reason text on the task_cancelled events is:

not_needed: overlay: tree did not exercise; no follow-up dispatched (goldfive#16…

Tasks that should run downstream of a completed task are getting reaped because the overlay sees no dispatch — but the dispatch was about to happen on the next turn or via the coordinator's normal flow.

Reproduction

Drive the v18 4-turn presentation matrix (solar panels → solar flares → save location → 2 slide constraint). In turn 1:

seq=18 22:27:16 task_transitioned    research_panels RUNNING->COMPLETED
seq=29 22:28:35 task_cancelled       reason='not_needed: overlay: tree did not exercise; no follow-up dispatched (goldfive#16…'
seq=30 22:28:35 task_transitioned    draft_slides PENDING->NOT_NEEDED
seq=40 22:30:36 task_cancelled       reason='not_needed: overlay: tree did not exercise...'
seq=41 22:30:36 task_transitioned    review_presentation PENDING->NOT_NEEDED

draft_slides and review_presentation go from PENDING to NOT_NEEDED without ever entering RUNNING.

The slideshow was eventually produced — but in turn 3 (~22:40:11), triggered by the user's factual question 'Where will the slides be saved?'. The system silently re-exercised draft + review tasks 9+ minutes into turn 3.

Why it matters

• User asks for a slideshow → no slideshow produced for that turn.
• Drift cascade fires GOAL_DRIFT sev=3 + HUMAN_INTERVENTION_REQUIRED sev=3 at the end of turn 1 (seq=21,22) — goldfive itself notices the situation but the run completes anyway.
• Pattern repeats at the end of turn 3 (seq=115,116).

Proposed direction

The overlay's 'tree did not exercise' heuristic is too eager. Possible fixes:

1. Don't mark downstream tasks NOT_NEEDED until the run is actually exiting (i.e., the coordinator has signaled completion or no-progress).
2. Require multiple turns of inactivity before declaring a task NOT_NEEDED.
3. Tie the heuristic to the structural plan: only reap tasks whose dependency chain has visibly stalled (no upstream RUNNING, no in-flight invocation).

Reference: the reason text mentions goldfive#16; that issue's old design may be load-bearing here.

Scope

This is the highest-priority functional regression from the v18 run. Pairs with #319 (judge cancellation, fixed) but is independent.

[pedapudi]

Code-validated root cause

The original (a)/(b)/(c) options miss the actual mechanism. Direct code investigation:

Reaper emit site: goldfive/executors/sequential.py:963-990 in SequentialExecutor._run_overlay:

live_plan = session.plan or plan
pending_ids = [t.id for t in list(getattr(live_plan, "tasks", None) or ())
               if t.status is TaskStatus.PENDING and t.id]
if pending_ids:
    ...
    for tid in pending_ids:
        await steerer.transition(tid, TaskStatus.NOT_NEEDED,
            detail="overlay: tree did not exercise; no follow-up dispatched (goldfive#163)", ...)

Trigger: turn boundary, NOT a drift handler. The reaper fires unconditionally the moment _run_overlay's while True loop breaks out (kind == "result"). No inactivity window, no DAG check, no escalation gate.

The smoking gun: _invoke_passthrough_with_control (lines 1039-1128) does NOT honor session.paused_for_human_intervention. The per-task path _apply_pre_task_controls (1156-1228) DOES, but the overlay path never calls it.

Sequence in v18:

1. Research completes → judge fires GOAL_DRIFT sev=3 → ladder routes to PAUSE_ESCALATE (steerer.py:2966-2970) → escalation primitive emits HUMAN_INTERVENTION_REQUIRED + sets paused_for_human_intervention=True (steerer.py:5502-5528)
2. Meanwhile the in-flight overlay invocation just keeps running, returns naturally
3. Reaper instantly sweeps every PENDING task to NOT_NEEDED

The escalation primitive runs in parallel with the executor and they don't coordinate in the overlay path — the same bug class as the missing paused_for_human_intervention honor in _invoke_passthrough_with_control.

Goldfive#163 lineage

The reason text references goldfive#163 (truncated to "#16…" in the UI). #163's design intent: "After the root invocation completes, any PENDING task becomes NOT_NEEDED. The tree did what it naturally does." Conservative fix to a 40+min flow-coordinator amplification bug. The current behavior matches that intent — but #163 assumed "natural completion" means the coordinator consciously chose not to run X. It does NOT account for goldfive's drift system interrupting the coordinator before its flow reached X.

Correct fix (different from original (a)/(b)/(c))

Two small additions, not a rewrite:

1. Honor paused_for_human_intervention in _invoke_passthrough_with_control — cancel the in-flight invoke and pause-block or abort, the same way _apply_pre_task_controls does for per-task dispatch.
2. Gate the reaper on "no escalation pending" — if paused_for_human_intervention is True at invocation end, OR a HUMAN_INTERVENTION_REQUIRED / GOAL_DRIFT (CRITICAL) was emitted during this invocation, don't reap. Either leave PENDING for the next user turn, or abort the run with the drift's reason.

Important constraint: the comments at sequential.py:842-856 and 938-960 document _MAX_NUDGE_REPLAYS as an explicit safeguard against re-introducing #163 amplification. A naive "re-invoke once more on drift" fix will be (correctly) rejected by that safeguard. The clean fix is the orthogonal one above.

Files to read for the fix

• goldfive/executors/sequential.py lines 772-990 (overlay loop), 1039-1128 (invoke control, missing pause honor), 1140-1230 (per-task drain showing the correct shape)
• goldfive/steerer.py lines 2966-2970 (GOAL_DRIFT ladder), 5502-5528 (escalation pause flag setter)
• docs/design/STATE-MACHINE.md:50,154 (current documented design)

[pedapudi]

Re-investigation: previous diagnosis is correct on mechanism but proposed fix only stops the lying, not the looping

Verified by reading the code directly:

Mechanism confirmed:

• The overlay's invoke returns naturally (kind == "result" at sequential.py:870-927), reaches the reaper at line 963.
• v18's run_completed event at seq=44 (not run_aborted) confirms the overlay path went through the natural-completion branch, not the cancelled branch (line 870-886 returns early on cancelled).
• invocation_cancelled events at seq=25, 27 are on child agent invocations (the looping research_agent dispatches), not the overlay's outer Future.

Validator-side judge text confirms the actual root cause goes deeper:

"goal drift detected: The recent activity re-invokes the research agent despite the research task already being marked completed, indicating a loop rather than advancing to the pending drafting task."

The judge correctly identified the right corrective action (move to drafting). Goldfive then routes GOAL_DRIFT sev=3 → PAUSE_ESCALATE (Level 4) → emits HUMAN_INTERVENTION_REQUIRED + sets paused_for_human_intervention=True. This is a pause + escalate-to-user, not a nudge-the-agent-forward.

The _LADDER_BY_KIND table at steerer.py:2966-2970 has GOAL_DRIFT routing only to PAUSE_ESCALATE — there is no "redirect to next plan stage" transition for it. The nudge mechanism (compose_corrective_user_message, steerer.py:2812/2834/3049) is wired for ABSORB/REDIRECT levels (1/2), not for sev=3 escalation.

Two-part diagnosis

Bug A (immediate, what my proposed fix addresses): Overlay reaper conflates "no agent dispatched" with "task obviated." Even after sev=3 escalation, the natural-completion branch fires the reaper, which marks downstream PENDING tasks NOT_NEEDED — silently lying about the user's intent.

Bug B (deeper, what my proposed fix does NOT address): Goldfive lacks an automatic "advance to next plan stage" steering action for GOAL_DRIFT. The judge has the necessary information (research is done, drafting is next), but no mechanism translates that into a corrective_user_message or a coordinator-steering move. So even with Bug A fixed, the agent will still loop — it just won't have downstream tasks falsely marked NOT_NEEDED.

Honest fix shape

Bug A (file as a sub-issue or add scope to this one):

• Honor paused_for_human_intervention in _invoke_passthrough_with_control (sequential.py:1039-1128) — same shape as the per-task path's _apply_pre_task_controls (lines 1156-1228) already does.
• Gate the reaper at line 977 on "no escalation pending."
• Result: tasks stay PENDING for next turn instead of being lied-about as NOT_NEEDED. User can drive forward with another input.

Bug B (separate, larger): add a "stage-advance nudge" path for GOAL_DRIFT when the judge's evidence shows a completed task being re-invoked. This requires:

• Steerer ladder entry for GOAL_DRIFT that routes to REDIRECT (not just PAUSE_ESCALATE) when the next-plan-stage is unambiguous from the plan DAG.
• A compose_advance_message analog that injects a nudge like "task X is complete; please proceed to task Y."
• The existing _MAX_NUDGE_REPLAYS safeguard (sequential.py:842-856) protects against runaway re-invokes — this would consume a replay slot, which is acceptable because it's bounded.

The two fixes are independent. Bug A alone is a useful correctness/honesty improvement. Bug B alone wouldn't help if Bug A still wipes tasks. Together they actually deliver "user asked for slides, slides got drafted."

Why this matters for the user's challenge

The originally proposed fix (gate reaper on escalation pending) is a correct partial fix — it stops goldfive from compounding the problem. But it does not deliver the user's requested outcome on its own: the agent will still loop on research and the user will still need to manually steer. The deeper Bug B is the loop itself.