[Bug] Parallel executor consumer-side wakeup loss: AC task parks on async_generator_asend while the event loop is idle and all timers are dead
Version: ouroboros-ai 0.51.15 (also reproduced identically on CPython 3.13.11 — not Python-version specific)
Component: orchestrator/parallel_executor.py (_execute_ac_batch / _stream_provider_call) + orchestrator/leaf_dispatcher.py (dispatch stream) + anyio task groups
Runtime backend: pi (pi --mode json); likely backend-independent (the same signature appeared in the LLM-adapter path used by bounce decomposition)
Summary
During ooo auto executions with multiple ACs, an AC worker task permanently stops consuming its provider generator. The affected task awaits an async_generator_asend whose generator reports ag_await is None (parked at a yield) — an await that should resolve immediately — while:
- the event loop is idle in epoll (faulthandler thread dump of the main thread shows
selectors.py select → base_events._run_once → run_forever), i.e. nothing is runnable;
- every timer is dead: the anyio
CancelScope stall deadline (STAL_TIMEOUT_SECONDS), a nested asyncio.wait(timeout=...), and asyncio.wait_for timeouts all fail to fire in this state;
- downstream backpressure propagates: the provider subprocess wrapper blocks writing to its stdout pipe, stderr-collector tasks accumulate, and the affected AC never reaches any terminal state — the whole execution freezes until externally killed or the top-level pipeline deadline.
The stall-retry ladder therefore never runs (its cancellation is "delivered" by the scope but never lands in the parked task), so each occurrence burns the attempt only via the outermost watchdogs. Over a long execution this exhausts per-AC retry budgets and the run ends failed.
Reproduction
- ~19 full
ooo auto --runtime pi --timeout 21600 runs on a real repo audit workload; the hang appears within 30–90 minutes of execution in nearly every run (more often with concurrent ACs).
- Provider: self-hosted OpenAI-compatible router that intermittently fails long-context requests (
terminated/502) after which the provider CLI sits silently for many minutes — the parked state is entered around such windows, but the parked state itself persists forever even after the provider process dies.
Forensic signature (SIGURG asyncio task dump, cr_await chain walk)
Captured with a dual-signal instrumentation (SIGUSR2 = faulthandler all-thread dump; SIGURG = Python handler walking asyncio.all_tasks() along cr_await, expanding ag_frame/ag_await — Task.get_stack() truncates at future waits and is insufficient). Full dumps attached (taskdump-17c.txt, taskdump-19.txt). Key frames:
--- task ParallelACExecutor._execute_ac_batch.<locals>._run_ac
[0] anyio/_core/_tasks.py:275 in _run_coro
[1] parallel_executor.py:3696 in _run_ac
[2] parallel_executor.py:6713 in _execute_single_ac
[3] parallel_executor.py:8688 in _execute_atomic_ac
[4] parallel_executor.py:8624 in _stream_provider_call
[5] leaf_dispatcher.py:705 in stream
[6] (awaiting async_generator_asend) # generator: ag_await=None → parked at a yield
--- (queue variant, run #19)
task PiRuntime.execute_task.<locals>._pump_stream
[0] pi_runtime.py:893 in _pump_stream
[1] asyncio/queues.py:120 in put # backpressured: consumer never resumes
Sibling evidence:
agent_process:execute_seed:... task awaits the batch task-group __aexit__ forever;- 5+ leaked
PiRuntime._collect_stream_lines tasks per hang (one per stalled attempt);
- two
_run_ac tasks parked on provider_admission.wait (AdmissionSequence Event) — they never proceed because the predecessor is the parked task;
- faulthandler: main thread idle in
select; all worker threads parked in aiosqlite/waitpid.
What we ruled out (all reproduced the hang anyway)
- Generator structure: replaced the nested
async for consumption inside the runtime with an independent pump task + asyncio.Queue and a single flat await queue.get() — the dispatcher then parked the same way (and the pump backpressured on put).
- In-generator keepalives/timers: a 240 s
asyncio.wait(timeout=) keepalive around the stream read never fired during the parked state (0 occurrences across all runs); wait_for idle watchdogs equally dead.
- Python version: rebuilt the entire environment on CPython 3.13.11 (was 3.12.12) — identical signature and frequency.
- Provider side: an external process watchdog kills a silently-hung provider after 600 s; the parked dispatcher does not notice (its generator never resumed to observe the stream EOF).
Suspected mechanism
A task awaiting a coroutine-chain that ends in an async generator asend — with the generator suspended at yield — under anyio's asyncio backend inside CancelScopes/task groups: some interaction (possibly anyio's cancel-scope bookkeeping with Task.uncancel() on 3.11+, or a generator athrow/aclose racing the asend) leaves the awaiter with nothing scheduled: the loop is idle, no timer will wake it, and scope cancellation no longer lands. The task is permanently runnable-but-never-run.
Workaround we deployed locally (for other users)
A hard watchdog in the dispatcher: consume the provider generator in a pump task; the dispatch loop pulls from a queue with asyncio.wait(timeout=stall+grace); on timeout, pump_task.cancel() — a direct task-level cancel that always lands in a parked asend — then set dispatch_state.stalled = True so the existing stall-retry ladder recovers the attempt. This converts the permanent freeze into a normal stall-restart. (Also: MAX_STALL_RETRIES raised, STALL_TIMEOUT lowered, per-AC activity heartbeat for observability.)
A minimal reproducer outside ouroboros has not yet been isolated; the dual-signal dump instrumentation is trivially portable and I'm happy to run suggested probes against a live hang (we can trigger one within ~1 hour on demand).
Environment
- ouroboros-ai 0.51.15 (uv tool env), CPython 3.12.12 and 3.13.11, Linux x86_64
- anyio version bundled with 0.51.15; asyncio backend
- Attachments: taskdump-17c.txt, taskdump-19.txt (full SIGURG dumps), plus a faulthandler thread dump available on request
[Bug] Parallel executor consumer-side wakeup loss: AC task parks on async_generator_asend while the event loop is idle and all timers are dead
Version: ouroboros-ai 0.51.15 (also reproduced identically on CPython 3.13.11 — not Python-version specific)
Component:
orchestrator/parallel_executor.py(_execute_ac_batch/_stream_provider_call) +orchestrator/leaf_dispatcher.py(dispatch stream) + anyio task groupsRuntime backend: pi (
pi --mode json); likely backend-independent (the same signature appeared in the LLM-adapter path used by bounce decomposition)Summary
During
ooo autoexecutions with multiple ACs, an AC worker task permanently stops consuming its provider generator. The affected task awaits anasync_generator_asendwhose generator reportsag_await is None(parked at ayield) — an await that should resolve immediately — while:selectors.py select → base_events._run_once → run_forever), i.e. nothing is runnable;CancelScopestall deadline (STAL_TIMEOUT_SECONDS), a nestedasyncio.wait(timeout=...), andasyncio.wait_fortimeouts all fail to fire in this state;The stall-retry ladder therefore never runs (its cancellation is "delivered" by the scope but never lands in the parked task), so each occurrence burns the attempt only via the outermost watchdogs. Over a long execution this exhausts per-AC retry budgets and the run ends
failed.Reproduction
ooo auto --runtime pi --timeout 21600runs on a real repo audit workload; the hang appears within 30–90 minutes of execution in nearly every run (more often with concurrent ACs).terminated/502) after which the provider CLI sits silently for many minutes — the parked state is entered around such windows, but the parked state itself persists forever even after the provider process dies.Forensic signature (SIGURG asyncio task dump, cr_await chain walk)
Captured with a dual-signal instrumentation (SIGUSR2 = faulthandler all-thread dump; SIGURG = Python handler walking
asyncio.all_tasks()alongcr_await, expandingag_frame/ag_await—Task.get_stack()truncates at future waits and is insufficient). Full dumps attached (taskdump-17c.txt,taskdump-19.txt). Key frames:Sibling evidence:
agent_process:execute_seed:...task awaits the batch task-group__aexit__forever;PiRuntime._collect_stream_linestasks per hang (one per stalled attempt);_run_actasks parked onprovider_admission.wait(AdmissionSequence Event) — they never proceed because the predecessor is the parked task;select; all worker threads parked in aiosqlite/waitpid.What we ruled out (all reproduced the hang anyway)
async forconsumption inside the runtime with an independent pump task +asyncio.Queueand a single flatawait queue.get()— the dispatcher then parked the same way (and the pump backpressured onput).asyncio.wait(timeout=)keepalive around the stream read never fired during the parked state (0 occurrences across all runs);wait_foridle watchdogs equally dead.Suspected mechanism
A task awaiting a coroutine-chain that ends in an async generator
asend— with the generator suspended atyield— under anyio's asyncio backend insideCancelScopes/task groups: some interaction (possibly anyio's cancel-scope bookkeeping withTask.uncancel()on 3.11+, or a generatorathrow/acloseracing the asend) leaves the awaiter with nothing scheduled: the loop is idle, no timer will wake it, and scope cancellation no longer lands. The task is permanently runnable-but-never-run.Workaround we deployed locally (for other users)
A hard watchdog in the dispatcher: consume the provider generator in a pump task; the dispatch loop pulls from a queue with
asyncio.wait(timeout=stall+grace); on timeout,pump_task.cancel()— a direct task-level cancel that always lands in a parked asend — then setdispatch_state.stalled = Trueso the existing stall-retry ladder recovers the attempt. This converts the permanent freeze into a normal stall-restart. (Also: MAX_STALL_RETRIES raised, STALL_TIMEOUT lowered, per-AC activity heartbeat for observability.)A minimal reproducer outside ouroboros has not yet been isolated; the dual-signal dump instrumentation is trivially portable and I'm happy to run suggested probes against a live hang (we can trigger one within ~1 hour on demand).
Environment