harness: level-triggered forever watch can stampede duplicate Agent Runs

[rkrkrkk]

Summary

A managed vibe watch --forever can create an unbounded burst of duplicate Agent Runs when its waiter is level-triggered and keeps returning exit code 0 after a condition becomes true.

This is easy to hit when an Agent needs "retry until ready, then resume once": retry exit codes are only retried in forever mode, but forever also repeats after success. The two lifecycle requirements are currently coupled.

Observed incident

On 2026-07-22, a watch polled a local provider-readiness predicate:

• while not ready, it returned 75 and used a 30-second retry delay;
• once ready, it returned 0 and remained true;
• the watch targeted one existing Agent Session.

After the predicate became true, the watch created 274 real follow-up runs in 18.139 seconds (about 15 runs/second):

• 174 were dequeued and marked succeeded, mostly as fast no-op turns;
• 100 remained queued and had to be canceled manually;
• the watch itself had to be removed to stop further growth.

The numbers above exclude the synthetic runtime:<definition_id> run row. No credentials, prompts, or user data are included here.

Current behavior

The successful-cycle path:

1. enqueues a follow-up for every exit code 0;
2. disables only once watches;
3. immediately continues for forever watches, with no delay or edge detection.

See core/watches.py lines 674-692.

Each event then creates a fresh run with no coalescing by definition/session. See core/watches.py lines 793-816 and core/scheduled_tasks.py lines 947-980.

retry_delay_seconds applies to configured retry exit codes, but not to successful cycles.

Minimal reproduction shape

1. Create a forever watch whose command returns 75 until a local condition is true.
2. Make the condition true so the command returns 0 on every subsequent invocation.
3. Observe vibe runs list --definition-id <watch-id> --all.

The run count grows continuously until the watch is paused or removed.

Expected behavior

Avibe should safely support "retry until the first success, enqueue one follow-up, then stop" without requiring every Agent to hand-roll a blocking/edge-triggered waiter.

A single configuration mistake should also not create unbounded Agent Runs.

Suggested acceptance criteria

• Add an explicit until-success lifecycle, or allow once watches to retry configured retry exit codes but stop after the first 0.
• Prevent more than one queued/running follow-up for the same watch definition and target Session, or coalesce repeated events while one is in flight.
• Add a success-cycle interval, rate limit, or circuit breaker for forever watches so a persistent 0 cannot spin.
• Preserve legitimate edge-triggered forever watches that block until each new event.
• Add a regression test with a waiter that returns persistent 0 and assert bounded follow-up creation.
• Surface a warning or terminal error when a watch exceeds a short-window trigger threshold.

Root-cause classification

The incident was triggered by an Agent choosing the wrong waiter shape, but the blast radius is a Harness safety/design issue: forever conflates retry-until-ready with repeat-after-success, and the queue has no deduplication, backpressure, or fuse for one watch definition.