fix(agent): escalate stream retry delays and honor Retry-After

[lorenzozanee]

Summary

• Escalate the single stream-retry delay across consecutive retryable failures at both retry sites (swarm worker and AgentLoop) instead of sleeping a fixed constant.
• Honor Retry-After on 429/529 by extracting it once onto ProviderStreamError, so every current and future retry site can use it.

Why

Closes #1208 — the two layers left open after #1210. During a sustained provider overload each site still burned its single retry after a flat 1s, and nothing in this repo's own LLM retry layers reads Retry-After. The OpenAI SDK honors it for pre-stream errors, but mid-stream failures (connection reset, no headers) and the swarm/agent-loop layers compose to exhaust the retry budget in seconds. Retry-After extraction lives on ProviderStreamError at the provider layer, per the discussion on #1208, so every retry site benefits.

Changes

• ProviderStreamError carries retry_after_s, parsed defensively from the original exception's response headers (absent/garbage → None, no behavior change for callers that ignore it).
• Both retry sites track consecutive retryable stream failures: the first failure sleeps the configured SWARM_STREAM_RETRY_DELAY_S / VT_STREAM_RETRY_DELAY_S exactly as before, each consecutive failure doubles the delay (capped, exponent clamped at 62 like the worker-level backoff), and only a clean iteration resets the streak — a successful retry does not.
• A provider-suggested Retry-After replaces the exponential for that attempt, bounded by the new SWARM_STREAM_RETRY_MAX_DELAY_S / VT_STREAM_RETRY_MAX_DELAY_S knobs (default 30s) so a broken header cannot freeze a run.
• New knobs follow fix(swarm): add bounded backoff to worker retries #1210's pattern: ge=0, cap >= base validation; SWARM_STREAM_RETRY_MAX_DELAY_S documented in agent/.env.example (VT_* stream tuning has never been listed there, so the VT knob follows existing precedent).
• No jitter at this layer: a task has at most one in-flight stream retry; swarm-wide de-synchronization is already handled by fix(swarm): add bounded backoff to worker retries #1210's equal-jitter worker backoff.
• Deliberately out of scope: SDK-level retries, retry counts, and the worker-level backoff from fix(swarm): add bounded backoff to worker retries #1210 (unchanged).

Test Plan

• Existing tests pass (pytest agent/tests/test_swarm_worker_stream_retry.py agent/tests/test_agent_loop_stream_retry.py agent/tests/test_swarm_worker_retry_backoff.py agent/tests/test_swarm_worker_content_filter.py agent/tests/test_chat_llm_streaming.py agent/tests/test_env_schema.py -q → 137 passed)
• New tests added (agent/tests/test_stream_retry_escalation.py, 23 tests; 160 passed for the seven suites together): escalation across consecutive iterations with successful retries, streak reset on a clean iteration, first-failure backward compatibility, Retry-After honored/clamped/garbage-fallback, extraction unit tests, config validation. All provider behavior mocked; no live provider calls.
• py_compile / compileall passed on all changed files
• Tested manually (no live provider call was made)

Checklist

• No changes to protected areas (src/agent/, src/session/, src/providers/) without prior discussion — the changes to src/agent/loop.py and src/providers/chat.py implement exactly the layers flagged as open and accepted on Retry delays don't escalate across the three retry layers — a sustained provider overload burns every retry budget in seconds #1208, with provider-layer placement per the maintainer comment there
• No hardcoded values (API keys, file paths, magic numbers)
• Code follows CONTRIBUTING.md guidelines
• Documentation updated (agent/.env.example)

Risk and boundary

No broker, live-trading, MCP, network, secret, or deployment behavior is changed. Retry counts are unchanged (one stream retry per site). With the new cap knobs at their defaults the worst added wait per attempt is 30s; setting the existing SWARM_STREAM_RETRY_DELAY_S=0 restores immediate stream retry timing. Rollback: revert the commit.

[warren618]

Merged. Putting retry_after_s on ProviderStreamError is the right place — that was the shape suggested on #1208 and it means every current and future retry site gets it. The escalation, the cap validation, and else: streak = 0 resetting only on a clean first attempt all check out.

One follow-up pushed on top (ba23b6ed). The delay went from a flat 1.0s to a capped exponential reaching 30s, and a provider Retry-After can request the cap on the very first failure — but both sites still served it with a blocking sleep. Pressing Stop mid-backoff was therefore not observed until the whole delay elapsed, and the retry stream was issued afterwards anyway.

Both sites now wait on the cancellation event, which returns the moment it is set, and skip the retry when it is. The swarm worker keeps time.sleep when no cancel_event was injected. The regression test asserts the 30s delay is requested but the run returns in under 5s with the retry stream never issued; against a plain sleep it takes 37s and fails.