Turn guard 409s internal retries against a stuck in-flight turn, producing a fatal client-facing stream error and false account cooldowns #18716

Summary

When an in-flight turn stalls, the turn execution guard (acquireTurnExecution, open-sse/handlers/chatCore/turnExecutionGuard.ts) 409s not only genuine client retries but also OmniRoute's own executor/account-fallback re-entries of the same request. Each internal retry carries the caller's headers and an identical body, so it maps to the same effective idempotency key and is blocked while the original turn is still held. The result is a cascade that fails the client request and poisons healthy accounts:

1. Original turn stalls (slow upstream, proxy blip) and holds the guard slot.
2. The executor retries (RETRY codex/x retryable pre-output 503 — retrying same account once) or the account fallback advances to the next account. Both re-enter chatCore with the same headers and body.
3. chatCore's guard returns 409 turn_in_progress for every one of those re-entries.
4. The account layer treats the 409 as an account-level rate limit: Account ... unavailable (409), trying fallback → codex | all N active accounts rate limited (reset after 5s) | lastErrorCode=409.0, lastError=An identical request is already in progress → COOLDOWN_RETRY ... waiting 5s before retry, which collides again.
5. The combo surfaces [target/model] An identical request is already in progress (reset after 5s) mid-stream (stream_error). Clients that classify that in-stream error as non-retryable lose the whole turn, even though the guard's own response headers advertise Retry-After: 1.

Production evidence

One long-running deployment, ~/.omniroute/logs/application/app.log, 3 weeks of logs:

• 1,450 TURN_GUARD duplicate blocked lines across 230 distinct incidents.
• Typical originals are held 10-33s; one cluster on Oct 31 (18:31-19:02 UTC) held originals for 100-244s while duplicates kept arriving.
• Cascade excerpt (2026-11-01T05:18:12-23Z, one stuck original, six blocked re-entries):

RETRY     codex/gpt-6-astra retryable pre-output 503 — retrying same account once after 2680ms
TURN_GUARD duplicate blocked cid=e61d6cc4 retry=1 ageMs=12159
AUTH      Account 68ac9e36... unavailable (409), trying fallback
TURN_GUARD duplicate blocked cid=9c586bd6 retry=2 ageMs=13060
AUTH      codex | all 2 active accounts rate limited (reset after 5s) | lastErrorCode=409.0, lastError=An identical requ...
COOLDOWN_RETRY codex/gpt-6-astra all connections cooling down (reset after 5s) — waiting 5s before retry 1/3

The reset after 5s in the client-visible message is formatResetSuffix output from that account cooldown, so a request-scoped self-collision is being reported as if accounts were rate limited.

Proposed direction

Serialize duplicates instead of failing them: on a blocked acquire, wait (bounded, e.g. 30s, polling a few hundred ms) for the in-flight turn to release, then acquire. After the bound, fall back to today's 409. This keeps the guard's stated guarantee (a client retry must not open a second upstream execution) while internal retries and patient clients survive a stuck original instead of cascading into account cooldowns.

I have a patch plus tests ready and will open a PR referencing this issue.