Skip to content

fix(sse): scale the direct response-start window by request payload size - #18404

Open
jjxyxsjr wants to merge 2 commits into
cheaper-inference:release/v3.8.53from
jjxyxsjr:fix/direct-headers-volume-tier
Open

jjxyxsjr wants to merge 2 commits into
cheaper-inference:release/v3.8.53from
jjxyxsjr:fix/direct-headers-volume-tier

Conversation

@jjxyxsjr

@jjxyxsjr jjxyxsjr commented Oct 28, 2026 •

Copy link
Copy Markdown

Problem and behavior

Large request bodies can legitimately sit silent well past the flat 30s direct response-start window: prefill scales with input size, and a plain large-context request carries no reasoning_effort selector for the existing reasoning ceiling (REASONING_READINESS_CEILING_MS) to key on — so the fetch-layer guard cut healthy slow-prefill upstreams at exactly the floor and surfaced 504s the stream-readiness layer would have permitted.

Changes

  • resolveFlatDirectHeadersFloorMs raises the flat floor by payload tier: ≤512 KB unchanged (a small body overrunning the probe still smells like a dead connection, per fix(network): direct dispatcher stalls opencode-go and command-code until service restart #10214), >512 KB ×2, >2 MB ×3, capped at 180 s — composing with the existing effort/local raises via raise-only semantics.
  • New env OMNIROUTE_DIRECT_HEADERS_VOLUME_ADAPTIVE (0/false/off pins the flat behavior), registered in .env.example and docs/reference/ENVIRONMENT.md.

Validation

TDD — tests/unit/direct-response-start-timeout-volume-tier.test.ts failed before the fix on 4/7 (the resolver returned the flat 30000 for every tier), passes 7/7 after:

not ok 2 - 2. >512KB doubles the floor                       (expected 60000, got 30000)
not ok 3 - 3. >2MB triples the floor                         (expected 90000, got 30000)
not ok 4 - 4. tier raise is capped at the 180s volume ceiling (expected 180000, got 30000)
not ok 7 - 7. resolver binds the body once per request        (expected 60000, got 30000)
  • Regressions: direct-response-start-timeout-reasoning-504.test.ts + direct-response-start-timeout-settled-guard-12861.test.ts — 17 pass.
  • npm run check:env-doc-sync in sync · eslint clean on the touched module.
  • Only the files the failing test proves need changing are touched; no test weakened or deleted.

Notes

  • Raise-only composition: the volume tier applies last as max(base, min(base * factor, 180s)), composes order-independently with the effort and local/LAN raises, and can never pull the window below an operator override or the local floor (test 4 pins the cap, test 5 pins the never-lower rule).
  • Byte length uses TextEncoder on the already-serialized body string the resolver binds once per request (same binding the reasoning-effort regex uses) — no extra serialization on the request path.
  • Default (OMNIROUTE_DIRECT_HEADERS_VOLUME_ADAPTIVE unset) = enabled tiering as described; the kill switch restores today's flat behavior exactly.
  • Base: release/v3.8.53 (active release branch); aware of the 3.8.55 v4-feature retarget policy.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant