fix(proxy): stop capping every generation at aiohttp's 300s default

[FarukZahiragic]

The bug

The inference proxy built its session as a bare aiohttp.ClientSession():

session = aiohttp.ClientSession()

That inherits aiohttp's default ClientTimeout(total=300). total covers the whole exchange, including every streamed chunk, so the gateway has been aborting any generation that runs past five minutes — no matter how healthy the upstream is. Nothing in the code says 300 anywhere; it's an inherited default, which is why it wasn't obvious.

How it surfaced

The evals team hit it right after we lifted their rate limit: benchmarks whose tasks carry a large max_gen_toks fail with request timeouts. Their harness (lm-eval via --backend openai, local-completions) does not stream, so a single HTTP request spans the entire completion. A long reasoning answer on a 70B model clears 300s easily.

It isn't evals-specific, though — the same ceiling truncates any long chat answer, streaming included, since total doesn't care that chunks are still flowing.

The fix

Pass the timeout explicitly, shaped to how each mode reports progress:

• Streaming — sock_read, no total. Chunks arrive continuously, so a gap means the upstream died, whereas elapsed time means only that the answer is long. Bounding the gap lets a legitimately long generation finish while still reaping dead connections.
• Non-streaming — an overall cap. There's no gap to measure because nothing comes back until the completion is done, so one long silence is the expected shape and a total is the only bound available.

Both are configurable (UPSTREAM_TIMEOUT_SECONDS, default 3600; UPSTREAM_STREAM_STALL_SECONDS, default 300). sock_connect stays at 30s so an unreachable upstream still fails fast rather than hanging for an hour.

Deploy dependency

The prod ingress sets proxy-read-timeout: "600", which non-streaming requests will now hit before the new cap. That needs raising in the same window or this only moves the ceiling from 5 minutes to 10 — breithorn-poc/rob-poc PR to follow.

Test plan

• Full suite green (149 passed)
• New test_upstream_timeout.py covers both modes, including a regression guard that neither path can inherit total=300 again
• After deploy: a non-streaming completion with a large max_tokens that runs >5 min returns a result instead of an error
• A streaming request stays open well past 5 min while tokens flow
• A wedged upstream still gets reaped rather than held open

Made with Cursor