feat(openai): prevent silent streaming hangs in ChatOpenAI#36949
Merged
Mason Daugherty (mdrxy) merged 13 commits intomasterfrom Apr 23, 2026
Merged
feat(openai): prevent silent streaming hangs in ChatOpenAI#36949Mason Daugherty (mdrxy) merged 13 commits intomasterfrom
ChatOpenAI#36949Mason Daugherty (mdrxy) merged 13 commits intomasterfrom
Conversation
…ean up comments/docstrings
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Behavior change on upgrade — minor bump (
1.1.16→1.2.0).Streaming calls now raise
StreamChunkTimeoutError(aTimeoutErrorsubclass — existingexcept TimeoutError:/except asyncio.TimeoutError:handlers catch it) after 120s of content silence instead of hanging forever. Opt out withstream_chunk_timeout=NoneorLANGCHAIN_OPENAI_STREAM_CHUNK_TIMEOUT_S=0.Kernel-level TCP keepalive /
TCP_USER_TIMEOUTare applied via a customhttpxtransport.httpxdisables its env-proxy auto-detection (HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/NO_PROXYand macOS/Windows system proxy) whenever a transport is supplied, so to avoid silently breaking enterprise proxy users,ChatOpenAInow detects the "proxy-env-shadow" shape at construction and skips the custom transport entirely when all of these hold:http_socket_optionsleft at default (None)http_clientorhttp_async_clientsuppliedopenai_proxysuppliedOn that shape the instance falls back to pre-PR behavior and env-proxy auto-detection still applies. A one-time
INFOrecords the bypass.Users who explicitly set
http_socket_options=[...]alongside an env proxy still get the shadowed behavior with a one-timeWARNINGlog — they opted in. Full opt-outs below.Streaming chat completions can hang forever when the underlying TCP connection silently dies mid-stream (idle NAT/LB timeouts, sandboxed runtimes killing long-lived connections, peer gone without a FIN or RST). httpx's read timeout doesn't help here because it's reset by any bytes arriving on the socket, including OpenAI's SSE keepalive comments, so a stream that's quiet on content but still producing keepalives looks alive forever.
This PR adds two knobs to
ChatOpenAI, both on by default with opt-outs:stream_chunk_timeout(default 120s): wraps the async streaming iterator inasyncio.wait_forper chunk. Measures the gap between parsed SSE chunks, so keepalives don't reset it. Fires on genuine content silence and raisesStreamChunkTimeoutError— aTimeoutErrorsubclass carryingtimeout_s,model_name, andchunks_receivedas structured attributes (mirrored in the WARNING log'sextra=) for alerting without message-regex. Override with the kwarg orLANGCHAIN_OPENAI_STREAM_CHUNK_TIMEOUT_S.http_socket_options: appliesSO_KEEPALIVE+TCP_KEEPIDLE/TCP_KEEPINTVL/TCP_KEEPCNT+TCP_USER_TIMEOUTon Linux (macOS equivalents where available). On platforms missing some options, they're dropped silently and the remaining set still does useful work.Pool limits are set explicitly on the custom transport to mirror the
openaiSDK — without that, passingtransport=tohttpx.AsyncClientsilently shrinks the connection pool.Behavior change
The default-shape proxy-env bypass (above) covers the common enterprise case. Beyond that:
StreamChunkTimeoutError.http_socket_optionswhile also relying on env proxies will see a one-timeWARNINGand lose env-proxy auto-detection — the custom transport shadows it. This is the original shipped behavior, retained for anyone who wants socket tuning on top of an env-proxied setup.Full opt-outs:
stream_chunk_timeout=NoneorLANGCHAIN_OPENAI_STREAM_CHUNK_TIMEOUT_S=0http_socket_options=()orLANGCHAIN_OPENAI_TCP_KEEPALIVE=0http_clientandhttp_async_client.http_socket_optionsis applied per side: passing only one still leaves the other side's default builder getting socket options. Supply both (or combine withhttp_socket_options=()) to take full control.Unparseable or negative values for the
LANGCHAIN_OPENAI_*env vars fall back to the default with aWARNINGlog rather than silently being accepted, so a misconfigured environment still boots but the fallback is discoverable.