feat(anthropic): prevent silent streaming hangs in ChatAnthropic#36964
Closed
Mason Daugherty (mdrxy) wants to merge 2 commits intomasterfrom
Closed
feat(anthropic): prevent silent streaming hangs in ChatAnthropic#36964Mason Daugherty (mdrxy) wants to merge 2 commits intomasterfrom
ChatAnthropic#36964Mason Daugherty (mdrxy) wants to merge 2 commits intomasterfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of #36949 from
langchain-openaitolangchain-anthropic. StreamingChatAnthropiccalls could hang forever when the underlying TCP connection silently died mid-stream (idle NAT/LB timeouts, sandboxed runtimes killing long-lived connections, peer gone without a FIN or RST). httpx's read timeout doesn't help because it resets on any bytes arriving on the socket, including the Messages API'sevent: pingSSE heartbeats — a stream that's silent on content but still pinging looks alive forever. This adds two bounded-hang knobs with safe defaults.Changes
stream_chunk_timeouttoChatAnthropic(default 120s,None/0to disable, env overrideLANGCHAIN_ANTHROPIC_STREAM_CHUNK_TIMEOUT_S). Wraps the async SDK stream iterator inasyncio.wait_forper chunk. Measures the gap between parsed SSE chunks, so the anthropic SDK's ping-filter keeps heartbeats from resetting the timer. Syncstream()is untouched.StreamChunkTimeoutError— subclass ofasyncio.TimeoutErrorandTimeoutErroron both Python 3.10 and 3.11+ via a dynamic base-class tuple, so existingexcept TimeoutError:handlers still catch it. Carries structured attributestimeout_s,model_name,chunks_receivedmirrored in a WARNING log'sextra=for alerting without message-regex. Exported from the top-level package.http_socket_optionstoChatAnthropic. Defaults toSO_KEEPALIVE+TCP_KEEPIDLE/TCP_KEEPINTVL/TCP_KEEPCNT+TCP_USER_TIMEOUTon Linux (macOS equivalents where available, Windows falls back toSO_KEEPALIVEonly). Unsupported options are probed against a throwaway socket and silently dropped so the default set is non-fatal across platforms. Env overridesLANGCHAIN_ANTHROPIC_TCP_KEEPALIVE(kill-switch),LANGCHAIN_ANTHROPIC_TCP_KEEPIDLE,LANGCHAIN_ANTHROPIC_TCP_KEEPINTVL,LANGCHAIN_ANTHROPIC_TCP_KEEPCNT,LANGCHAIN_ANTHROPIC_TCP_USER_TIMEOUT_MS. Unparseable or negative env values fall back with a discoverable WARNING rather than silently applying a surprising default.httpxtransport disables httpx's env-proxy auto-detection. To avoid silently breaking users relying onHTTP_PROXY/HTTPS_PROXY/ALL_PROXY/ system proxies,ChatAnthropicdetects the "proxy-env-shadow" pattern and skips the custom transport entirely whenhttp_socket_optionsis at default (None), noanthropic_proxyis supplied, and a proxy env var / system proxy is visible to httpx. A one-timeINFOrecords the bypass. Users who explicitly sethttp_socket_options=[...]alongside an env proxy get the original shadowing behavior with a one-timeWARNING._resolved_socket_optionsproperty threads the resolved option tuple into both_clientand_async_clientso the bypass / shadow warnings fire once per instance, not once per client build. Whenanthropic_proxyand socket options are combined, proxy is wrapped inhttpx.Proxy(...)on the transport and the Client-levelproxy=key is popped to avoid httpx's double-configuration error.httpx.Limits(max_connections=1000, max_keepalive_connections=100, keepalive_expiry=5.0)) — matchesanthropic.DEFAULT_CONNECTION_LIMITS. Without this, passingtransport=tohttpx.AsyncClientsilently shrinks the connection pool.stream_chunk_timeout(e.g., hydrated from YAML/JSON configs) fall back to the default with a WARNING via a pydanticfield_validator, rather than silently treating negatives as an opt-out.Noneand0remain the documented off-switches.Behavior change
StreamChunkTimeoutErrorafter 120s of content silence. Existingexcept TimeoutError:/except asyncio.TimeoutError:handlers catch it unchanged.stream_chunk_timeout=None(or…_TIMEOUT_S=0),http_socket_options=()(or…_TCP_KEEPALIVE=0).