fix(cn): serve concurrent PRPC requests on one connection - #1739
Draft
aocsa wants to merge 1 commit into
Draft
Conversation
The FE multiplexes every RPC to a CN over one brpc connection, pairing responses with requests by correlation id, but the CN served a connection's requests strictly in line: read a frame, run the handler to completion, write the response, read the next. A fetch_data long-poll (or a blocking exec_plan_fragment) therefore held every request queued behind it -- including the cancel_plan_fragment the FE sends when the CN reports a fragment failure. MEASURED 2026-09-05 on TPC-H q18 at 2 CNs (arm W2b-2cn): the CN reported the failure at 02:46:07.569, the FE logged updateStatus -> cancelInternal at once, then eight RpcTimerTask timeouts on its one channel to that CN, and getNext only saw CANCELLED at 02:50:22 when the query timed out. The client waited 4 min 20 s for a failure the CN had reported in 2 ms. handle_connection now splits the socket, spawns one task per request (the Tower service is cloned per request, as it already was per connection), and writes responses in completion order under a writer lock. Per-sender frame order over the exchange is unaffected: a peer awaits each transmit_packed reply before sending the next frame of a drain. A test drives a slow and a fast request down one connection and reads the fast answer first.
aocsa
added
starrocks
rust
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Serve PRPC requests received on one TCP connection concurrently. The server now splits the socket, clones its Tower service per request, and serializes only response writes. Responses may therefore arrive in completion order while their correlation ids continue to pair them with requests.
This prevents a long
fetch_datapoll or blockingexec_plan_fragmentfrom holding a latercancel_plan_fragmentbehind it on the FE's multiplexed CN connection. The included regression test sends a slowfetch_dataand a fastcancel_plan_fragmenton one connection and proves that the fast response arrives first.Scope and prerequisites
This is one self-contained file and one preserved source commit. It does not require the peer PRPC client to compile or test. The motivating FE-to-CN exchange/cancellation path still needs the checked-in exchange RPC interface from #1707 (or its landed equivalent) and the later distributed CN runtime; this change is the server-side concurrency primitive that path uses.
Validation
pixi run cargo fmt --manifest-path experimental/starrocks/Cargo.toml --check5e71059c.git diff --check upstream/dev...HEADThe Cargo test suite was not run: this standalone clone intentionally does not initialize the StarRocks submodule, and the extraction does not warrant a GPU build or rebuild.