[key-server] Apply the configured RPC timeout to the fullnode gRPC client#623
Merged
Conversation
RpcConfig.timeout (default 60s) was exposed to operators but never applied: build_grpc_client set no per-request timeout on the tonic client, so every fullnode RPC (package resolution, MVR lookup, alias check, policy simulation, zkLogin verification) could hang indefinitely, pinning a task/connection per inbound request until the process needs a restart. Thread the configured timeout into the client via with_response_headers_timeout for both the key server and the aggregator. Expiry surfaces as DeadlineExceeded, which the existing retry layer already handles. Adds a regression test using a hanging LedgerService stub.
tamashi095
requested review from
abhinavg6,
benr-ml,
jonas-lj and
joyqvq
as code owners
July 24, 2026 08:21
joyqvq
approved these changes
Jul 24, 2026
joyqvq
left a comment
Collaborator
There was a problem hiding this comment.
small edits, lgtm, thanks for the contribution
joyqvq
enabled auto-merge (squash)
July 24, 2026 13:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #624
[key-server] Apply the configured RPC timeout to the fullnode gRPC client
Summary
RpcConfig.timeout(default 60s) is exposed to operators inKeyServerOptionsand documented as "Timeout for individual RPC requests", but it was never
applied anywhere:
Server::new(server.rs) passed onlyrpc_config.retry_configintoSuiRpcClient::new, andbuild_grpc_client(sui_rpc_client.rs) constructed the tonic client withno per-request timeout.
As a result, every fullnode RPC the key server makes — package resolution,
MVR lookup, address-alias check, policy simulation, zkLogin signature
verification — could hang indefinitely.
This change threads the configured timeout into the gRPC client via
SuiGrpcClient::with_response_headers_timeout, for both the key server andthe aggregator. Expiry surfaces as
DeadlineExceeded, which the existingretry layer already handles, so behavior on a healthy fullnode is unchanged.
Why it matters
The key server performs several fullnode RPCs per
/v1/fetch_keyrequestbefore the certificate signature is verified, and has no in-process
concurrency limit. With a stalled or slow fullnode (or an exhausted fullnode
rate quota — which unauthenticated traffic itself can cause via these
pre-signature RPCs), every inbound request pins a task, a connection, and
memory for an unbounded time. Under modest request volume this grows
linearly until the process exhausts file descriptors or memory and needs a
restart.
With this fix, each RPC is bounded by
rpc_config.timeout(× retry attemptsfullnode incidents.
Reproduced locally with a stub fullnode that answers
GetEpochbut hangs allother RPCs: at 100 unauthenticated requests/second the stock server
accumulated ~6,000 in-flight requests / open FDs in 60s with linearly growing
RSS and never answered a single request; with the timeout applied (set to 2s
for the test) every request failed within ~7s and in-flight stayed flat.
Test plan
sui_rpc_client::tests::test_build_grpc_client_applies_timeout: spinsup a gRPC
LedgerServicethat never responds and asserts the call failswith
DeadlineExceededshortly after the configured timeout instead ofhanging forever.
options.rpc_config.timeoutinthe key server and aggregator; a fixed 30s in test helpers).
Notes
the certificate signature before issuing outbound RPCs, negative-cache MVR
resolution failures, and add a global in-flight request limit.