sdk/rs: retry JSON-RPC response errors that carry a transient status - #4273
Open
ayushsingh82 wants to merge 1 commit into
Open
sdk/rs: retry JSON-RPC response errors that carry a transient status#4273ayushsingh82 wants to merge 1 commit into
ayushsingh82 wants to merge 1 commit into
Conversation
is_retryable_rpc_error matched only Io/Reqwest/Middleware, so a transient
failure decoded into an RpcResponseError envelope was never retried: an
HTTP 429/5xx a provider LB put in the envelope code, a -32005/-32004/-32429
"busy, retry later" node code, or transient wording ("service unavailable")
with no machine-readable code. Rust survived the 2026-07-28 incident only
because those 503s happened to arrive as Reqwest errors; as decoded
envelopes — the shape Go received — every CLI read would have failed on
the first attempt.
Classification now mirrors the Go fix from malbeclabs#4100 (tools/solana/pkg/jsonrpc).
Onchain and request-level rejections (-32002 preflight, -32602, -32601),
-32003 signature-verification failure and -32011 history-not-available stay
non-retryable. No send_transaction path is wrapped with this predicate, so
an accepted-but-unacknowledged send is never resent.
Part of malbeclabs#4101.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #4100 (which fixed this for Go) into the Rust SDK behind the
doublezeroCLI. #4098's core finding — Go and Rust disagreed on what's retryable — was only half resolved: the Rust predicateis_retryable_rpc_error(smartcontract/sdk/rs/src/client.rs) matched onlyClientErrorKind::Io/Reqwest/Middlewareand never looked at a decoded JSON-RPC error envelope.So the two shapes the 2026-07-28 incident and
malbeclabs/infra#2100actually arrived in were still one-shot failures for every Rust CLI read:code(RPCPool's503— the Go JSON-RPC retry classification is broken; Go and Rust disagree on what's retryable #4098 shape)-32603 "Service unavailable, please try again later."from Helius — the infra#2100 shape)Rust rode out 07-28 only because those particular 503s surfaced as
Reqwesterrors. Had they arrived as decoded envelopes, as they did for Go, every CLI invocation would have failed on the first attempt.What changed
is_retryable_rpc_errornow also classifiesClientErrorKind::RpcError, mirroring the Go classifier intools/solana/pkg/jsonrpc:codethat is really an HTTP status (429/500/502/503/504)-32005,-32004,-32429service unavailable,too many requests,bad gateway,gateway timeout,rate limited) when no recognizable code is set-32002preflight,-32602,-32601, plus-32003(signature-verification failure) and-32011(no history) — deterministic, an identical retry gets the same answer-32003is listed as retryable in the issue text, but #4100 deliberately excludes it as deterministic; I followed #4100.No
send_transactionpath in this SDK is wrapped with this predicate — every retry site is a read (get_account,get_program_accounts,get_signatures_for_address, …) — so an accepted-but-unacknowledged send is never resent. The classifier keeps that property and its doc comment states it.Testing Verification
cargo test -p doublezero_sdk— the newclient_testscases assert: an envelope503/429retries;-32005/-32004/-32429retry; a-32603whose message says "Service unavailable" retries while a-32603"Internal error" does not;-32002/-32602/-32601/-32003/-32011do not retry; transport (Io) still retries.