Skip to content

sdk/rs: retry JSON-RPC response errors that carry a transient status - #4273

Open
ayushsingh82 wants to merge 1 commit into
malbeclabs:mainfrom
ayushsingh82:fix/4101-rs-retry-classification
Open

sdk/rs: retry JSON-RPC response errors that carry a transient status#4273
ayushsingh82 wants to merge 1 commit into
malbeclabs:mainfrom
ayushsingh82:fix/4101-rs-retry-classification

Conversation

@ayushsingh82

Copy link
Copy Markdown
Contributor

Follows #4100 (which fixed this for Go) into the Rust SDK behind the doublezero CLI. #4098's core finding — Go and Rust disagreed on what's retryable — was only half resolved: the Rust predicate is_retryable_rpc_error (smartcontract/sdk/rs/src/client.rs) matched only ClientErrorKind::Io / Reqwest / Middleware and never looked at a decoded JSON-RPC error envelope.

So the two shapes the 2026-07-28 incident and malbeclabs/infra#2100 actually arrived in were still one-shot failures for every Rust CLI read:

Rust rode out 07-28 only because those particular 503s surfaced as Reqwest errors. Had they arrived as decoded envelopes, as they did for Go, every CLI invocation would have failed on the first attempt.

What changed

is_retryable_rpc_error now also classifies ClientErrorKind::RpcError, mirroring the Go classifier in tools/solana/pkg/jsonrpc:

  • retry an envelope code that is really an HTTP status (429/500/502/503/504)
  • retry the "busy / retry later" node codes -32005, -32004, -32429
  • retry transient message wording (service unavailable, too many requests, bad gateway, gateway timeout, rate limited) when no recognizable code is set
  • keep onchain / request-level rejections non-retryable: -32002 preflight, -32602, -32601, plus -32003 (signature-verification failure) and -32011 (no history) — deterministic, an identical retry gets the same answer

-32003 is listed as retryable in the issue text, but #4100 deliberately excludes it as deterministic; I followed #4100.

No send_transaction path in this SDK is wrapped with this predicate — every retry site is a read (get_account, get_program_accounts, get_signatures_for_address, …) — so an accepted-but-unacknowledged send is never resent. The classifier keeps that property and its doc comment states it.

Testing Verification

  • cargo test -p doublezero_sdk — the new client_tests cases assert: an envelope 503/429 retries; -32005/-32004/-32429 retry; a -32603 whose message says "Service unavailable" retries while a -32603 "Internal error" does not; -32002/-32602/-32601/-32003/-32011 do not retry; transport (Io) still retries.

is_retryable_rpc_error matched only Io/Reqwest/Middleware, so a transient
failure decoded into an RpcResponseError envelope was never retried: an
HTTP 429/5xx a provider LB put in the envelope code, a -32005/-32004/-32429
"busy, retry later" node code, or transient wording ("service unavailable")
with no machine-readable code. Rust survived the 2026-07-28 incident only
because those 503s happened to arrive as Reqwest errors; as decoded
envelopes — the shape Go received — every CLI read would have failed on
the first attempt.

Classification now mirrors the Go fix from malbeclabs#4100 (tools/solana/pkg/jsonrpc).
Onchain and request-level rejections (-32002 preflight, -32602, -32601),
-32003 signature-verification failure and -32011 history-not-available stay
non-retryable. No send_transaction path is wrapped with this predicate, so
an accepted-but-unacknowledged send is never resent.

Part of malbeclabs#4101.
@ayushsingh82
ayushsingh82 requested a review from a team September 4, 2026 13:11
@ayushsingh82

ayushsingh82 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

@ben-dz @nikw9944 — addresses #4101, the Rust-SDK parity half that #4098 flagged as still open. Classification here mirrors your merged Go fix in #4100 (tools/solana/pkg/jsonrpc) — tagging you two since you authored/reviewed that one and filed #4101.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant