sdk/rs: retry JSON-RPC response errors that carry a transient status

CHANGELOG.md

@@ -11,6 +11,7 @@ All notable changes to this project will be documented in this file.
 
 - SDK
   - The TypeScript and Python `GlobalState` deserializers expose `ip_verifier_authority_pk`, the RFC-27 trust root the Go SDK and the Rust state already carried, so those consumers can read which key signs IP ownership proofs. The field is appended, so an account written before the upgrade decodes it as the default pubkey rather than failing. (#4231)
+  - The Rust SDK (`smartcontract/sdk/rs`) behind the `doublezero` CLI now retries a transient failure carried inside a decoded JSON-RPC error envelope, not just one that surfaced at transport level. `is_retryable_rpc_error` previously matched only `Io`/`Reqwest`/`Middleware`, so an `RpcResponseError` holding an HTTP 429/5xx that a provider LB put in the envelope `code`, a `-32005`/`-32004`/`-32429` "busy, retry later" node code, or transient wording like "service unavailable" with no machine-readable code (the Helius `-32603` shape) was a one-shot failure for every CLI read. Rust rode out the 2026-07-28 incident only because those 503s happened to arrive as `Reqwest` errors; had they arrived as decoded envelopes — as they did for Go — every CLI invocation would have failed on the first attempt. Classification now mirrors the Go fix from #4100 (`tools/solana/pkg/jsonrpc`). Onchain and request-level rejections (`-32002` preflight, `-32602`, `-32601`), `-32003` signature-verification failure and `-32011` history-not-available stay non-retryable, and no `send_transaction` path is wrapped with this predicate, so an accepted-but-unacknowledged send is never resent. (#4101)
 - CI
   - The e2e matrix runs 5 round-robin shards instead of 4. Shards are filled by test count, not by duration, so the heaviest one was carrying ~3735s of tests against a 15-minute job timeout and was cancelled mid-run; the extra shard brings the worst case back to ~3022s. Adding `e2e (shard 6)` to the required status checks in the main ruleset is a separate, manual step.
   - shreds-e2e pins one heavy test to its own shard instead of three. `TestE2E_MultiUserInstantAllocationAndWithdrawal` and `TestE2E_DeviceScale` no longer exist in doublezero-shreds, so the pin validation failed every run and the matrix was never built. Only `TestE2E_FeedSubscriptionOracleExpiryTeardown` stays pinned, leaving 1 pinned + 2 round-robin shards. Dropping `shard-e2e (shard 4)` and `shard-e2e (shard 5)` from the required status checks in the main ruleset is a separate, manual step — until it happens those contexts are required but never reported.

smartcontract/sdk/rs/src/client.rs

@@ -305,13 +305,84 @@ impl DZClient {
         }
     }
 
-    /// Returns true for transient network errors that are worth retrying.
-    /// Returns false for permanent errors like AccountNotFound or RPC response errors.
+    /// Returns `true` for transient failures worth another attempt, `false` for a
+    /// permanent one (`AccountNotFound`, a bad request, a deterministic program
+    /// rejection).
+    ///
+    /// Transport errors (`Io` / `Reqwest` / `Middleware`) were already covered.
+    /// The gap this closes — the shape the 2026-07-28 incident and
+    /// `malbeclabs/infra#2100` arrived in for Go, and which Rust survived only
+    /// because those particular 503s happened to surface as `Reqwest` — is a
+    /// transient status carried *inside* a decoded JSON-RPC envelope
+    /// (`RpcResponseError`), which was never retried:
+    ///
+    /// - an HTTP status a provider LB put in the envelope `code` (429/5xx),
+    /// - a "busy, retry later" node code (`-32005`, `-32004`, `-32429`),
+    /// - transient wording with no machine-readable code
+    ///   (`-32603 "Service unavailable, please try again later."`).
+    ///
+    /// Mirrors the Go classifier from #4100 (`tools/solana/pkg/jsonrpc`).
+    /// Onchain/request-level rejections (`-32002` preflight, `-32602`, `-32601`),
+    /// `-32003` (signature-verification failure) and `-32011` (no history) stay
+    /// non-retryable — an identical later request gets the same answer. This
+    /// predicate gates only read RPCs here; no `send_transaction` path is wrapped
+    /// with it, so an accepted-but-unacknowledged send is never resent.
     fn is_retryable_rpc_error(err: &ClientError) -> bool {
-        matches!(
-            err.kind.as_ref(),
-            ClientErrorKind::Io(_) | ClientErrorKind::Reqwest(_) | ClientErrorKind::Middleware(_)
-        )
+        match err.kind.as_ref() {
+            ClientErrorKind::Io(_)
+            | ClientErrorKind::Reqwest(_)
+            | ClientErrorKind::Middleware(_) => true,
+            ClientErrorKind::RpcError(rpc_err) => Self::is_retryable_rpc_response(rpc_err),
+            _ => false,
+        }
+    }
+
+    /// HTTP statuses that mean the endpoint (or a proxy in front of it) is
+    /// shedding load, not that the request is wrong.
+    fn is_retryable_http_status(code: i64) -> bool {
+        matches!(code, 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// JSON-RPC codes for a node that cannot serve this request right now but
+    /// could serve an identical one later: `-32005` NODE_UNHEALTHY, `-32004`
+    /// BLOCK_NOT_AVAILABLE, and the provider-minted `-32429` (HTTP 429 wrapped in
+    /// an envelope). `-32003` (signature verification) and `-32011` (no history)
+    /// are deterministic and deliberately absent.
+    fn is_retryable_rpc_code(code: i64) -> bool {
+        matches!(code, -32005 | -32004 | -32429)
+    }
+
+    /// Transient wording from providers that return no machine-readable code — a
+    /// load balancer shedding load is the same failure whether it labels itself
+    /// 503 or just says so.
+    fn message_is_transient(message: &str) -> bool {
+        let msg = message.to_ascii_lowercase();
+        [
+            "service unavailable",
+            "too many requests",
+            "bad gateway",
+            "gateway timeout",
+            "gateway time-out",
+            "rate limited",
+        ]
+        .iter()
+        .any(|needle| msg.contains(needle))
+    }
+
+    fn is_retryable_rpc_response(err: &solana_rpc_client_api::request::RpcError) -> bool {
+        use solana_rpc_client_api::request::RpcError;
+        match err {
+            RpcError::RpcResponseError { code, message, .. } => {
+                Self::is_retryable_http_status(*code)
+                    || Self::is_retryable_rpc_code(*code)
+                    || Self::message_is_transient(message)
+            }
+            // No code to classify; only retry on explicitly transient wording.
+            RpcError::RpcRequestError(msg) | RpcError::ForUser(msg) => {
+                Self::message_is_transient(msg)
+            }
+            RpcError::ParseError(_) => false,
+        }
     }
 
     pub fn get_balance(&self) -> eyre::Result<u64> {
@@ -775,6 +846,66 @@ mod client_tests {
         let transport = ClientError::from(ClientErrorKind::Custom("connection reset".into()));
         assert_eq!(DZClient::parse_transaction_error(&transport), None);
     }
+
+    fn rpc_response_error(code: i64, message: &str) -> ClientError {
+        ClientError::from(ClientErrorKind::RpcError(
+            solana_rpc_client_api::request::RpcError::RpcResponseError {
+                code,
+                message: message.to_string(),
+                data: solana_rpc_client_api::request::RpcResponseErrorData::Empty,
+            },
+        ))
+    }
+
+    /// Transport errors stay retryable.
+    #[test]
+    fn is_retryable_rpc_error_covers_transport() {
+        let io = ClientError::from(ClientErrorKind::Io(std::io::Error::other("reset")));
+        assert!(DZClient::is_retryable_rpc_error(&io));
+    }
+
+    /// A transient status carried inside a decoded JSON-RPC envelope is now
+    /// retried — the shape that reached Go as `*RPCError` on 2026-07-28.
+    #[test]
+    fn is_retryable_rpc_error_covers_envelope_status_and_busy_codes() {
+        // HTTP status a provider LB put in the envelope code.
+        assert!(DZClient::is_retryable_rpc_error(&rpc_response_error(
+            503,
+            "backend unavailable"
+        )));
+        assert!(DZClient::is_retryable_rpc_error(&rpc_response_error(
+            429,
+            "slow down"
+        )));
+        // "busy, retry later" node codes.
+        for code in [-32005, -32004, -32429] {
+            assert!(
+                DZClient::is_retryable_rpc_error(&rpc_response_error(code, "node is behind")),
+                "code {code} should retry"
+            );
+        }
+        // Transient wording with no recognizable code (the Helius `-32603` shape).
+        assert!(DZClient::is_retryable_rpc_error(&rpc_response_error(
+            -32603,
+            "Service unavailable, please try again later."
+        )));
+    }
+
+    /// Deterministic rejections and request-level errors do not retry.
+    #[test]
+    fn is_retryable_rpc_error_excludes_permanent_rpc_errors() {
+        for code in [-32002, -32602, -32601, -32003, -32011] {
+            assert!(
+                !DZClient::is_retryable_rpc_error(&rpc_response_error(code, "rejected")),
+                "code {code} must not retry"
+            );
+        }
+        // A generic -32603 with no transient wording is not retryable either.
+        assert!(!DZClient::is_retryable_rpc_error(&rpc_response_error(
+            -32603,
+            "Internal error"
+        )));
+    }
 }
 
 #[cfg(all(test, feature = "cli-context"))]