EVM: self-heal stalled/wedged HTTP/2 backend connections (h2 ping keepalive)

[cranycrane]

Summary

Blockbook's EVM RPC client (bchain/coins/eth) dials backends with go-ethereum's
default HTTP client, which uses http.DefaultTransport — no HTTP/2 health-checking
and no reconnect-on-failure. Over HTTPS, all JSON-RPC traffic to a backend is
multiplexed onto a single HTTP/2 connection. If that connection stalls or
silently dies (e.g. a load balancer between Blockbook and a 3rd-party backend drops
it while Go still considers it alive), every call multiplexed on it hangs until its
per-call context deadline (rpc_timeout, e.g. 25s), and the condition persists
until Blockbook is restarted.

We should add HTTP/2 connection self-healing (ping-based keepalive) so a stalled/dead
connection is detected and reaped automatically instead of requiring a restart.

Symptoms

• Sporadic context deadline exceeded on even the cheapest calls — e.g. GetChainInfo
  (net_version + web3_clientVersion) — despite the backend being healthy.
• Many RPC methods spike in latency simultaneously, consistent with a single shared
  connection rather than a per-method backend issue.
• Latency stays elevated until the Blockbook process is restarted, after which it
  recovers — the hallmark of a stuck pooled connection, not backend slowness.
• With a 3rd-party backend provider, the provider reported fast server-side fetch
  (<100 ms) but slow edge→client response delivery (tens of seconds) for large
  responses — i.e. the time is in the connection/delivery path, not backend compute.

Root cause

• dialRPC (bchain/coins/eth/ethrpc.go) calls rpc.DialOptions without
  WithHTTPClient, so go-ethereum falls back to new(http.Client) → http.DefaultTransport.
• DefaultTransport has ForceAttemptHTTP2: true; over an https:// backend that
  advertises h2 via ALPN, Go multiplexes all requests onto one TCP connection.
• There is no HTTP/2 keepalive/health-check and no reconnect-on-failure in this path.
  A wedged connection is reused indefinitely; only a process restart rebuilds the pool.

Proposed fix

Inject a customized transport for http(s):// backends via rpc.WithHTTPClient,
enabling HTTP/2 ping keepalive. On Go 1.24+ (this repo is on go 1.25.0) this is
available in the standard library with no new dependency:

t := http.DefaultTransport.(*http.Transport).Clone() // keeps ForceAttemptHTTP2 → h2 preserved
t.HTTP2 = &http.HTTP2Config{
    SendPingTimeout: 15 * time.Second, // ping a silent connection after 15s idle
    PingTimeout:     10 * time.Second, // close it if the ping is unanswered (well under rpc_timeout)
}
t.IdleConnTimeout = 30 * time.Second   // shrink stale-idle window (default 90s); applies to h1.1 too
// dialRPC: only for http/https URLs; leave ws:// dialing untouched
opts = append(opts, rpc.WithHTTPClient(&http.Client{Transport: t}))

• HTTP/2 multiplexing is preserved (important for high request volume).
• ws:// / wss:// dialing is unaffected (guarded by URL scheme; WithHTTPClient
  only affects the HTTP codec).
• Expose SendPingTimeout / PingTimeout as coin-config knobs with the above defaults
  (set SendPingTimeout: 0 to disable).

Scope / limitations

• Fixes: dead/wedged HTTP/2 connections that currently require a restart.
• Does NOT fix: load-driven contention where a large response (e.g.
  debug_traceBlockByHash used for internal-transaction tracing) saturates the shared
  connection and starves other calls — the connection is alive and still answers pings,
  so it won't be reaped. That needs a separate change (a dedicated HTTP/2 client for
  heavy sync/trace traffic so it can't starve latency-sensitive calls). Suggested as a
  follow-up.
• No impact on plain-http:// self-hosted backends: those run HTTP/1.1 (Go does not
  do h2c automatically), where HTTP2Config is ignored — and they don't share the
  single-connection failure mode anyway. IdleConnTimeout is the only setting that also
  applies to HTTP/1.1 pools (mildly beneficial). No behavior regression for them.

Acceptance criteria

• EVM http(s):// backends use an HTTP/2-ping-keepalive transport; a killed/stalled
  backend connection is detected and re-dialed without a Blockbook restart.
• ws:// / wss:// and plain-http:// behavior unchanged.
• Ping timeouts configurable per coin, disabled when set to 0.