EVM: block sync permanently halts on persistent backend RPC failure (unbounded GetBlockHash retry, no sync-side reconnect)

[cranycrane]

Summary

On EVM chains, a persistent backend RPC failure can permanently halt block sync until
Blockbook is manually restarted. When Blockbook falls behind (≥ the parallel-sync
threshold) and enters ParallelConnectBlocks, the coordinator loop retries
GetBlockHash(h) for the same height forever — no attempt bound, no reconnect — if
the call keeps failing (e.g. the HTTP call connection is wedged, or the backend keeps
returning errors/timeouts). The loop never returns, so resyncIndex → ResyncIndex →
the TickAndDebounce sync driver never returns either, and no further sync happens.

Crucially, the existing tipWatchdog does not catch this: it monitors feed
liveness (the newHeads subscription), not sync-worker progress. The newHeads feed
runs over a separate WebSocket connection and keeps advancing the cached tip, so
markSubscriptionAlive() keeps the watchdog satisfied and it never fires a stall — while
the sync worker is frozen. Only a manual restart (which rebuilds the RPC client) recovers.

Symptoms

• Block sync stops: BlockbookBestHeight stays flat while BackendBestHeight keeps
  climbing (the chain moved on; the DB did not).
• No watchdog_stall events in blockbook_backend_subscription_events during the
  stall (the feed/WS side is healthy; watchdog_tip_advanced continues normally).
• watchdog_tick keeps incrementing (the watchdog goroutine is alive — it just has
  nothing to watch here).
• Logs show GetBlockHash error … repeating roughly every rpc_timeout seconds.
• Correlates with a period where some RPC methods were intermittently timing out
  (context deadline exceeded) against a 3rd-party backend. Recovers only on restart.

Root cause

1. Unbounded retry with no reconnect — db/sync.go, ParallelConnectBlocks
   coordinator loop:

   default:
       hash, err = w.chain.GetBlockHash(h)
       if err != nil {
           glog.Error("GetBlockHash error ", err)
           w.metrics.IndexResyncErrors....Inc()
           time.Sleep(time.Millisecond * 500)
           continue            // retries the SAME height forever; h never increments
       }
       ...
       h++

   The loop only breaks on abortCh (a worker error) or an OS signal. A persistent
   GetBlockHash failure — a wedged HTTP/2 connection, a rate-limit 429, a backend
   5xx, a partial outage — is none of those, so it spins indefinitely. Because it never
   returns, the whole sync driver (ResyncIndex under common.TickAndDebounce) is wedged.

2. No sync-side reconnect — only the feed tipWatchdog calls reconnectRPC() to
   rebuild a bad connection. The sync path has no equivalent, so a wedged call
   connection is never replaced while sync is stuck on it.

3. Watchdog blind spot — tipWatchdog keys liveness on lastSubNotifyNs, refreshed
   by onFeedHeader on every tip advance (bchain/coins/eth/ethrpc.go). Tip advance is
   independent of sync-worker progress, so a frozen sync worker leaves the watchdog
   happy and no stall/reconnect is triggered.

Relationship to #1696

#1696 (HTTP/2 ping keepalive / connection self-heal) only helps the subset of this where
the call connection is fully dead (no frames → ping unanswered → reap → the retry
re-dials). It does not fix:

• a grey/contended connection (still returning some frames → ping stays answered →
  never reaped) while GetBlockHash keeps timing out, or
• non-connection persistent failures (429, 5xx, auth), which no ping-pong resolves.

So this is a distinct and higher-severity bug: the latency in #1696 was survivable; this
permanent halt was not.

Proposed fix

• Bound the retry in ParallelConnectBlocks (and audit the sequential
  connectBlocks path): after N consecutive failures at a height, return an error so
  ResyncIndex unwinds and retries fresh on the next trigger instead of trapping the loop.
• Reconnect on repeated sync failures: after N consecutive GetBlockHash/GetBlock
  failures, force a call-client reconnect (CloseRPC + re-dial) so a wedged connection
  self-heals without a manual restart.
• Add a sync-progress watchdog: if BackendBestHeight > BlockbookBestHeight with no
  ConnectBlock progress for N minutes, abort in-flight sync RPCs (CloseRPC) and
  reconnect. This closes the "watchdog only watches the feed" gap.

How to confirm (from an affected instance)

• Logs: GetBlockHash error … repeating ~every rpc_timeout seconds during the stall.
• Metrics: BackendBestHeight climbing while BlockbookBestHeight is flat;
  watchdog_tick still incrementing; no watchdog_stall.
• Goroutine dump (SIGQUIT / /debug/pprof/goroutine?debug=2) before restart: the sync
  goroutine parked in GetBlockHash → HeaderByNumber.

Acceptance criteria

• A persistent backend failure at a height causes ParallelConnectBlocks to return an
  error in bounded time (not spin forever); ResyncIndex retries on the next trigger.
• After sustained sync failure, the call client is reconnected automatically; sync
  resumes without a manual restart once the backend recovers.
• A frozen sync worker (backend ahead of DB, no ConnectBlock progress) is detected and
  recovered independently of newHeads feed liveness.