fix(api): indexer stalls and falls behind, restart can deadlock

[tony8713]

Symptom

The Starknet indexer (apps/api, built on @snapshot-labs/checkpoint) periodically halts and falls thousands of blocks behind chain tip (recently observed ~17k blocks behind). When it stalls, downstream data goes stale: voting numbers on the interface stop updating until the indexer is manually recovered.

This has recurred (it has happened before on a network and was flagged to investigate), so it is not a one-off.

Constraint: cannot be naively auto-restarted

We currently restart it manually. It cannot be safely auto-restarted because the indexing path does not run inside DB transactions, so a restart can leave it in an inconsistent state and end up in a deadlock. Quote from the discussion: "we dont have transactions so when we restart there is a chance we end up in deadlock."

Goal

1. Investigate the root cause of the stalls (why the indexer head stops advancing).
2. Make restart safe and idempotent so recovery (manual or automated) cannot deadlock. Options to consider:
   • Transactional checkpointing (wrap block-write + checkpoint cursor in a single DB transaction so a restart resumes cleanly).
   • A documented safe-restart procedure / lock handling if full transactions are not feasible short term.

Infra context

DigitalOcean hosts; Cloudflare domains (api-arb1 / api-indexer-2.snapshot.box, some old/killed); Checkpoint console at console.checkpoint.box.

Notes

Filed in sx-monorepo since apps/api is the deployed indexer service that consumes Checkpoint. If the fix turns out to be in the framework itself, the related repo is snapshot-labs/checkpoint.

Related: monitoring/alerting tracked in #2163

To handle next week.

[tony8713]

Root-cause investigation (apps/api + @snapshot-labs/checkpoint)

I traced the indexer loop end to end. Summary: the indexing loop itself never exits on error (it catches + retries), so the stall is not the loop dying. The realistic stall and deadlock causes are below, with file/function references. The substantive fix lives in the @snapshot-labs/checkpoint framework, not apps/api, so I have not shipped a speculative app-side fix for this issue (the lag/alerting half is PR #2164, which at least makes the stall visible).

Versions inspected: @snapshot-labs/checkpoint@0.1.0-beta.70, apps/api Starknet indexer.

How the loop works (so we know what can and cannot stall it)

Container.process() (checkpoint/dist/src/container.js) is a while (true) loop. Every iteration calls provider.processBlock(). The whole body is wrapped in try/catch: on any error it logs, re-queues the block, sleep(fetch_interval) and continues. So a single throwing block handler will not kill indexing - it will retry that same block forever. That matters: a block that deterministically throws becomes an infinite stall on one block (head stops advancing), not a crash.

last_indexed_block is written in StarknetProvider.processBlock() (providers/starknet/provider.js:80) only after handleBlock() fully succeeds. So the persisted cursor only advances on a clean block.

Stall cause 1 (most likely): infinite retry inside getEventsRangeForAddress

providers/starknet/provider.js:290-311. The pagination loop is:

do { try { getEvents(...) } catch (e) { log.error(...); await sleep(5000); } } while (continuationToken);

On a thrown error, continuationToken is never updated/reset and the loop just sleeps 5s and retries forever. If the RPC persistently errors on a given getEvents page (rate limit, a block/range it chokes on, a bad continuation token), this never returns - the outer block loop is blocked inside one call. Head freezes, logs show a repeating getEvents failed. This is consistent with "stalls and falls thousands of blocks behind" while the process stays up. Fix belongs in checkpoint: cap retries / surface as a throw so the outer loop's normal retry+requeue handles it, and reset the continuation token on retry.

Stall cause 2: a deterministically-throwing handler pins one block

Any writer that throws outside its local try/catch will make process() retry the same block indefinitely. In apps/api/src/starknet/writers.ts, most network calls are guarded, but several BigInt(...) / validateAndParseAddress(...) / array-shape assumptions on event.* are not. A malformed event would throw on every retry of that block = permanent stall at that height. Lower probability than cause 1 but same symptom. Mitigation: in checkpoint, after N failed retries of the same block, either skip-with-loud-error or escalate; in apps/api, defensively guard the parse paths.

Deadlock on restart ("we don't have transactions")

The writes for a single block are not in one DB transaction. handleTx -> writer functions do many independent entity.save() / updateCounter() calls, then processBlock separately does setBlockHash and setLastIndexedBlock. On restart, getStartBlockNum() (container.js:366) resumes at last_indexed_block + 1, but partial writes from the interrupted block are already committed. If a restart happens while two processes briefly overlap (note startIndexer even has a 60s PRODUCTION_INDEXER_DELAY sleep specifically "to prevent multiple processes indexing at the same time" - apps/api/src/indexer.ts:12,70-75), both can issue conflicting row updates on the same entities/counters with no serializing transaction, which is exactly the lock-contention/deadlock scenario described. handleReorg is the only path that uses knex.transaction(...) (container.js:306); the normal write path is not transactional, and removeFutureData even has a TODO: when we have full transaction support comment.

Safe fix (framework-side, the real one): wrap per-block writes + the last_indexed_block/block_hash cursor update in a single knex.transaction, so a block is all-or-nothing and a restart always resumes from a consistent boundary. Combined with a strict single-writer guarantee (a DB advisory lock keyed on indexer name, taken at start()), restart cannot deadlock because a second writer cannot run. Both changes are in @snapshot-labs/checkpoint (Container/CheckpointsStore), so they belong in that repo, not this PR.

Interim safe-restart procedure (no code change): ensure only one indexer process per network at a time (stop old before starting new; do not rely solely on the 60s delay), and restart during low write activity. The advisory-lock change above makes this automatic.

What I shipped vs left as follow-up

• Shipped (PR feat(api): add indexer sync-lag health endpoint #2164, this repo): /health lag endpoint so stalls become visible/alertable (issue feat(api): add monitoring and alerts for indexer sync lag #2163).
• Left as documented follow-up (snapshot-labs/checkpoint): (a) bound the getEventsRangeForAddress retry loop + reset continuation token; (b) per-block transactional writes incl. cursor; (c) DB advisory single-writer lock; (d) skip-after-N-retries so one poison block can't pin the head forever.

Confidence: high that the loop does not die and that non-transactional writes + overlapping writers is the deadlock surface; medium-high that cause 1 (infinite getEvents retry) is the specific stall, since it is the one unbounded wait on the hot path. Happy to open the checkpoint-side PR if the team wants me to.

[tony8713]

Log evidence (2026-06-09 stall)

Source: sx-api-arb1 Betterstack logs.

Timeline. Both the sn (Starknet) and mnt (Mantle) indexers froze at the SAME instant, 06:51 UTC (last sn block 10625919), and stayed frozen ~14h until a manual restart at 20:53 UTC. The other 6 indexers (arb1/bnb/ape/oeth/base/eth) kept logging normally the whole window, so the service appeared up. Gap ~19k blocks.

Nature of the failure. This is a SILENT per-indexer hang, NOT a process crash (the process stayed alive). The loop stopped emitting inside Checkpoint Container.process -> StarknetProvider.processPool, while getBlockWithReceipts repeatedly returned "Invalid params" / "block not found" chasing the head. Stack:

getBlockWithReceipts (providers/starknet/provider.js:252)
  <- processPool        (provider.js:84)
  <- Container.process  (container.js:230)

Significance. sn + mnt (the two slow / special-RPC chains) froze TOGETHER -> points to a shared Checkpoint container/scheduler stall, not a Starknet-specific bad block. This confirms the unbounded-retry / await-that-never-resolves theory; the writer-throw theory is unlikely (no crash, no fatal log).

Alerting note (for #2163). The failure is SILENT, so the right signal is the ABSENCE of per-indexer progress (no "handling block" line for a chain for >10-15 min), NOT the "Invalid params" error string (which fires constantly during normal head-chase and is benign). Global log volume stayed high, masking the per-chain stall.

[tony8713]

2026-06-11 double-box freeze: fresh log evidence + EVM-side root cause

Re-investigated today's incident where BOTH indexer boxes stalled (authorized follow-up). New evidence below; it confirms the silent-per-indexer-hang theory and adds the specific code path on the EVM side (not just Starknet).

Sources: sx-api-indexer-1 / box "arb1" (BetterStack source 2363475, table t60058.sx_api_arb1) and sx-api-indexer-2 / box "droplet-2" (source 2392882, table t60058.sx_api_droplet_2).

Symptoms (from logs, exact timeline)

Box arb1: sn, base, ape all stopped emitting within ~1.5s of each other:

• base: last normal line handling block 47184785 at 06:21:58.978, then silence
• ape: last handling block 39534417 at 06:21:58.808, then silence
• sn: last handling block 10689668 at 06:21:56.951, then silence
• The other 5 indexers on the SAME process (arb1/bnb/eth/mnt/oeth) kept logging continuously through 06:21, 06:22, 06:23 ... up to the manual restart ~12:22. Verified minute-by-minute: no gap.

Box droplet-2: only sn froze, last handling block 10690242 at 06:47:13.016, then silence. All 7 EVM indexers kept logging fine through restart.

Two facts matter:

1. The frozen loops' last line is a NORMAL handling block, not an error. There is no getLogs failed / getEvents failed spam preceding the freeze. The loop did not enter the visible 5s-retry loop; it just stopped after a healthy line. (The Invalid params / block not found on sn is the benign optimistic head-chase noise on pre_confirmed - it ran continuously before AND is not the cause.)
2. Siblings in the same Node process kept running. So this is NOT a process-wide event-loop block and NOT a crash. It is independent per-indexer loops each hanging on an awaited call that never resolves.

The 3-chain simultaneity on the arb1 box (sn+base+ape within 1.5s) points to a SHARED upstream: all EVM indexers use a single gateway https://rpc.snapshot.org/<chainId> (apps/api/src/evm/config.ts:57), and Starknet uses its own node URL. A gateway hiccup at 06:21:58 (accept connection, stop responding) would hang exactly the loops that had an in-flight request at that instant.

What the code shows (checkpoint @0.1.0-beta.70, as installed)

The hang is on the block-fetch / events-fetch path, and the precise vector is a fetch with no timeout:

• EVM _getLogs (providers/evm/provider.js:242-282): the abort signal is set ONLY when the filter has blockHash:

  if ('blockHash' in filter) { signal = AbortSignal.timeout(CLIENT_TIMEOUT); ... }

  For the fromBlock/toBlock RANGE path - which is what getLogs() (provider.js:283) and getLogsForSources use - signal stays undefined, so the underlying fetch has NO timeout. A socket that connects but never returns a body hangs that await fetch forever. handleBlock logs handling block (provider.js:90) and then calls into the source-logs path; a hang there leaves exactly the observed signature: one handling block line, then nothing.
• Starknet getBlockWithReceipts (providers/starknet/provider.js:234-256): bare fetch(...) with NO signal at all -> same unbounded hang on a silent socket. This is the sn freeze vector on both boxes.
• The outer getLogs loop (provider.js:283-317) is while(true) { try {...} catch { log; sleep(5000) } } - it only re-loops on a thrown error. A hung fetch never throws, so it never reaches the catch and never logs. (This is the EVM twin of the Starknet getEventsRangeForAddress infinite-retry already noted in this issue.)

Container.process() (container.js:181-264) is the single while(true) per indexer; the whole body awaits provider.processBlock(). One indexer's hung awaited fetch stalls only that indexer's loop. Other indexers are separate process() instances, which is why they kept running.

Hypothesis (confidence: high on mechanism, medium on the exact trigger)

A transient upstream RPC event around 06:21:58 (arb1 box) and 06:47:13 (droplet-2) caused in-flight requests to hang rather than error. Because the block/log fetch path has no request timeout on the range/Starknet code paths, the affected per-indexer loops blocked forever on await fetch, emitting no further logs, while the process and all other indexers stayed healthy. The 3-chain co-freeze on the arb1 box is consistent with the shared rpc.snapshot.org gateway. I cannot prove the upstream event from our logs alone (the failure is silent by construction); gateway-side logs at those timestamps would confirm.

Suggested fix (framework-side, snapshot-labs/checkpoint)

1. Put a timeout on EVERY provider request. In _getLogs, set signal = AbortSignal.timeout(CLIENT_TIMEOUT) for the range path too (currently only the blockHash path has it). Add a signal: AbortSignal.timeout(CLIENT_TIMEOUT) to the Starknet getBlockWithReceipts fetch. This alone converts every hang into a throw, which the existing retry/requeue handles - so a frozen chain self-recovers instead of going dark until manual restart.
2. Bound the inner getLogs / getEventsRangeForAddress retry loops (cap retries, then throw to the outer loop) so a persistently-erroring page also surfaces instead of spinning forever.
3. Per-indexer watchdog: if an indexer emits no progress for > N minutes, log loudly (and optionally restart that indexer). The right alert signal remains ABSENCE of per-chain handling block progress, not any error string (as noted for feat(api): add monitoring and alerts for indexer sync lag #2163).
4. unhandledRejection logging: apps/api/src/index.ts only handles uncaughtException. Add a process.on('unhandledRejection', ...) logger so any swallowed rejection on these paths becomes visible.

Items 1-2 are the load-bearing fix: no provider call should be able to await forever.

[Metavcl]

For indexers, I’d separate two problems: safe checkpointing and RPC head-lag detection.

If block writes and the checkpoint cursor are not committed atomically, restart can easily resume from a partially indexed state.

I’d suggest:

• wrap block processing + checkpoint update in one DB transaction
• track indexer head vs RPC chain tip
• alert when block lag keeps increasing
• use a secondary RPC to confirm whether the stall is indexer-side or provider-side

For Starknet indexers, a stable private RPC endpoint also helps reduce false stalls caused by provider latency or temporary RPC lag. BlockPI could be useful here as a Starknet private/fallback RPC for head checks and recovery validation.