Subsquid Indexer on Polygon Halts with "already indexed block was not found on chain" Error (Detailed Logs Included) #406
Description
I'm running a Subsquid indexer for the Polygon network, and it has unexpectedly stopped. I've conducted some initial investigations but am still puzzled by the root cause and would appreciate any insights the community could offer.
Logs and Context
Below are the logs from the moment the indexer started until it crashed. The logs show the service initialization, database migration check, the startup of multiple processors for different block ranges, and finally, the fatal error.
{"level":2,"time":1751859175495,"ns":"sqd:commands","msg":"MIGRATION:APPLY"}
query: SELECT version()
...
query: SELECT * FROM "migrations" "migrations" ORDER BY "id" DESC
No migrations are pending
+ node -r dotenv/config lib/main.js
done
// Multiple processors begin processing in parallel
{"level":2,"time":1751859176346,"ns":"sqd:processor","msg":"processing blocks from 7392037"}
{"level":2,"time":1751859176346,"ns":"sqd:processor","msg":"using chain RPC data source"}
{"level":2,"time":1751859176349,"ns":"sqd:processor","msg":"processing blocks from 354844716"}
{"level":2,"time":1751859176349,"ns":"sqd:processor","msg":"processing blocks from 73713068"}
// ... other processor logs ...
// Logs from the processor directly related to the error
{"level":2,"time":1751859176350,"ns":"sqd:processor","msg":"last processed final block was 73423490"}
{"level":2,"time":1751859176350,"ns":"sqd:processor","msg":"processing blocks from 73423491"}
// ... other processor logs and data source switching ...
{"level":2,"time":1751859176487,"ns":"sqd:processor","msg":"using archive data source"}
{"level":2,"time":1751859176489,"ns":"sqd:processor","msg":"using archive data source"}
{"level":2,"time":1751859176668,"ns":"sqd:processor","msg":"prometheus metrics are served at port 40735"}
{"level":2,"time":1751859176671,"ns":"sqd:processor","msg":"prometheus metrics are served at port 34903"}
// The final fatal error
{"level":5,"time":1751859176689,"ns":"sqd:processor","err":{"stack":"Error: already indexed block 73423490#51ea5 was not found on chain\n at Runner.assertWeAreOnTheSameChain (/app/node_modules/@subsquid/util-internal-processor-tools/lib/runner.js:107:19)\n at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n at async Runner.run (/app/node_modules/@subsquid/util-internal-processor-tools/lib/runner.js:47:17)"}}
indexer died
Investigation Steps Taken
-
Identified the Faulty Block
The error message clearly points to block height73423490. The logs also confirm that the processor was attempting to process73423491after this block. -
Queried the Local Database
I inspected the indexer's status database (chain_137_processor.status) and found the record for this block height:- Block Height:
73423490 - Local Hash:
0x51ea535a04328aea22aee24d8aacb0cc2ebc7d5af5b806b69558ae3737c4506c
- Block Height:
-
Compared with On-Chain Data
I verified the block's official hash using Polygonscan, which shows a mismatch with my local data:- Polygonscan Link: https://polygonscan.com/block/73423490
- Official Hash:
0xda83359d90a50115537a35472874149022c24151a307567781522a36d26c9f22
This confirms a discrepancy between the indexer's local state and the canonical chain.
-
Checked for Chain Reorgs
I reviewed Polygonscan's forked block records but found no public reorg events associated with this block height.
My Questions and Concerns
The logs suggest my setup may be somewhat complex, running multiple processors in parallel and switching between chain RPC and archive data sources. The error occurs when the indexer tries to validate its last synced block (73423490) and discovers that the hash does not exist on the RPC endpoint it's currently connected to.
Considering all the information above, my questions are:
- Transient RPC Inconsistency: Is it possible that my RPC provider (perhaps one of several nodes behind a load balancer) experienced a transient inconsistency or sync delay? This could have caused my indexer to ingest a "ghost" block that existed briefly on a micro-fork but was never part of the canonical chain.
- Impact of Complex Architecture: Could my architecture—running multiple processors in parallel and switching data sources—increase the risk of encountering such synchronization issues?