Skip to content

fix: Automatic detection and backfilling of gaps in block history at startup#4

Open
mouradski wants to merge 1 commit intoflare-foundation:mainfrom
mouradski:main
Open

fix: Automatic detection and backfilling of gaps in block history at startup#4
mouradski wants to merge 1 commit intoflare-foundation:mainfrom
mouradski:main

Conversation

@mouradski
Copy link

Problem:

During batch indexing, if a block fails to insert (timeout, constraint violation, etc.), it is silently ignored due to INSERT IGNORE.

On application restart, the indexer would simply resume after MAX(number) from the blocks table, leaving permanent gaps in the history.

Example:
Blocks in DB: [1000, 1001, 1002, 1005, 1006] ← blocks 1003-1004 missing
Previous behavior: restart → starts at 1007
Result: blocks 1003-1004 are permanently lost

Solution:

  • Added a continuity check at startup in getStartIndex():
  • Retrieve statistics: MIN(number), MAX(number) and COUNT(*)
  • Gap detection: compare expected vs actual block count
  • Locate first gap: SQL query with NOT EXISTS to find the first missing block
  • Smart resume: indexer resumes from the first gap instead of max

Robustness:
✅ Executed once at startup → negligible cost
✅ Self-healing → after a crash, the next restart automatically fills gaps
✅ Performance → SQL query is O(n log n) with index on number
✅ Backward compatible → unchanged behavior if no gaps detected

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant