Sliding-Window Ledger Event Sync Re-indexer Catching Horizon Disruptions

[JamesEjembi]

Problem Statement

Temporary connection drops with Stellar Horizon nodes can result in missing contract state modification logs. When the sync worker disconnects during a ledger close event, it has no mechanism to detect the gap, causing the local database to permanently miss state transitions. Over days, these gaps compound, and the node begins rejecting attestation proofs due to out-of-date local state.

Technical Bounds & Invariants

• Horizon event stream: Server-Sent Events (SSE) on /transactions endpoint
• Ledger close interval: 5-6 seconds on Stellar testnet
• Event gap detection window: up to 256 consecutive missed ledgers
• Re-sync throughput: 500 events/second (limited by Horizon rate limits)
• Must not re-process already-confirmed events (idempotent replay)

Codebase Navigation Guide

• Primary target: /src/blockchain/event_sync.ts
• Horizon SSE client: /src/blockchain/horizon_client.ts — streamEvents() at line 50
• Event processor: /src/blockchain/event_processor.ts — processLedgerEvents() at line 80
• Cursor persistence: /src/database/sync_cursor.sql

Step-by-Step Resolution Blueprint

1. Maintain a sliding-window cursor table sync_cursors(ledger_seq BIGINT PRIMARY KEY, processed_at TIMESTAMPTZ) that records the last 256 confirmed ledger sequences; after each successful event batch, upsert the highest processed sequence and purge entries older than 256 behind the high-water mark
2. Add a findGaps() routine that runs every 60 seconds: SELECT generate_series(low + 1, high - 1) AS gap FROM (SELECT MIN(ledger_seq) AS low, MAX(ledger_seq) AS high FROM sync_cursors) ranges WHERE low < high — this detects any missing sequences in the window
3. On gap detection, spawn a backfillWorker(gapStart, gapEnd) that calls HorizonClient.getLedgerDetails(seq) for each missing ledger in sequence, respecting Horizon rate limits (10 requests/second via token-bucket)
4. For each backfilled ledger, pass the events through EventProcessor.processLedgerEvents() — the processor uses an upsert on (event_id, ledger_seq) to guarantee idempotency
5. After backfill completes, update the sync cursor high-water mark to gapEnd and log the gap range, count of events recovered, and recovery duration via OpenTelemetry
6. Implement a health-check endpoint GET /health/sync that returns { "ledger_gap_count": N, "last_processed_seq": M, "horizon_lag_seconds": L } for monitoring alerts
7. Add a chaos test that drops the SSE connection mid-stream and verifies that gaps are detected and backfilled within 2 gap-detection cycles (120s)

[BountySpaghetti]

Hello maintainer, I understand the expected outcome and would love to work on this.

[grantfox-oss]

🦊 GrantFox — @BountySpaghetti has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #218)
2. Your PR will be reviewed by the VeriNode-Labs maintainers

Good luck! Track your progress on GrantFox.