Fix indexer freeze when a single event fails to process (#1214)

[Hamda-gbade]

Summary

Fixes #1214 — a single failing Soroban event could freeze the event indexer permanently.

Root cause: In fetchAndProcessEvents, lastCursor/lastLedger only advanced while !hasError. Once any event in a batch failed, the cursor stopped advancing for every subsequent event in that batch — including successfully processed ones. A single always-failing event (e.g. a malformed body) caused every later event to be re-fetched and reprocessed on every poll indefinitely, freezing the indexer and reprocessing the tail of the event stream forever.

Changes

Indexer worker (backend/src/workers/soroban-event-worker.ts)

• Cursor fix: Removed the !hasError cursor guard. The cursor now always advances past every successfully processed event, even when an earlier event in the same batch failed.
• Dead-lettering: Failed events are upserted into the new IndexerDeadLetterEvent table keyed by RPC event id, preserving the raw payload, the last error message, and an incremented attempts counter so operators can triage them manually.
• Retry cap: New INDEXER_DEAD_LETTER_MAX_RETRIES env var (default 5). Once an event has failed that many times it is abandoned — logged, recorded in the dead-letter table, and the cursor advances past it so the indexer never stalls on a permanently-bad event.
• A dead-letter write failure never aborts the batch; the event is simply left in place for the next poll.

Schema & migration

• New IndexerDeadLetterEvent model in backend/prisma/schema.prisma (eventId unique, ledger, transactionHash, rawPayload, errorMessage, attempts, lastAttemptAt, createdAt) with indexes on ledger, txHash, and createdAt.
• New migration 20260830000000_add_indexer_dead_letter_event. Deployed environments need prisma migrate deploy.

Tests (backend/tests/soroban-event-worker.test.ts)

• Updated the old mixed-batch test, which asserted the buggy frozen-cursor behavior, to assert the cursor now advances past the failed event and the failure is dead-lettered.
• New regression test (acceptance criteria): a batch of five events with one deliberately malformed event in the middle — the other four are processed exactly once, the malformed event is never written but is dead-lettered with its raw payload, and the cursor advances past the malformed event.
• New retry-cap test: an always-failing tail event is retried across polls, then abandoned after the cap (set to 2) with the cursor advancing past it — the next poll re-fetches nothing.

Docs

• Documented INDEXER_DEAD_LETTER_MAX_RETRIES and the dead-letter behavior in backend/docs/SSE_ARCHITECTURE.md.

Testing

• tsc --noEmit clean; Prisma client regenerated.
• Unit suite: 317 passed / 3 skipped (real-DB integration tests).
• Mocked-DB integration suite: 71 passed / 13 skipped.
• The single pre-existing unhandled-rejection in tests/workers.index.test.ts (stream-runway-worker mock gap) is unrelated to this change and reproduces on the base branch.

Closes #1214

[K1NGD4VID]

resolve conflict on this issue @Hamda-gbade