fix(backend): make ReconciliationJob resilient to transient errors and tick failures (#380)

[s6pa1rta3n-lab]

Resolves #380

Summary of Changes

1. Per-Period Error Isolation in `run_once`:

   • `ReconciliationJob::run_once` and `MissingSubmissionHandler::run_once` now isolate per-period fetch errors (on-chain read timeouts or off-chain DB errors) so that sibling periods in the same tick are processed without interruption.
   • Per-period outcomes are structured via `PeriodOutcome` and `PeriodReconciliationStatus` (`Clean`, `Discrepant`, `Failed`).

2. Durable & In-Memory Retry Backlog:

   • Introduced `ReconciliationStateStore` trait (with default `InMemoryReconciliationStore`) to track failed-to-reconcile periods across ticks and process restarts.
   • Periods that fail due to transient network/RPC errors are recorded in the retry backlog and retried on subsequent ticks until resolved.
   • Clean or discrepant comparisons mark the period reconciled, clearing pending retries.

3. Backlog Bounding & Stampede Protection:

   • Configurable `max_batch_size` ensures that after prolonged outages, the backlog is processed in rate-limited batches rather than causing synchronous backlog stampedes.

4. Resilient Background Daemon (`run_forever` & `run_until_shutdown`):

   • `run_forever` catches tick errors, emits alerts/logs, applies exponential backoff via `BackoffConfig`, and continues ticking.
   • Added `run_until_shutdown` for graceful cooperative shutdown.

5. Comprehensive Test Suite:

   • Added unit and integration tests in `backend/tests/reconciliation_fault_tolerance_test.rs` validating sibling result preservation, retry on recovery, state persistence across process restarts, backoff in daemon loops, and rate-limited batch processing.

Acceptance Criteria Checklist

• A failure fetching one period's data does not discard reconciliation results already computed for other periods in the same `run_once` call.
• A failure in one tick does not stop `run_forever` from continuing to tick.
• Failed-to-reconcile periods are tracked and retried, not silently dropped.
• The retry/backlog approach's behavior across a process restart is explicitly defined and tested.
• Existing `cargo test` suite stays green (44 passed).

Payout Routing

• EVM (Base/Arbitrum/Polygon/ETH): `0xF46C9F6d70C50BF81ef3588AB523a90a594a2F89`
• Stellar: `GCL6OXAMLD75BMTINA6EMRUDWK5THQUSHMYNLSNBCJAPZJHNYJTUNIBC`