ReconciliationJob::run_forever dies permanently on the first transient error

[christabel888]

Summary

ReconciliationJob::run_forever (backend/src/reconciliation/compare.rs) propagates the first error from any single reconciliation tick straight out of its loop, permanently terminating the background reconciliation daemon. Within a single tick, run_once also aborts entirely on the first period that fails to fetch, discarding results already computed for every other period in that cycle.

Location

backend/src/reconciliation/compare.rs — ReconciliationJob::run_once and ReconciliationJob::run_forever.

pub async fn run_forever(&self, interval: Duration) -> Result<(), ReconciliationError> {
    let mut ticker = tokio::time::interval(interval);
    loop {
        ticker.tick().await;
        self.run_once().await?;   // <-- any single error kills the loop forever
    }
}

for period in &periods {
    let offchain = self.offchain.get_aggregate(*period).await?;   // <-- aborts remaining periods
    let onchain = self.onchain.get_snapshot(*period).await?;      // <-- aborts remaining periods
    ...
}

Current gap / Motivation

Reconciliation is the system's ground truth for detecting drift between on-chain snapshots and off-chain aggregates, and (via MissingSubmissionHandler) for resubmitting missing data. It is explicitly the kind of job that must be resilient to transient failure — one RPC hiccup fetching a single period's on-chain snapshot should not:

1. Discard reconciliation results for every other period already fetched successfully in that cycle, and
2. Permanently kill the background job for the lifetime of the process, silently, with no retry, no backoff, no alert distinguishing "the daemon died" from "no discrepancies were found."

Whatever calls run_forever() (presumably spawned once at startup) has no way to know the difference between "reconciliation has been quietly not running for six hours" and "reconciliation ran and found nothing wrong" unless it's specifically watching task liveness — and even then, there's no way for it to resume without restarting the whole process (or the whole task, blindly, potentially in a crash loop against a persistently-unhealthy dependency).

The hard part

Making a scheduler like this correctly fault-tolerant is a genuinely subtle design problem:

1. Per-period isolation without silently losing discrepancy detection. A period that fails to fetch cannot simply be skipped and forgotten — that period never gets reconciled, meaning a real discrepancy in it goes undetected forever unless something tracks "periods that failed to reconcile and must be retried" as durable state, not just an in-memory Vec.
2. Distinguishing retryable from non-retryable failures. A network timeout fetching one period's on-chain snapshot should be retried; a systematic error (e.g. the on-chain snapshot for a period was never submitted at all, which is itself a legitimate discrepancy MissingSubmissionHandler should act on) must not be treated as a transient fetch failure and silently retried forever instead of being surfaced as the real discrepancy it is.
3. Backoff that doesn't turn a brief outage into a reconciliation backlog stampede. If the on-chain RPC endpoint is down for 10 minutes, run_forever needs to keep trying without hammering it, and then — once it recovers — needs a defined strategy for catching up on every period that was skipped during the outage without either recomputing an unbounded backlog synchronously (blocking real-time-relevant reconciliation) or losing track of which periods are still outstanding.
4. Correctness under crash-and-restart, not just in-process error recovery. If the whole process restarts (deploy, OOM-kill, node eviction), run_forever's state is gone. The real fix needs to define what "resumable" reconciliation progress looks like — is reconcilable_periods() itself already idempotent/replayable from durable state, or does the job need to persist a watermark of "periods reconciled so far" independently?

Implementation

• Make run_once collect and report per-period failures without aborting the batch; return a report that distinguishes "reconciled clean," "reconciled with discrepancy," and "failed to reconcile (retry needed)" per period.
• Make run_forever never propagate a single tick's error out of the loop — log/alert on tick failure and continue ticking, with explicit backoff for consecutive failures.
• Define and implement the retry/backlog strategy for periods that failed to reconcile, including how it survives a process restart.
• Add explicit test coverage: a single failing period doesn't lose reconciliation results for sibling periods in the same tick; a single failing tick doesn't stop subsequent ticks; a simulated outage-then-recovery correctly catches up without duplicate alerts or lost discrepancies.

Acceptance criteria

• A failure fetching one period's data does not discard reconciliation results already computed for other periods in the same run_once call.
• A failure in one tick does not stop run_forever from continuing to tick.
• Failed-to-reconcile periods are tracked and retried, not silently dropped.
• The retry/backlog approach's behavior across a process restart is explicitly defined and tested.
• Existing cargo test suite stays green.

[oyinade247]

Can I work on this pleaseee

[GazzyLee]

gm gm, I’d be happy to work on this. I’ll make ReconciliationJob resilient to individual period and tick failures by isolating per-period reconciliation, tracking failed periods for retry, and ensuring run_forever continues running with bounded backoff instead of terminating on transient errors. I’ll add tests covering partial period failures, continued execution after failed ticks, outage recovery, retry/backlog handling, and process restart scenarios to ensure no discrepancies are lost or alerts duplicated. I’ll also document the retry and recovery behavior and verify the existing cargo test suite remains green. Please assign