Core payment-reliability/latency-percentile computation engine does not exist

[christabel888]

Summary

The project's stated purpose (README: "Real-time payment analytics for Stellar... measuring and improving cross-border payment reliability") requires computing reliability and latency-percentile metrics (P50/P95/P99-class statistics) over a stream of ledger-close events that can arrive out of order and on a different clock than wall time — but no such computation exists anywhere in the codebase. There is substantial infrastructure for ingesting, indexing, reconciling, and serving contract/event data (event_indexer, reconciliation, realtime, contract_ops), but the actual core analytics computation the whole system exists to support is missing.

Location

None yet — this is a request to build a new component. Closest related existing code: backend/src/event_indexer/ (ingestion), backend/src/reconciliation/compare.rs (which already depends on the concept of a "period" being reconcilable — i.e. finalized — which is exactly the watermarking problem below), backend/src/observability/metrics.rs (currently a 5-line stub).

Current gap / Motivation

grep -rln "reliability\|percentile\|watermark" backend/src returns nothing. There is no module computing the payment-reliability or latency statistics the product is named for. This isn't a missing test or a missing edge case in an existing function — it's a missing core subsystem, and it happens to be one of the genuinely hard problems in streaming systems engineering.

The hard part

1. Streaming, mergeable percentile estimation at scale. Exact percentiles require sorting the full dataset, which doesn't scale to a continuously-arriving, unbounded stream. A correct implementation needs a mergeable summary structure (e.g. t-digest, HDR histogram, or a KLL sketch) with a proven, bounded error on P50/P95/P99 estimates — and the error bound needs to be understood and documented, not just "close enough in testing." The structure also needs to be mergeable across time windows and across parallel ingestion shards without re-processing raw data, since re-scanning the full event history on every query is not viable at any real scale.
2. Out-of-order event handling with a real watermarking strategy. Stellar ledger-close events can arrive at the indexer out of ledger-close order (RPC/Horizon replication lag, retries, multi-source ingestion). A percentile computation keyed by time window must decide, for each window, when it is safe to consider "final" (a watermark) versus when late-arriving events must still be admitted and the window's stats retroactively corrected — and it must do this without either (a) waiting so long that "real-time" stops being true, or (b) finalizing prematurely and silently under/over-counting reliability incidents that arrived a few seconds late.
3. Two clocks, one truth. Stellar ledger close time and wall-clock ingestion time diverge under network conditions, indexer backpressure, and ledger production irregularities. "Latency" for a cross-border payment has to be defined precisely — latency of what, measured against which clock, and how do the two get reconciled when they disagree — before any percentile of it means anything. This needs an explicit, documented decision, because getting it wrong doesn't fail loudly; it just produces plausible-looking, wrong numbers.
4. Consistency with the reconciliation subsystem's notion of "finalized." reconciliation/compare.rs already operates on the concept of "reconcilable periods" — implying some existing notion of when a period's data is considered complete enough to reconcile. Whatever watermarking strategy this new component adopts needs to either reuse that notion or have an explicit, justified reason for a different one; two different, disagreeing definitions of "this time window is final" in the same system is a correctness bug waiting to happen.
5. Backpressure-safe under ingestion bursts. A ledger backlog replay (e.g. after indexer downtime) can produce a burst of events far exceeding steady-state rate. The percentile/reliability computation must not fall over, produce garbage intermediate results, or block the real-time path when this happens — it needs a defined degraded-mode behavior that's still correct, even if delayed.

Implementation

This is intentionally not prescriptive beyond the constraints above — the right sketch structure, windowing strategy, and clock-reconciliation policy are real design decisions with tradeoffs that should be argued for, not assumed. At minimum, a complete solution needs to:

• Choose and justify a mergeable percentile sketch with documented, bounded error characteristics.
• Define the watermarking policy precisely (how long a window stays open, how late data is handled once it closes) and reconcile it with reconciliation's existing period-finality concept.
• Define latency's clock basis precisely and document the tradeoff.
• Demonstrate correctness under out-of-order arrival, including a worst-case reordering/lateness scenario, with a test proving the computed percentiles converge to within the sketch's documented error bound of the true values.
• Demonstrate the system doesn't degrade correctness (only, at worst, timeliness) under a simulated ingestion burst.

Acceptance criteria

• A percentile/reliability computation exists with a documented, tested error bound versus exact percentiles on the same dataset.
• Out-of-order event arrival is handled per an explicit, documented watermarking policy, with a test proving late data within the policy's tolerance is still correctly incorporated and data outside it is handled in a defined (not silently wrong) way.
• The latency clock basis (ledger-close time vs. ingestion time, and how divergence is handled) is explicitly documented and consistently applied.
• The watermarking policy here and reconciliation's period-finality concept are shown to agree, or the deliberate difference is documented with rationale.
• A simulated ingestion burst does not produce incorrect (as opposed to merely delayed) results.
• Existing cargo test suite stays green.

Out of scope

The frontend visualization of these metrics; this issue is specifically about the correctness of the underlying computation.

[Jenola344]

Hi maintainer, I will love to work on this project kindly assign me

[Agbasimere]

Hello maintainer, I'd like to work on this issue.

[Chidi-Dev1]

Hi maintainer I will love to work on this issue.
Please assign it to me.
Thank you.

[DNO8]

I have experience building backend systems involving event processing, data aggregation, and correctness-sensitive logic. I would approach this by first tracing the existing event-indexing and reconciliation flows, especially how reconciliation/compare.rs determines when a period is finalized, so the new analytics engine does not introduce a conflicting notion of finality.

For the percentile computation, I would use a mergeable sketch such as KLL, with its error characteristics explicitly documented and tested against exact percentiles for P50/P95/P99. I would keep the sketch, event-time windowing, watermarking, and reliability counters as separate components.

I would define latency using Stellar ledger-close time rather than ingestion wall-clock time, while retaining ingestion lag as an operational metric. For out-of-order events, I would implement an explicit watermark/allowed-lateness policy: events arriving within the lateness bound are incorporated into their original window, while events arriving after finalization follow a defined correction/rejection path rather than being silently ignored.

Finally, I would add tests for reordered events, late arrivals, percentile error bounds, reconciliation/finality consistency, and large ingestion bursts. The key correctness property would be that ingestion bursts or replay can increase processing latency, but must never change the final computed metrics.

[jchinedu]

let me work on this issue

[GazzyLee]

hello boss, I’d like to take this on. I’ll design the streaming analytics component around a mergeable percentile sketch with a documented error bound, while defining a clear watermark and late-event policy that aligns with the reconciliation subsystem.

I’ll also establish a consistent latency clock model, handle out-of-order and bursty ingestion safely, and add tests comparing estimates against exact percentiles under late-arrival and high-load scenarios.

[Iker2522]

I would like to work on this issue because percentile and reliability metrics need to be correct and well-defined, especially when events arrive out of order or in bursts. I can start by documenting and justifying the percentile sketch, watermarking policy, and latency clock basis, including how they align with reconciliation finality. Then I can implement the computation with focused tests covering out-of-order and late events, worst-case lateness, ingestion bursts, and the documented error bound against exact percentiles, while keeping the existing Cargo test suite green.