Implement Service-Level Objectives, Burn-Rate Alerting, and Traceable Pipeline Health

[Inkman007]

Problem

Metrics, tracing, dashboards, and SLO documentation exist, but production readiness requires a tested causal path from user-visible objectives to alerts and runbooks. Component-up health checks do not detect stale ingestion, delayed scores, dropped webhooks, or invalid model output.

Scope

• Define SLIs/SLOs for ingestion freshness/completeness, score availability/latency, API correctness/latency, webhook delivery, model validity, and on-chain publication lag.
• Implement multi-window burn-rate alerts with paging/ticket thresholds.
• Propagate trace context from chain ingestion through feature computation, inference, persistence, API/webhook, and Soroban publishing.
• Add freshness/readiness checks that fail on stale checkpoints, incompatible/missing models, exhausted queues, or unavailable required dependencies.
• Validate metric cardinality and tenant/privacy redaction.
• Link every actionable alert to a tested runbook and dashboard.

Acceptance criteria

• Synthetic failure tests prove each critical alert fires and clears.
• Alerts measure user-visible symptoms and include diagnostic context without secrets/wallet leakage.
• Readiness removes an unhealthy replica before it serves incorrect/stale results.
• SLO calculations and burn windows are unit-tested with recorded fixtures.
• Trace sampling retains errors and anomalous scores while enforcing a cost budget.
• A game-day validates detection and response for at least ingestion stall, model failure, and storage saturation.