Phase 5: Observability stack — spanmetrics, dashboards, runbook

[pratikmankawde]

PR Chain: #6436 → #6437 → #6438 → #6424 → #6425 → #6426 → #6427 (this PR) → #6433 / #6439
Base: pratik/otel-phase4-consensus-tracing

High Level Overview of Change

Add the observability stack for consuming trace data: OTel Collector spanmetrics connector for deriving RED metrics, Prometheus for metrics storage, and three pre-built Grafana dashboards (RPC Performance, Consensus Health, Transaction Overview). Includes an operator runbook for setup, configuration, and troubleshooting.

Context of Change

Phase 5 of the OpenTelemetry distributed tracing project. Phases 2–4 produce trace spans but operators need dashboards and metrics to get value from the data. The spanmetrics connector in the OTel Collector derives rate/error/duration (RED) metrics from trace spans without additional instrumentation. Prometheus scrapes these metrics, and Grafana provides visualization.

The three dashboards cover the primary tracing domains:

• RPC Performance: request rate, p95 latency, error rate, latency heatmap
• Consensus Health: round duration, proposal rate, validation rate
• Transaction Overview: processing rate, latency, sync vs async path distribution

Design doc: OpenTelemetryPlan/ directory in this repo.

Type of Change

• New feature (non-breaking change which adds functionality)
• Documentation update

API Impact

No API impact. This change only affects the Docker-based observability stack and documentation.

Test Plan

• docker compose up and verify Prometheus scrapes spanmetrics at :8889.
• Verify Grafana dashboards load with the auto-provisioned Prometheus datasource.
• Submit transactions and verify metrics flow: rippled → Collector → spanmetrics → Prometheus → Grafana.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.7%. Comparing base (735ecd1) to head (85f583f).

Additional details and impacted files

@@                         Coverage Diff                          @@
##           pratik/otel-phase4-consensus-tracing   #6427   +/-   ##
====================================================================
  Coverage                                  79.7%   79.7%           
====================================================================
  Files                                       851     851           
  Lines                                     67952   67952           
  Branches                                   7594    7589    -5     
====================================================================
+ Hits                                      54184   54189    +5     
+ Misses                                    13768   13763    -5     

Files with missing lines	Coverage Δ
include/xrpl/telemetry/TraceContextPropagator.h	96.4% <ø> (ø)
src/libxrpl/telemetry/Telemetry.cpp	22.5% <ø> (ø)
src/xrpld/app/main/Application.cpp	70.4% <ø> (ø)

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.