OpenTelemetry Integration Testing Guide

This document describes how to verify the rippled OpenTelemetry telemetry pipeline end-to-end, from span generation through the observability stack (otel-collector, Jaeger, Prometheus, Grafana).

Prerequisites

Build xrpld with telemetry

conan install . --build=missing -o telemetry=True
cmake --preset default -Dtelemetry=ON
cmake --build --preset default --target xrpld

The binary is at .build/xrpld.

Required tools

Docker with docker compose (v2)
curl
jq (JSON processor)

Verify binary

.build/xrpld --version

Test 1: Single-Node Standalone (Quick Verification)

This test verifies RPC and transaction spans in standalone mode. Consensus spans will not fire because standalone mode does not run consensus.

Step 1: Start the observability stack

docker compose -f docker/telemetry/docker-compose.yml up -d

Wait for services to be ready:

# otel-collector health
curl -sf http://localhost:13133/ && echo "collector ready"

# Jaeger UI
curl -sf http://localhost:16686/ > /dev/null && echo "jaeger ready"

Step 2: Start xrpld in standalone mode

.build/xrpld --conf docker/telemetry/xrpld-telemetry.cfg -a --start

Wait a few seconds for the node to initialize.

Step 3: Exercise RPC spans

# server_info
curl -s http://localhost:5005 \
  -d '{"method":"server_info"}' | jq .result.info.server_state

# server_state
curl -s http://localhost:5005 \
  -d '{"method":"server_state"}' | jq .result.state.server_state

# ledger
curl -s http://localhost:5005 \
  -d '{"method":"ledger","params":[{"ledger_index":"current"}]}' \
  | jq .result.ledger_current_index

Step 4: Submit a transaction

Close the ledger first (required in standalone mode):

curl -s http://localhost:5005 -d '{"method":"ledger_accept"}'

Submit a Payment from the genesis account:

curl -s http://localhost:5005 -d '{
  "method": "submit",
  "params": [{
    "secret": "snoPBrXtMeMyMHUVTgbuqAfg1SUTb",
    "tx_json": {
      "TransactionType": "Payment",
      "Account": "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh",
      "Destination": "rPMh7Pi9ct699iZUTWzJaUMR1o42VEfGqF",
      "Amount": "10000000"
    }
  }]
}' | jq .result.engine_result

Expected result: "tesSUCCESS".

Close the ledger again to finalize:

curl -s http://localhost:5005 -d '{"method":"ledger_accept"}'

Step 5: Verify traces in Jaeger

Wait 5 seconds for the batch export, then:

JAEGER="http://localhost:16686"

# Check rippled service is registered
curl -s "$JAEGER/api/services" | jq '.data'

# Check RPC spans
curl -s "$JAEGER/api/traces?service=rippled&operation=rpc.request&limit=5&lookback=1h" \
  | jq '.data | length'

curl -s "$JAEGER/api/traces?service=rippled&operation=rpc.process&limit=5&lookback=1h" \
  | jq '.data | length'

curl -s "$JAEGER/api/traces?service=rippled&operation=rpc.command.server_info&limit=5&lookback=1h" \
  | jq '.data | length'

# Check transaction spans
curl -s "$JAEGER/api/traces?service=rippled&operation=tx.process&limit=5&lookback=1h" \
  | jq '.data | length'

Or open the Jaeger UI: http://localhost:16686

Step 6: Teardown

# Kill xrpld (Ctrl+C or)
kill $(pgrep -f 'xrpld.*xrpld-telemetry')

# Stop observability stack
docker compose -f docker/telemetry/docker-compose.yml down

# Clean xrpld data
rm -rf data/

Expected spans (standalone mode)

Span Name	Expected	Notes
`rpc.request`	Yes	Every HTTP RPC call
`rpc.process`	Yes	Every RPC processing
`rpc.command.server_info`	Yes	server_info RPC
`rpc.command.server_state`	Yes	server_state RPC
`rpc.command.ledger`	Yes	ledger RPC
`rpc.command.submit`	Yes	submit RPC
`rpc.command.ledger_accept`	Yes	ledger_accept RPC
`tx.process`	Yes	Transaction submission
`tx.receive`	No	No peers in standalone
`consensus.*`	No	Consensus disabled standalone

Test 2: 6-Node Consensus Network (Full Verification)

This test verifies ALL span categories including consensus and peer transaction relay, using a 6-node validator network.

Automated

Run the integration test script:

bash docker/telemetry/integration-test.sh

The script will:

Start the observability stack
Generate 6 validator key pairs
Create config files for each node
Start all 6 nodes
Wait for consensus ("proposing" state)
Exercise RPC, submit transactions
Verify all span categories in Jaeger
Verify spanmetrics in Prometheus
Print results and leave the stack running

Manual

If you prefer to run the steps manually:

Step 1: Start observability stack

docker compose -f docker/telemetry/docker-compose.yml up -d

Step 2: Generate validator keys

Start a temporary standalone xrpld:

.build/xrpld --conf docker/telemetry/xrpld-telemetry.cfg -a --start &
TEMP_PID=$!
sleep 5

Generate 6 key pairs:

for i in $(seq 1 6); do
  curl -s http://localhost:5005 \
    -d '{"method":"validation_create"}' | jq '.result'
done

Record the validation_seed and validation_public_key for each. Kill the temporary node:

kill $TEMP_PID
rm -rf data/

Step 3: Create node configs

For each node (1-6), create a config file. Template:

[server]
port_rpc
port_peer

[port_rpc]
port = {5004 + node_number}
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http

[port_peer]
port = {51234 + node_number}
ip = 0.0.0.0
protocol = peer

[node_db]
type=NuDB
path=/tmp/xrpld-integration/node{N}/nudb
online_delete=256

[database_path]
/tmp/xrpld-integration/node{N}/db

[debug_logfile]
/tmp/xrpld-integration/node{N}/debug.log

[validation_seed]
{seed from step 2}

[validators_file]
/tmp/xrpld-integration/validators.txt

[ips_fixed]
127.0.0.1 51235
127.0.0.1 51236
127.0.0.1 51237
127.0.0.1 51238
127.0.0.1 51239
127.0.0.1 51240

[peer_private]
1

[telemetry]
enabled=1
endpoint=http://localhost:4318/v1/traces
exporter=otlp_http
sampling_ratio=1.0
batch_size=512
batch_delay_ms=2000
max_queue_size=2048
trace_rpc=1
trace_transactions=1
trace_consensus=1
trace_peer=0
trace_ledger=1

[rpc_startup]
{ "command": "log_level", "severity": "warning" }

[ssl_verify]
0

Step 4: Create validators.txt

[validators]
{public_key_1}
{public_key_2}
{public_key_3}
{public_key_4}
{public_key_5}
{public_key_6}

Step 5: Start all 6 nodes

for i in $(seq 1 6); do
  .build/xrpld --conf /tmp/xrpld-integration/node$i/xrpld.cfg --start &
  echo $! > /tmp/xrpld-integration/node$i/xrpld.pid
done

Step 6: Wait for consensus

Poll each node until server_state = "proposing":

for port in 5005 5006 5007 5008 5009 5010; do
  while true; do
    state=$(curl -s http://localhost:$port \
      -d '{"method":"server_info"}' \
      | jq -r '.result.info.server_state')
    echo "Port $port: $state"
    [ "$state" = "proposing" ] && break
    sleep 5
  done
done

Step 7: Exercise RPC and submit transaction

# RPC calls
curl -s http://localhost:5005 -d '{"method":"server_info"}'
curl -s http://localhost:5005 -d '{"method":"server_state"}'
curl -s http://localhost:5005 -d '{"method":"ledger","params":[{"ledger_index":"current"}]}'

# Submit transaction
curl -s http://localhost:5005 -d '{
  "method": "submit",
  "params": [{
    "secret": "snoPBrXtMeMyMHUVTgbuqAfg1SUTb",
    "tx_json": {
      "TransactionType": "Payment",
      "Account": "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh",
      "Destination": "rPMh7Pi9ct699iZUTWzJaUMR1o42VEfGqF",
      "Amount": "10000000"
    }
  }]
}'

Wait 15 seconds for consensus and batch export.

Step 8: Verify in Jaeger

See the "Verification Queries" section below.

Expected Span Catalog

All 16 production span names instrumented across Phases 2-5:

Span Name	Source File	Phase	Key Attributes	How to Trigger
`rpc.request`	ServerHandler.cpp:271	2	--	Any HTTP RPC call
`rpc.process`	ServerHandler.cpp:573	2	--	Any HTTP RPC call
`rpc.ws_message`	ServerHandler.cpp:384	2	--	WebSocket RPC message
`rpc.command.<name>`	RPCHandler.cpp:161	2	`xrpl.rpc.command`, `xrpl.rpc.version`, `xrpl.rpc.role`	Any RPC command
`tx.process`	NetworkOPs.cpp:1227	3	`xrpl.tx.hash`, `xrpl.tx.local`, `xrpl.tx.path`	Submit transaction
`tx.receive`	PeerImp.cpp:1273	3	`xrpl.peer.id`	Peer relays transaction
`tx.apply`	BuildLedger.cpp:88	5	`xrpl.ledger.tx_count`, `xrpl.ledger.tx_failed`	Ledger close (tx set)
`consensus.proposal.send`	RCLConsensus.cpp:177	4	`xrpl.consensus.round`	Consensus proposing phase
`consensus.ledger_close`	RCLConsensus.cpp:282	4	`xrpl.consensus.ledger.seq`, `xrpl.consensus.mode`	Ledger close event
`consensus.accept`	RCLConsensus.cpp:395	4	`xrpl.consensus.proposers`, `xrpl.consensus.round_time_ms`	Ledger accepted
`consensus.validation.send`	RCLConsensus.cpp:753	4	`xrpl.consensus.ledger.seq`, `xrpl.consensus.proposing`	Validation sent
`ledger.build`	BuildLedger.cpp:31	5	`xrpl.ledger.seq`	Ledger build
`ledger.validate`	LedgerMaster.cpp:915	5	`xrpl.ledger.seq`, `xrpl.ledger.validations`	Ledger validated
`ledger.store`	LedgerMaster.cpp:409	5	`xrpl.ledger.seq`	Ledger stored
`peer.proposal.receive`	PeerImp.cpp:1667	5	`xrpl.peer.id`, `xrpl.peer.proposal.trusted`	Peer sends proposal
`peer.validation.receive`	PeerImp.cpp:2264	5	`xrpl.peer.id`, `xrpl.peer.validation.trusted`	Peer sends validation

Verification Queries

Jaeger API

Base URL: http://localhost:16686

JAEGER="http://localhost:16686"

# List all services
curl -s "$JAEGER/api/services" | jq '.data'

# List operations for rippled
curl -s "$JAEGER/api/services/rippled/operations" | jq '.data'

# Query traces by operation
for op in "rpc.request" "rpc.process" \
          "rpc.command.server_info" "rpc.command.server_state" "rpc.command.ledger" \
          "tx.process" "tx.receive" "tx.apply" \
          "consensus.proposal.send" "consensus.ledger_close" \
          "consensus.accept" "consensus.validation.send" \
          "ledger.build" "ledger.validate" "ledger.store" \
          "peer.proposal.receive" "peer.validation.receive"; do
  count=$(curl -s "$JAEGER/api/traces?service=rippled&operation=$op&limit=5&lookback=1h" \
    | jq '.data | length')
  printf "%-35s %s traces\n" "$op" "$count"
done

Prometheus API

Base URL: http://localhost:9090

PROM="http://localhost:9090"

# Span call counts (from spanmetrics connector)
curl -s "$PROM/api/v1/query?query=traces_span_metrics_calls_total" \
  | jq '.data.result[] | {span: .metric.span_name, count: .value[1]}'

# Latency histogram
curl -s "$PROM/api/v1/query?query=traces_span_metrics_duration_milliseconds_count" \
  | jq '.data.result[] | {span: .metric.span_name, count: .value[1]}'

# RPC calls by command
curl -s "$PROM/api/v1/query?query=traces_span_metrics_calls_total{span_name=~\"rpc.command.*\"}" \
  | jq '.data.result[] | {command: .metric["xrpl.rpc.command"], count: .value[1]}'

StatsD Metrics (beast::insight)

rippled's built-in beast::insight framework emits StatsD metrics over UDP to the OTel Collector on port 8125. These appear in Prometheus alongside spanmetrics.

Requires [insight] config in xrpld.cfg:

[insight]
server=statsd
address=127.0.0.1:8125
prefix=rippled

Verify StatsD metrics in Prometheus:

# Ledger age gauge
curl -s "$PROM/api/v1/query?query=rippled_LedgerMaster_Validated_Ledger_Age" | jq '.data.result'

# Peer counts
curl -s "$PROM/api/v1/query?query=rippled_Peer_Finder_Active_Inbound_Peers" | jq '.data.result'

# RPC request counter
curl -s "$PROM/api/v1/query?query=rippled_rpc_requests" | jq '.data.result'

# State accounting
curl -s "$PROM/api/v1/query?query=rippled_State_Accounting_Full_duration" | jq '.data.result'

# Overlay traffic
curl -s "$PROM/api/v1/query?query=rippled_total_Bytes_In" | jq '.data.result'

Key StatsD metrics (prefix rippled_):

Metric	Type	Source
`LedgerMaster_Validated_Ledger_Age`	gauge	LedgerMaster.h:373
`LedgerMaster_Published_Ledger_Age`	gauge	LedgerMaster.h:374
`State_Accounting_{Mode}_duration`	gauge	NetworkOPs.cpp:774
`State_Accounting_{Mode}_transitions`	gauge	NetworkOPs.cpp:780
`Peer_Finder_Active_Inbound_Peers`	gauge	PeerfinderManager.cpp:214
`Peer_Finder_Active_Outbound_Peers`	gauge	PeerfinderManager.cpp:215
`Overlay_Peer_Disconnects`	gauge	OverlayImpl.h:557
`job_count`	gauge	JobQueue.cpp:26
`rpc_requests`	counter	ServerHandler.cpp:108
`rpc_time`	histogram	ServerHandler.cpp:110
`rpc_size`	histogram	ServerHandler.cpp:109
`ios_latency`	histogram	Application.cpp:438
`pathfind_fast`	histogram	PathRequests.h:23
`pathfind_full`	histogram	PathRequests.h:24
`ledger_fetches`	counter	InboundLedgers.cpp:44
`ledger_history_mismatch`	counter	LedgerHistory.cpp:16
`warn`	counter	Logic.h:33
`drop`	counter	Logic.h:34
`{category}_Bytes_In/Out`	gauge	OverlayImpl.h:535 (57 traffic categories)
`{category}_Messages_In/Out`	gauge	OverlayImpl.h:535 (57 traffic categories)

Grafana

Open http://localhost:3000 (anonymous admin access enabled).

Pre-configured dashboards (span-derived):

RPC Performance: Request rates, latency percentiles by command, top commands, WebSocket rate
Transaction Overview: Transaction processing rates, apply duration, peer relay, failed tx rate
Consensus Health: Consensus round duration, proposer counts, mode tracking, accept heatmap
Ledger Operations: Build/validate/store rates and durations, TX apply metrics
Peer Network: Proposal/validation receive rates, trusted vs untrusted breakdown (requires trace_peer=1)

Pre-configured dashboards (StatsD):

Node Health (StatsD): Validated/published ledger age, operating mode, I/O latency, job queue
Network Traffic (StatsD): Peer counts, disconnects, overlay traffic by category
RPC & Pathfinding (StatsD): RPC request rate/time/size, pathfinding duration, resource warnings

Pre-configured datasources:

Jaeger: Trace data at http://jaeger:16686
Prometheus: Metrics at http://prometheus:9090

Troubleshooting

No traces in Jaeger

Check otel-collector logs:

docker compose -f docker/telemetry/docker-compose.yml logs otel-collector

Verify xrpld telemetry config has enabled=1 and correct endpoint

Check that otel-collector port 4318 is accessible:

curl -sf http://localhost:4318 && echo "reachable"

Increase batch_delay_ms or decrease batch_size in xrpld config

Nodes not reaching "proposing" state

Check that all peer ports (51235-51240) are not in use:

for p in 51235 51236 51237 51238 51239 51240; do
  ss -tlnp | grep ":$p " && echo "port $p in use"
done

Verify [ips_fixed] lists all 6 peer ports
Verify validators.txt has all 6 public keys
Check node debug logs: tail -50 /tmp/xrpld-integration/node1/debug.log
Ensure [peer_private] is set to 1 (prevents reaching out to public network)

Transaction not processing

Verify genesis account exists:

curl -s http://localhost:5005 \
  -d '{"method":"account_info","params":[{"account":"rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh"}]}' \
  | jq .result.account_data.Balance

Check submit response for error codes
In standalone mode, remember to call ledger_accept after submitting

Spanmetrics not appearing in Prometheus

Verify otel-collector config has spanmetrics connector

Check that the metrics pipeline is configured:

service:
  pipelines:
    metrics:
      receivers: [spanmetrics]
      exporters: [prometheus]

Verify Prometheus can reach collector:

curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets'

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenTelemetry Integration Testing Guide

Prerequisites

Build xrpld with telemetry

Required tools

Verify binary

Test 1: Single-Node Standalone (Quick Verification)

Step 1: Start the observability stack

Step 2: Start xrpld in standalone mode

Step 3: Exercise RPC spans

Step 4: Submit a transaction

Step 5: Verify traces in Jaeger

Step 6: Teardown

Expected spans (standalone mode)

Test 2: 6-Node Consensus Network (Full Verification)

Automated

Manual

Step 1: Start observability stack

Step 2: Generate validator keys

Step 3: Create node configs

Step 4: Create validators.txt

Step 5: Start all 6 nodes

Step 6: Wait for consensus

Step 7: Exercise RPC and submit transaction

Step 8: Verify in Jaeger

Expected Span Catalog

Verification Queries

Jaeger API

Prometheus API

StatsD Metrics (beast::insight)

Grafana

Troubleshooting

No traces in Jaeger

Nodes not reaching "proposing" state

Transaction not processing

Spanmetrics not appearing in Prometheus

FilesExpand file tree

TESTING.md

Latest commit

History

TESTING.md

File metadata and controls

OpenTelemetry Integration Testing Guide

Prerequisites

Build xrpld with telemetry

Required tools

Verify binary

Test 1: Single-Node Standalone (Quick Verification)

Step 1: Start the observability stack

Step 2: Start xrpld in standalone mode

Step 3: Exercise RPC spans

Step 4: Submit a transaction

Step 5: Verify traces in Jaeger

Step 6: Teardown

Expected spans (standalone mode)

Test 2: 6-Node Consensus Network (Full Verification)

Automated

Manual

Step 1: Start observability stack

Step 2: Generate validator keys

Step 3: Create node configs

Step 4: Create validators.txt

Step 5: Start all 6 nodes

Step 6: Wait for consensus

Step 7: Exercise RPC and submit transaction

Step 8: Verify in Jaeger

Expected Span Catalog

Verification Queries

Jaeger API

Prometheus API

StatsD Metrics (beast::insight)

Grafana

Troubleshooting

No traces in Jaeger

Nodes not reaching "proposing" state

Transaction not processing

Spanmetrics not appearing in Prometheus