feat: add cumulative replication traffic accounting - #172
Conversation
The 2026-07-09 production bandwidth diagnosis (V2-623) found chunk fetch/offer and replication payloads invisible in the logs. Add cumulative, monotonic per-variant byte/message counters surfaced as INFO summary lines so the telegraf->Elasticsearch pipeline can attribute how much bandwidth each replication message type moves, per direction. Purely additive: no wire changes, no behaviour changes. - Add a process-global per-variant counter table in the replication protocol module (relaxed atomics), incremented at the single encode (tx) and decode (rx) choke points that every replication message passes through. - Emit replication traffic summary (cumulative) INFO lines (target ant_node::replication::traffic) from a new periodic task in the replication engine. The 17 variants x 4 flat keys exceed tracing's 32-field event cap, so the flat keys are split across three lines sharing the same target and message; the shipping pipeline lifts each into its own field. Verified: cargo check (default and --no-default-features), cargo fmt --all, cargo clippy --all-features -D warnings -D clippy::panic -D clippy::unwrap_used -D clippy::expect_used all clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds process-wide, cumulative per-ReplicationMessageBody traffic counters and periodically emits INFO summary lines so the logging pipeline can attribute replication bandwidth by message type and direction (tx/rx), without changing wire format or runtime behavior.
Changes:
- Increment per-variant tx/rx byte + message counters at the
ReplicationMessage::encode/decodechoke points. - Add a process-global relaxed-atomic counter table and a
log_traffic_summary()emitter that splits fields across 3 tracing events. - Start a new periodic background task in the replication engine to emit the cumulative summaries.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/replication/protocol.rs | Adds global relaxed-atomic counter tables, per-variant indexing, and INFO summary logging; hooks tx/rx accounting into encode/decode. |
| src/replication/mod.rs | Adds a periodic background task that emits the traffic summary at a fixed cadence. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| fn record_tx(body: &ReplicationMessageBody, bytes: usize) { | ||
| let i = body.variant_index(); | ||
| REPL_TX_BYTES[i].fetch_add(bytes as u64, Ordering::Relaxed); | ||
| REPL_TX_COUNT[i].fetch_add(1, Ordering::Relaxed); | ||
| } | ||
|
|
||
| /// Record a decoded (rx) replication message against its variant. | ||
| fn record_rx(body: &ReplicationMessageBody, bytes: usize) { | ||
| let i = body.variant_index(); | ||
| REPL_RX_BYTES[i].fetch_add(bytes as u64, Ordering::Relaxed); | ||
| REPL_RX_COUNT[i].fetch_add(1, Ordering::Relaxed); | ||
| } |
| @@ -70,8 +76,14 @@ impl ReplicationMessage { | |||
| max_size: MAX_REPLICATION_MESSAGE_SIZE, | |||
| }); | |||
| } | |||
| postcard::from_bytes(data) | |||
| .map_err(|e| ReplicationProtocolError::DeserializationFailed(e.to_string())) | |||
| let message: Self = postcard::from_bytes(data) | |||
| .map_err(|e| ReplicationProtocolError::DeserializationFailed(e.to_string()))?; | |||
|
|
|||
| // V2-623: cumulative per-variant rx accounting. Every replication | |||
| // receive funnels through here, so this is the single rx choke point. | |||
| record_rx(&message.body, data.len()); | |||
|
|
|||
| Ok(message) | |||
| let shutdown = self.shutdown.clone(); | ||
| let handle = tokio::spawn(async move { | ||
| loop { | ||
| tokio::select! { | ||
| () = shutdown.cancelled() => break, | ||
| () = tokio::time::sleep(std::time::Duration::from_secs( | ||
| TRAFFIC_SUMMARY_INTERVAL_SECS, | ||
| )) => { | ||
| protocol::log_traffic_summary(); | ||
| } | ||
| } | ||
| } | ||
| debug!("Replication traffic summary loop shut down"); | ||
| }); |
dirvine
left a comment
dirvine
left a comment
There was a problem hiding this comment.
Hermes review — conditionally ready after CI and ingestion smoke test
No wire-format or replication-behaviour change found. The encode/decode accounting is internally consistent, and local cargo check plus cargo check --no-default-features passed at head a86521345d8cdd63d795537765f341610532122a.
Caveats:
- TX counters measure successfully encoded replication messages, not confirmed network sends. Failed sends/retries can therefore exceed actual egress. Please name/document these as encoded/attempted traffic unless accounting is moved to the send-success path.
- The fixed 17-variant index/table is currently exhaustive and in bounds, but future enum/table drift could panic. A focused test asserting unique in-range indices and coverage of all emitted groups would cheaply guard this.
- The three sparse INFO events are workable for per-field cumulative queries, but please verify all three groups and all 68 fields survive the real Telegraf/Elasticsearch pipeline.
Current CI: format, Clippy, docs, security audit, no-logging test, and Linux/macOS builds pass; Windows build and platform tests are still pending.
Recommendation: no code-level production blocker found for diagnostic use, subject to all required CI completing green and one >5-minute ingestion smoke test. Not approved yet while CI is pending.
dirvine
left a comment
dirvine
left a comment
There was a problem hiding this comment.
Follow-up approval
No code-level blocker found at head a86521345d8cdd63d795537765f341610532122a.
The counters consistently measure successfully encoded/decoded replication protocol traffic. TX is encoded/attempted volume rather than confirmed network delivery; that is acceptable for diagnostic use provided the query retains that interpretation.
Format, Clippy, documentation, security audit, no-logging and all platform builds are green. macOS and Windows tests were still pending at the latest check; approval does not waive those required jobs.
A Telegraf/Elasticsearch smoke test is optional canary validation of the three grouped events and field mappings, not a code-safety gate.
Summary
Part of V2-623 (log-based traffic accounting). Chunk fetch/offer and
replication payloads are invisible in the logs today. This adds cumulative,
monotonic per-variant byte/message counters as INFO summary lines so the
telegraf→Elasticsearch pipeline can attribute how much bandwidth each
replication message type moves, per direction.
Purely additive: no wire changes, no behaviour changes.
(relaxed atomics), incremented at the single
encode(tx) anddecode(rx)choke points that every replication message passes through.
replication traffic summary (cumulative)INFO lines (targetant_node::replication::traffic) from a periodic task in the replicationengine.
Note on line splitting: the 17 variants × 4 flat keys (68) exceed
tracing's 32-field event cap, so the flat keys (fresh_offer_tx_bytes, …) aresplit across three lines sharing the same target and message and distinguished
by a
groupfield. The shipping pipeline lifts each key into its own field, sothe acceptance query (max per field per hour → delta → per-variant MB/h) is
unaffected.
Test plan
cargo check(default features)cargo check --no-default-features(logging compiled out)cargo clippy --all-features -- -D warnings -D clippy::panic -D clippy::unwrap_used -D clippy::expect_usedcargo fmt --all -- --check🤖 Generated with Claude Code