fix(metrics): dedup metrics v3 tags by rayz · Pull Request #1817 · DataDog/saluki

rayz · 2026-06-04T16:34:33Z

Summary

Fixes logical mismatches found when testing the v3 pipeline in validation mode where v2 tag hashes differed from v3 the v3 tag hash.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

How did you test this PR?

References

datadog-official · 2026-06-04T16:34:39Z

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 3 Pipeline jobs failed

DataDog/saluki | check-protos

DataDog/saluki | test-integration-macos-arm64

DataDog/saluki | build-correctness-tools-image

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 905d1e5 | Docs | Datadog PR Page | Give us feedback!}

pr-commenter · 2026-06-04T16:37:07Z

Regression Detector (Agent Data Plane)

Run ID: 06f782c7-8ac6-41bc-b7df-53417a3f1fd6
Baseline: 08cd4c40 · Comparison: 8fc15c08 · diff

Optimization Goals: ❌ 1 regression detected

experiment	goal	Δ mean %	links
dsd_uds_100mb_3k_contexts_cpu (erratic)	cpu	🔴 +33.27	metrics profiles logs

Fine details of change detection per experiment (34)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
otlp_ingest_logs_5mb_memory (ignored)	memory	⚪ +5.74	metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic)	cpu	⚪ +3.84	metrics profiles logs
otlp_ingest_logs_5mb_cpu (ignored)	cpu	⚪ +1.83	metrics profiles logs
dsd_uds_10mb_3k_contexts_cpu (erratic)	cpu	⚪ +1.11	metrics profiles logs
otlp_ingest_traces_5mb_cpu (erratic)	cpu	⚪ +0.91	metrics profiles logs
otlp_ingest_metrics_5mb_memory	memory	⚪ +0.44	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory	memory	⚪ +0.24	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory	memory	⚪ +0.23	metrics profiles logs
dsd_uds_512kb_3k_contexts_memory	memory	⚪ +0.20	metrics profiles logs
dsd_uds_1mb_3k_contexts_cpu (erratic)	cpu	⚪ +0.15	metrics profiles logs
otlp_ingest_traces_5mb_memory	memory	⚪ +0.13	metrics profiles logs
otlp_ingest_traces_5mb_throughput	throughput	⚪ -0.13	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput	throughput	⚪ -0.09	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ +0.05	metrics profiles logs
dsd_uds_1mb_3k_contexts_memory	memory	⚪ +0.03	metrics profiles logs
dsd_uds_100mb_3k_contexts_memory	memory	⚪ +0.02	metrics profiles logs
dsd_uds_500mb_3k_contexts_memory	memory	⚪ +0.02	metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput	throughput	⚪ -0.01	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput	throughput	⚪ -0.01	metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored)	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput	throughput	⚪ +0.00	metrics profiles logs
otlp_ingest_metrics_5mb_throughput	throughput	⚪ +0.01	metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput	throughput	⚪ +0.04	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.06	metrics profiles logs
dsd_uds_10mb_3k_contexts_memory	memory	⚪ -0.12	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.15	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.20	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ -0.22	metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic)	cpu	⚪ -1.13	metrics profiles logs
dsd_uds_512kb_3k_contexts_cpu (erratic)	cpu	⚪ -1.37	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic)	cpu	⚪ -1.41	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic)	cpu	⚪ -2.35	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 117 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 39.8 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 60.5 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 182 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 26.9 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

pr-commenter · 2026-06-04T18:51:51Z

Binary Size Analysis (Agent Data Plane)

Baseline: 08cd4c4 · Comparison: 905d1e5 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 37.93 MiB (baseline) vs 38.06 MiB (comparison)
Size Change: +142.26 KiB (+0.37%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
`core`	+84.97 KiB	14331
`saluki_components::encoders::datadog`	+76.28 KiB	373
`saluki_components::config_registry::datadog`	-57.32 KiB	68
`anyhow`	+31.21 KiB	1706
`axum`	-25.77 KiB	504
`otlp_protos::otlp_include::opentelemetry`	-22.45 KiB	246
`saluki_components::common::datadog`	+21.58 KiB	445
`figment`	-20.53 KiB	661
`tonic`	+18.15 KiB	474
`hashbrown`	+17.11 KiB	1113
`http_body_util`	-16.40 KiB	202
`prost`	+16.05 KiB	577
`serde_core`	+15.95 KiB	932
`[sections]`	+13.02 KiB	9
`saluki_components::forwarders::datadog`	+12.82 KiB	22
`&mut serde_json`	-12.68 KiB	98
`rustls`	-12.65 KiB	164
`serde_json`	+9.33 KiB	305
`saluki_core::topology::interconnect`	-7.68 KiB	76
`tokio_rustls`	+6.75 KiB	92

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +1.0%  +225Ki  +1.0%  +181Ki    [47866 Others]
  [NEW]  +151Ki  [NEW]  +151Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hdf43122119cb417f
  [NEW] +84.7Ki  [NEW] +84.5Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::h04cdcf5f98317be5
  [NEW] +67.4Ki  [NEW] +67.3Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::hdd0ec93af12084ef
  [NEW] +66.7Ki  [NEW] +66.5Ki    saluki_core::topology::built::BuiltTopology::spawn::_{{closure}}::h5f0395680bdd9ebd
  [NEW] +58.3Ki  [NEW] +58.1Ki    agent_data_plane::cli::debug::handle_debug_command::_{{closure}}::hbacedc5f90e31de8
  [NEW] +57.9Ki  [NEW] +57.7Ki    saluki_core::topology::blueprint::TopologyBlueprint::build::_{{closure}}::hfd190c8aff62510c
  [NEW] +56.8Ki  [NEW] +56.7Ki    core::ops::function::FnOnce::call_once::h85b8424640380127
  [NEW] +41.2Ki  [NEW] +41.0Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h3a46bad6c2ce93e7
  [NEW] +38.3Ki  [NEW] +38.1Ki    agent_data_plane::internal::env::workload::build_collector::_{{closure}}::h52ba18eaf2437ae0
  [DEL] -37.8Ki  [DEL] -37.7Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::h9670fb6d171f7edc
  [DEL] -38.0Ki  [DEL] -37.8Ki    agent_data_plane::internal::env::workload::build_collector::_{{closure}}::h48c3bde421c3ca9e
  [DEL] -42.8Ki  [DEL] -42.6Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h01bb1ce7d2bee930
  [DEL] -46.6Ki  [DEL] -46.5Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::h24d23376b04440eb
  [DEL] -55.3Ki  [DEL] -55.2Ki    saluki_components::config_registry::datadog::SUPPORTED_ANNOTATIONS::_{{closure}}::h80552fc4b6f8591d
  [DEL] -57.3Ki  [DEL] -57.1Ki    agent_data_plane::cli::debug::handle_debug_command::_{{closure}}::h62090eaa45b10b54
  [DEL] -57.9Ki  [DEL] -57.8Ki    saluki_core::topology::blueprint::TopologyBlueprint::build::_{{closure}}::h10b60228b955c2dd
  [DEL] -65.4Ki  [DEL] -65.3Ki    saluki_core::topology::built::BuiltTopology::spawn::_{{closure}}::h5b418dd5fcc8e912
  [DEL] -68.0Ki  [DEL] -67.9Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h3212bb8ddcd8b298
  [DEL] -84.1Ki  [DEL] -83.9Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hf1b9b95c6e5e7eda
  [DEL]  -153Ki  [DEL]  -152Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hdf07f62ba689f516
  +0.4%  +142Ki  +0.3% +99.0Ki    TOTAL

Copilot

Pull request overview

This PR adjusts the Datadog metrics V3 encoder to deduplicate tags during encoding, addressing mismatches observed in validation mode where V2 and V3 tag hashing/representation diverged due to duplicate tags.

Changes:

Introduce a reusable tag deduplicator for V3 batch encoding.
Deduplicate the combined tag iterator before setting metric tags in V3.
Deduplicate the tag iterator used when promoting device / dd.internal.resource:* tags into V3 resources.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

    let mut writer = v3::V3Writer::new();
+    let mut tags_deduplicator = ReusableDeduplicator::new();

    for metric in metrics {
-        write_metric_to_v3(&mut writer, metric, additional_tags);
+        write_metric_to_v3(&mut writer, metric, additional_tags, &mut tags_deduplicator);


+        .chain(metric.context().origin_tags());
+    let all_tags = tags_deduplicator
+        .deduplicated(chained_tags)
        .filter(|t| is_sketch || !is_v3_series_resource_tag(t) && !is_v3_series_device_tag(t))


## Summary  Fixes logical mismatches found when testing the v3 pipeline in validation mode where v2 tag hashes differed from v3 the v3 tag hash. ## Change Type - [x] Bug fix - [ ] New feature - [ ] Non-functional (chore, refactoring, docs) - [ ] Performance ## How did you test this PR?  ## References

dedup metrics v3 tags

a1cfcf4

dd-octo-sts Bot added area/components Sources, transforms, and destinations. encoder/datadog-metrics Datadog Metrics encoder. labels Jun 4, 2026

add log for testing deploy

8cfce90

rayz added 2 commits June 4, 2026 15:23

Remove logging

acdc04b

Debug requests log

5c1639b

dd-octo-sts Bot added the forwarder/datadog Datadog forwarder. label Jun 8, 2026

Use info logging

571ab82

rayz changed the title ~~dedup metrics v3 tags~~ fix(metrics): dedup metrics v3 tags Jun 10, 2026

rayz added 2 commits June 11, 2026 11:09

Remove logging

8fc15c0

More cleanup

905d1e5

dd-octo-sts Bot removed the forwarder/datadog Datadog forwarder. label Jun 11, 2026

rayz marked this pull request as ready for review June 11, 2026 15:44

Copilot AI review requested due to automatic review settings June 11, 2026 15:44

rayz requested a review from a team as a code owner June 11, 2026 15:44

Copilot started reviewing on behalf of rayz June 11, 2026 15:45 View session

rayz merged commit ef6d87a into tobz/datadog-metrics-v3-payload-support Jun 11, 2026
44 of 77 checks passed

rayz deleted the rayz/metrics-v3-dedup-tags branch June 11, 2026 15:45

Copilot AI reviewed Jun 11, 2026

View reviewed changes

Navigation Menu

Uh oh!

fix(metrics): dedup metrics v3 tags - #1817

fix(metrics): dedup metrics v3 tags#1817
rayz merged 7 commits into
tobz/datadog-metrics-v3-payload-supportfrom
rayz/metrics-v3-dedup-tags

rayz commented Jun 4, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Jun 4, 2026 •

edited by datadog-prod-us1-3 Bot

Loading

Uh oh!

pr-commenter Bot commented Jun 4, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Jun 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rayz commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change Type

How did you test this PR?

References

Uh oh!

datadog-official Bot commented Jun 4, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

Uh oh!

pr-commenter Bot commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector (Agent Data Plane)

Optimization Goals: ❌ 1 regression detected

Uh oh!

pr-commenter Bot commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Analysis (Agent Data Plane)

✅ Binary size difference within threshold

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rayz commented Jun 4, 2026 •

edited

Loading

datadog-official Bot commented Jun 4, 2026 •

edited by datadog-prod-us1-3 Bot

Loading

pr-commenter Bot commented Jun 4, 2026 •

edited

Loading

pr-commenter Bot commented Jun 4, 2026 •

edited

Loading