Skip to content

fix(dogstatsd): change stream handler task name to avoid unbounded cardinality (backport #1899 to 1.2.x) - #1902

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
releases/1.2.xfrom
tobz/backport-1899-1.2.x
Jun 22, 2026
Merged

fix(dogstatsd): change stream handler task name to avoid unbounded cardinality (backport #1899 to 1.2.x)#1902
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
releases/1.2.xfrom
tobz/backport-1899-1.2.x

Conversation

@tobz

@tobz tobz commented Jun 22, 2026

Copy link
Copy Markdown
Member

Summary

Backport of #1899 to the releases/1.2.x line.

Backport notes

Our cherry pick of aeda7d87a1bd7acc0f00e35d993862dc45c1a32d didn't apply cleanly itself but the diff we have here is identical to the diff in the original PR.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

How did you test this PR?

  • Manual local testing (original PR).
  • cargo check -p saluki-components on the backport branch.

References

DADP-2

…rdinality (#1899)

As stated in the PR title.

For DSD stream handlers tasks, we were naming them like so: `dogstatsd-stream-handler-{}-{}`, where the first portion was the listener type (`udp`, `unix`, etc) and the second was a "stream index.". For UDP and UDS datagram, this basically meant a single value: zero. For UDS streams, however, this value would grow over time as new connections were established.

On systems with a high volume of connect churn, this can lead to a number of orphaned metrics where the task name is used as a tag, thus creating a unique metric... which in turn becomes a memory leak, and CPU hog as more and more metrics have to be processed, copied during map resizes, and so on.

While we'd like to solve the problem of orphaned metrics not being automatically cleaned up, we're making a tactical fix here since the unbounded cardinality being removed will solve our immediate problem of said memory leakage.

- [x] Bug fix
- [ ] New feature
- [ ] Non-functional (chore, refactoring, docs)
- [ ] Performance

- Manual local testing.

DADP-2

Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com>
Copilot AI review requested due to automatic review settings June 22, 2026 18:21
@tobz
tobz requested a review from a team as a code owner June 22, 2026 18:21
@dd-octo-sts dd-octo-sts Bot added area/components Sources, transforms, and destinations. source/dogstatsd DogStatsD source. labels Jun 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Backports a DogStatsD fix to the releases/1.2.x line that prevents unbounded metric cardinality by removing a monotonically increasing stream index from DogStatsD stream handler task names (which are used as metric tags).

Changes:

  • Remove stream_idx state from the DogStatsD listener loop.
  • Update DogStatsD stream handler task naming to exclude the per-connection index, keeping only the listener type.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread lib/saluki-components/src/sources/dogstatsd/mod.rs
@pr-commenter

pr-commenter Bot commented Jun 22, 2026

Copy link
Copy Markdown

Binary Size Analysis (Agent Data Plane)

Baseline: b02c083 · Comparison: 7451d93 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 39.85 MiB (baseline) vs 39.95 MiB (comparison)
Size Change: +99.81 KiB (+0.24%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
tokio +54.71 KiB 1965
core +22.35 KiB 4415
datadog_protos::trace_piecemeal_include::datadog -15.21 KiB 15
piecemeal +14.73 KiB 20
saluki_components::sources::otlp -8.93 KiB 14
saluki_common::cache::CacheBuilder<K,V,W,H> +8.48 KiB 2
[sections] +6.16 KiB 7
saluki_components::transforms::trace_obfuscation +4.86 KiB 13
anon.b3d5d5fd69872b9304f15a2c9b377da7.6.llvm.7212443817926700567 -3.74 KiB 1
anon.e64bc2066bd8ff7bb460bd61255a7745.1128.llvm.15564187996750337528 +3.73 KiB 1
anon.4e30bd26b3561b4178ac48831e5a00b4.42.llvm.16543335135297065867 -3.45 KiB 1
anon.c7b4ec027e4589a074657c140d14b78f.19.llvm.7042418745101715639 +3.45 KiB 1
anyhow +3.15 KiB 453
anon.2890623bb822fa811707c690fe6ee6f9.759.llvm.12020083384501279044 -3.11 KiB 1
anon.ef44771aa2ff7d9d75d59b50744a8378.3.llvm.3301436067725601687 +3.11 KiB 1
anon.14b36ea4bd19552fe02b16005006dfd4.51.llvm.13321006570672589676 -2.65 KiB 1
anon.372d5e9f8d5cce6036695a230743f140.263.llvm.2124770014029419698 +2.65 KiB 1
alloc +2.62 KiB 661
futures_channel +2.49 KiB 43
anon.49c036daf795e72869b13248ffc2ab50.85.llvm.3359613750250240013 -2.46 KiB 1
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +1.1%  +100Ki  +0.4% +29.9Ki    [17320 Others]
  [NEW] +49.8Ki  [NEW] +49.6Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h7db6798142c795c5
  [NEW] +31.6Ki  [NEW] +31.5Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::ha21871c40ca9cb75
  [NEW] +30.1Ki  [NEW] +30.0Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hf6417ff87f736948
  [NEW] +24.4Ki  [NEW] +24.2Ki    agent_data_plane::internal::remote_agent::run_remote_agent_registration_loop::_{{closure}}::hd3d59513030a3a17
  [NEW] +24.1Ki  [NEW] +23.9Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::ha6b3e188798e584d
  [NEW] +22.6Ki  [NEW] +22.4Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::hd6b6e3cd8fbecbc9
  [NEW] +20.4Ki  [NEW] +20.3Ki    agent_data_plane::cli::debug::handle_debug_command::_{{closure}}::h02f9bfebf5bf906f
  [NEW] +18.5Ki  [NEW] +18.3Ki    _<backon::retry::Retry<B,T,E,Fut,FutureFn,SF$C&C$NF,AF> as core::future::future::Future>::poll::h1d3fbdb9aa9d6aa2
  [NEW] +17.0Ki  [NEW] +16.9Ki    agent_data_plane::main::_{{closure}}::hf74c76fa8f089484
  [NEW] +14.0Ki  [NEW] +13.9Ki    agent_data_plane::cli::run::write_sizing_guide::hfb58111bf9d25725
  [DEL] -14.2Ki  [DEL] -14.0Ki    saluki_components::transforms::trace_obfuscation::TraceObfuscation::obfuscate_span::h23c8d81d243b04bb
  [DEL] -17.0Ki  [DEL] -16.9Ki    agent_data_plane::main::_{{closure}}::h12c3b8fe741fc320
  [DEL] -18.7Ki  [DEL] -18.5Ki    _<backon::retry::Retry<B,T,E,Fut,FutureFn,SF$C&C$NF,AF> as core::future::future::Future>::poll::he34c035543edd635
  [DEL] -20.4Ki  [DEL] -20.3Ki    agent_data_plane::cli::debug::handle_debug_command::_{{closure}}::h0af248d922b07dc5
  [DEL] -22.6Ki  [DEL] -22.4Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h2e490f5a5e1fa24c
  [DEL] -24.2Ki  [DEL] -24.0Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::h1dd4120ec3ec6d5c
  [DEL] -24.6Ki  [DEL] -24.4Ki    agent_data_plane::internal::remote_agent::run_remote_agent_registration_loop::_{{closure}}::h6ae2b4ba5ed1f5ab
  [DEL] -30.1Ki  [DEL] -30.0Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::h11e8854c933a8098
  [DEL] -31.6Ki  [DEL] -31.5Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::h4e7da8d1ac8d0e9b
  [DEL] -49.8Ki  [DEL] -49.7Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h2f426cd320078741
  +0.2% +99.8Ki  +0.1% +29.2Ki    TOTAL

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jun 22, 2026

Copy link
Copy Markdown

Pipelines

Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/saluki | test-integration-windows-amd64   View in Datadog   GitLab

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7451d93 | Docs | Datadog PR Page | Give us feedback!

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit e8687b3 into releases/1.2.x Jun 22, 2026
83 of 85 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the tobz/backport-1899-1.2.x branch June 22, 2026 18:45
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jun 22, 2026
## Summary

Bumps the `agent-data-plane` crate version from `1.2.1` to `1.2.2`.

This release contains the backport of #1899 (DSD stream-handler task-name cardinality fix), landed via #1902.

## Change Type
- [ ] Bug fix
- [ ] New feature
- [x] Non-functional (chore, refactoring, docs)
- [ ] Performance

## How did you test this PR?

N/A

## References

DADP-2

Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com>
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jun 29, 2026
…ng expired/idle metrics (#1947)

## Summary

This PR introduces a new internal metrics registry to support evicting expired/idle metrics that would otherwise lead to unbounded memory growth over time.

Currently, we utilize the registry implementation from `metrics-util` which, while fast and optimized for low registration overhead, has one main limitation: no mechanism for automatically removing metrics which are "idle." This is approximated with helpers like `Recency` and `GenerationalAtomicStorage` (also from `metrics-util`) but can be somewhat convoluted and still requires user-managed integration to do correctly. Spurred by #1902, we wanted a way to automatically clean up metrics which have come and gone -- in particular: sparse, high-cardinality metrics -- so that we could limit the fallout of such misconfigured metrics in the future.

This PR introduces a new implementation of the internal metrics registry that supports detection (and eviction) of "idle" metrics as a first-class feature. At a high level, it looks like this:

- we created our own `MetricsRegistry`, which is just a synchronized set of hashmaps for each metric type (essentially identical to `metrics_util::registry::Registry`)
- likewise, we have our own metric handle types which now track the "idleness" of the metric (and, relatedly but not germane to the stated benefits: the level of the metric as well)
- all metric registration operations update the "idle" state of a metric: how recently was that metric retouched (this is mainly to cover cases where a metric is short-lived)
- during flush, we track the "idleness" of a metric: how many active references exist for this metric, how long ago was it last registered, did we see any updates to the metric for this flush
- every flush calculates both a set of metric updates (changes to active metrics, or new metrics) and a set of evictions (which metrics are now deemed truly idle and should be removed)
- the downstream code (`AggregatedMetricsProcessor`, etc) deals with this new flush format and appropriately upserts metric changes _or_ removes evicted ones 

I've tried to hew towards simplicity here: while the concurrent code is legitimately detail-oriented to ensure we don't lose updates, we own all of the code now, and we aren't using inherently complex techniques or code... just basic stuff like mutexes and atomics.

## Change Type
- [ ] Bug fix
- [x] New feature
- [ ] Non-functional (chore, refactoring, docs)
- [ ] Performance

## How did you test this PR?

Existing and new unit tests.

## References

DADP-2

Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com>
dd-octo-sts Bot pushed a commit that referenced this pull request Jun 29, 2026
…ng expired/idle metrics (#1947)

## Summary

This PR introduces a new internal metrics registry to support evicting expired/idle metrics that would otherwise lead to unbounded memory growth over time.

Currently, we utilize the registry implementation from `metrics-util` which, while fast and optimized for low registration overhead, has one main limitation: no mechanism for automatically removing metrics which are "idle." This is approximated with helpers like `Recency` and `GenerationalAtomicStorage` (also from `metrics-util`) but can be somewhat convoluted and still requires user-managed integration to do correctly. Spurred by #1902, we wanted a way to automatically clean up metrics which have come and gone -- in particular: sparse, high-cardinality metrics -- so that we could limit the fallout of such misconfigured metrics in the future.

This PR introduces a new implementation of the internal metrics registry that supports detection (and eviction) of "idle" metrics as a first-class feature. At a high level, it looks like this:

- we created our own `MetricsRegistry`, which is just a synchronized set of hashmaps for each metric type (essentially identical to `metrics_util::registry::Registry`)
- likewise, we have our own metric handle types which now track the "idleness" of the metric (and, relatedly but not germane to the stated benefits: the level of the metric as well)
- all metric registration operations update the "idle" state of a metric: how recently was that metric retouched (this is mainly to cover cases where a metric is short-lived)
- during flush, we track the "idleness" of a metric: how many active references exist for this metric, how long ago was it last registered, did we see any updates to the metric for this flush
- every flush calculates both a set of metric updates (changes to active metrics, or new metrics) and a set of evictions (which metrics are now deemed truly idle and should be removed)
- the downstream code (`AggregatedMetricsProcessor`, etc) deals with this new flush format and appropriately upserts metric changes _or_ removes evicted ones

I've tried to hew towards simplicity here: while the concurrent code is legitimately detail-oriented to ensure we don't lose updates, we own all of the code now, and we aren't using inherently complex techniques or code... just basic stuff like mutexes and atomics.

## Change Type
- [ ] Bug fix
- [x] New feature
- [ ] Non-functional (chore, refactoring, docs)
- [ ] Performance

## How did you test this PR?

Existing and new unit tests.

## References

DADP-2

Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com> 3752c5a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/components Sources, transforms, and destinations. mergequeue-status: done source/dogstatsd DogStatsD source.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants