fix(dogstatsd): change stream handler task name to avoid unbounded cardinality (backport #1899 to 1.2.x) - #1902
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit intoJun 22, 2026
Conversation
…rdinality (#1899) As stated in the PR title. For DSD stream handlers tasks, we were naming them like so: `dogstatsd-stream-handler-{}-{}`, where the first portion was the listener type (`udp`, `unix`, etc) and the second was a "stream index.". For UDP and UDS datagram, this basically meant a single value: zero. For UDS streams, however, this value would grow over time as new connections were established. On systems with a high volume of connect churn, this can lead to a number of orphaned metrics where the task name is used as a tag, thus creating a unique metric... which in turn becomes a memory leak, and CPU hog as more and more metrics have to be processed, copied during map resizes, and so on. While we'd like to solve the problem of orphaned metrics not being automatically cleaned up, we're making a tactical fix here since the unbounded cardinality being removed will solve our immediate problem of said memory leakage. - [x] Bug fix - [ ] New feature - [ ] Non-functional (chore, refactoring, docs) - [ ] Performance - Manual local testing. DADP-2 Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com>
thieman
approved these changes
Jun 22, 2026
There was a problem hiding this comment.
Pull request overview
Backports a DogStatsD fix to the releases/1.2.x line that prevents unbounded metric cardinality by removing a monotonically increasing stream index from DogStatsD stream handler task names (which are used as metric tags).
Changes:
- Remove
stream_idxstate from the DogStatsD listener loop. - Update DogStatsD stream handler task naming to exclude the per-connection index, keeping only the listener type.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Binary Size Analysis (Agent Data Plane)Baseline: b02c083 · Comparison: 7451d93 · diff ✅ Binary size difference within thresholdChanges by Module
Detailed Symbol Changes |
|
gh-worker-dd-mergequeue-cf854d
Bot
merged commit Jun 22, 2026
e8687b3
into
releases/1.2.x
83 of 85 checks passed
4 tasks
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Jun 22, 2026
## Summary Bumps the `agent-data-plane` crate version from `1.2.1` to `1.2.2`. This release contains the backport of #1899 (DSD stream-handler task-name cardinality fix), landed via #1902. ## Change Type - [ ] Bug fix - [ ] New feature - [x] Non-functional (chore, refactoring, docs) - [ ] Performance ## How did you test this PR? N/A ## References DADP-2 Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com>
Merged
4 tasks
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Jun 29, 2026
…ng expired/idle metrics (#1947) ## Summary This PR introduces a new internal metrics registry to support evicting expired/idle metrics that would otherwise lead to unbounded memory growth over time. Currently, we utilize the registry implementation from `metrics-util` which, while fast and optimized for low registration overhead, has one main limitation: no mechanism for automatically removing metrics which are "idle." This is approximated with helpers like `Recency` and `GenerationalAtomicStorage` (also from `metrics-util`) but can be somewhat convoluted and still requires user-managed integration to do correctly. Spurred by #1902, we wanted a way to automatically clean up metrics which have come and gone -- in particular: sparse, high-cardinality metrics -- so that we could limit the fallout of such misconfigured metrics in the future. This PR introduces a new implementation of the internal metrics registry that supports detection (and eviction) of "idle" metrics as a first-class feature. At a high level, it looks like this: - we created our own `MetricsRegistry`, which is just a synchronized set of hashmaps for each metric type (essentially identical to `metrics_util::registry::Registry`) - likewise, we have our own metric handle types which now track the "idleness" of the metric (and, relatedly but not germane to the stated benefits: the level of the metric as well) - all metric registration operations update the "idle" state of a metric: how recently was that metric retouched (this is mainly to cover cases where a metric is short-lived) - during flush, we track the "idleness" of a metric: how many active references exist for this metric, how long ago was it last registered, did we see any updates to the metric for this flush - every flush calculates both a set of metric updates (changes to active metrics, or new metrics) and a set of evictions (which metrics are now deemed truly idle and should be removed) - the downstream code (`AggregatedMetricsProcessor`, etc) deals with this new flush format and appropriately upserts metric changes _or_ removes evicted ones I've tried to hew towards simplicity here: while the concurrent code is legitimately detail-oriented to ensure we don't lose updates, we own all of the code now, and we aren't using inherently complex techniques or code... just basic stuff like mutexes and atomics. ## Change Type - [ ] Bug fix - [x] New feature - [ ] Non-functional (chore, refactoring, docs) - [ ] Performance ## How did you test this PR? Existing and new unit tests. ## References DADP-2 Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com>
dd-octo-sts Bot
pushed a commit
that referenced
this pull request
Jun 29, 2026
…ng expired/idle metrics (#1947) ## Summary This PR introduces a new internal metrics registry to support evicting expired/idle metrics that would otherwise lead to unbounded memory growth over time. Currently, we utilize the registry implementation from `metrics-util` which, while fast and optimized for low registration overhead, has one main limitation: no mechanism for automatically removing metrics which are "idle." This is approximated with helpers like `Recency` and `GenerationalAtomicStorage` (also from `metrics-util`) but can be somewhat convoluted and still requires user-managed integration to do correctly. Spurred by #1902, we wanted a way to automatically clean up metrics which have come and gone -- in particular: sparse, high-cardinality metrics -- so that we could limit the fallout of such misconfigured metrics in the future. This PR introduces a new implementation of the internal metrics registry that supports detection (and eviction) of "idle" metrics as a first-class feature. At a high level, it looks like this: - we created our own `MetricsRegistry`, which is just a synchronized set of hashmaps for each metric type (essentially identical to `metrics_util::registry::Registry`) - likewise, we have our own metric handle types which now track the "idleness" of the metric (and, relatedly but not germane to the stated benefits: the level of the metric as well) - all metric registration operations update the "idle" state of a metric: how recently was that metric retouched (this is mainly to cover cases where a metric is short-lived) - during flush, we track the "idleness" of a metric: how many active references exist for this metric, how long ago was it last registered, did we see any updates to the metric for this flush - every flush calculates both a set of metric updates (changes to active metrics, or new metrics) and a set of evictions (which metrics are now deemed truly idle and should be removed) - the downstream code (`AggregatedMetricsProcessor`, etc) deals with this new flush format and appropriately upserts metric changes _or_ removes evicted ones I've tried to hew towards simplicity here: while the concurrent code is legitimately detail-oriented to ensure we don't lose updates, we own all of the code now, and we aren't using inherently complex techniques or code... just basic stuff like mutexes and atomics. ## Change Type - [ ] Bug fix - [x] New feature - [ ] Non-functional (chore, refactoring, docs) - [ ] Performance ## How did you test this PR? Existing and new unit tests. ## References DADP-2 Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com> 3752c5a
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backport of #1899 to the
releases/1.2.xline.Backport notes
Our cherry pick of
aeda7d87a1bd7acc0f00e35d993862dc45c1a32ddidn't apply cleanly itself but the diff we have here is identical to the diff in the original PR.Change Type
How did you test this PR?
cargo check -p saluki-componentson the backport branch.References
DADP-2