fix(metrics): bound fairness_id label cardinality and prune GC'd flow series#2119
Merged
liu-cong merged 3 commits intoJul 21, 2026
Merged
Conversation
… series The fairness_id label on request and flow-control metrics is populated from a client request header (or agent-identity attribute), so its cardinality is not operator-bounded. Prometheus vectors never evict label combinations: every distinct fairness ID ever observed permanently allocated histogram series across both the deprecated and llm_d_epp metric families, growing pod RSS and scrape payloads without bound. Under flow-ID churn this was ~60% of remaining allocations in the full-path flow-control benchmark. Three changes, following the existing model-label precedent: - Bound fairness_id through a boundedLabel limiter (cap 1000, overflow to "other") in every record function that takes it, exactly as model names are already bounded via boundModels. - Bound model labels in RecordFlowControlRequestQueueDuration, which was missing the boundModels call its siblings have. - Prune a flow's flow-control series (both families) when the registry garbage-collects the flow, so the vectors track live flows. Pruning runs after cleanupFlowResources and outside the registry lock; DeletePartialMatch scans whole vectors and must not run under it. FullPath benchmark (registry/flow churn, M4 Pro, -benchtime=3s): 118k -> 213k d/s, 14.5KB -> 4.0KB per op, 264 -> 94 allocs per op. Signed-off-by: Luke Van Drie <lukevandrie@google.com>
…nality - Verify the request-metric family bounds fairness_id like the flow control family, and that RecordFlowControlRequestQueueDuration bounds its model labels. - Verify flow GC prunes the collected flow's metric series end to end. - Document the client-derived label caps and GC pruning in docs/metrics.md. Signed-off-by: Luke Van Drie <lukevandrie@google.com>
A flow whose client-chosen fairness ID equals the overflow value must not delete the shared series that aggregates every capped-out tenant, so DeleteFlowControlFlowSeries now no-ops on the overflow value. Also document the revive-during-GC gauge race as an accepted limitation and clarify in docs/metrics.md that the model labels share one cap pool and that caps apply over the process lifetime. Signed-off-by: Luke Van Drie <lukevandrie@google.com>
Contributor
Author
|
cc: @liu-cong |
liu-cong
approved these changes
Jul 21, 2026
elevran
pushed a commit
that referenced
this pull request
Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
What this PR does / why we need it:
The
fairness_idlabel on request and flow-control metrics is populated from a client request header (x-llm-d-inference-fairness-id) or an agent-identity attribute, so its cardinality is not operator-bounded. Prometheus vectors never evict label combinations: every distinct fairness ID ever observed permanently allocated histogram series across both the deprecated andllm_d_eppmetric families, growing pod RSS and scrape payloads without bound. This is a metrics-cardinality DoS against EPP memory and the metrics pipeline. Implements the cardinality policy proposed in #2106 (my proposal, awaiting maintainer sign-off). Sending the implementation alongside to make the proposal concrete and reviewable; if the discussion lands on a different option, I'll rework this PR to match.Three changes, following the existing model-label precedent:
fairness_idthrough aboundedLabellimiter (cap 1000, overflow toother) in every record function that takes it, exactly as model names are bounded viaboundModels.RecordFlowControlRequestQueueDuration, which was missing theboundModelscall its sibling functions have (model names come from the request body).cleanupFlowResourcesand outside the registry lock;DeletePartialMatchscans whole vectors and must not run under it.Accepted limitations (consistent with the model-label tradeoffs; see #2106 for the rationale): cap admission is first-come rather than allow-listed, and cap slots are never recycled. Memory stays bounded either way; per-tenant observability can degrade under a deliberate ID flood.
FullPath benchmark (registry/flow churn, M4 Pro,
-benchtime=3s): 14.5KB -> 4.0KB per op, 264 -> 94 allocs per op.Which issue(s) this PR fixes:
Per #2106; part of #1187.
Release note: