Skip to content

fix(metrics): bound fairness_id label cardinality and prune GC'd flow series#2119

Merged
liu-cong merged 3 commits into
llm-d:mainfrom
LukeAVanDrie:fix/fairness-label-cardinality
Jul 21, 2026
Merged

fix(metrics): bound fairness_id label cardinality and prune GC'd flow series#2119
liu-cong merged 3 commits into
llm-d:mainfrom
LukeAVanDrie:fix/fairness-label-cardinality

Conversation

@LukeAVanDrie

@LukeAVanDrie LukeAVanDrie commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind bug

What this PR does / why we need it:

The fairness_id label on request and flow-control metrics is populated from a client request header (x-llm-d-inference-fairness-id) or an agent-identity attribute, so its cardinality is not operator-bounded. Prometheus vectors never evict label combinations: every distinct fairness ID ever observed permanently allocated histogram series across both the deprecated and llm_d_epp metric families, growing pod RSS and scrape payloads without bound. This is a metrics-cardinality DoS against EPP memory and the metrics pipeline. Implements the cardinality policy proposed in #2106 (my proposal, awaiting maintainer sign-off). Sending the implementation alongside to make the proposal concrete and reviewable; if the discussion lands on a different option, I'll rework this PR to match.

Three changes, following the existing model-label precedent:

  • Bound fairness_id through a boundedLabel limiter (cap 1000, overflow to other) in every record function that takes it, exactly as model names are bounded via boundModels.
  • Bound model labels in RecordFlowControlRequestQueueDuration, which was missing the boundModels call its sibling functions have (model names come from the request body).
  • Prune a flow's flow-control series from both metric families when the registry garbage-collects the flow, so the vectors track live flows. Pruning runs after cleanupFlowResources and outside the registry lock; DeletePartialMatch scans whole vectors and must not run under it.

Accepted limitations (consistent with the model-label tradeoffs; see #2106 for the rationale): cap admission is first-come rather than allow-listed, and cap slots are never recycled. Memory stays bounded either way; per-tenant observability can degrade under a deliberate ID flood.

FullPath benchmark (registry/flow churn, M4 Pro, -benchtime=3s): 14.5KB -> 4.0KB per op, 264 -> 94 allocs per op.

Which issue(s) this PR fixes:

Per #2106; part of #1187.

Release note:

The fairness_id metric label is now capped at 1000 distinct values; additional fairness IDs are reported under the `other` label value. Flow control metric series for a fairness ID are removed when its flow is garbage collected. This bounds EPP memory and metrics-pipeline load, which were previously unbounded because the label derives from a client-controlled header.

… series

The fairness_id label on request and flow-control metrics is populated
from a client request header (or agent-identity attribute), so its
cardinality is not operator-bounded. Prometheus vectors never evict
label combinations: every distinct fairness ID ever observed permanently
allocated histogram series across both the deprecated and llm_d_epp
metric families, growing pod RSS and scrape payloads without bound.
Under flow-ID churn this was ~60% of remaining allocations in the
full-path flow-control benchmark.

Three changes, following the existing model-label precedent:

- Bound fairness_id through a boundedLabel limiter (cap 1000, overflow
  to "other") in every record function that takes it, exactly as model
  names are already bounded via boundModels.
- Bound model labels in RecordFlowControlRequestQueueDuration, which
  was missing the boundModels call its siblings have.
- Prune a flow's flow-control series (both families) when the registry
  garbage-collects the flow, so the vectors track live flows. Pruning
  runs after cleanupFlowResources and outside the registry lock;
  DeletePartialMatch scans whole vectors and must not run under it.

FullPath benchmark (registry/flow churn, M4 Pro, -benchtime=3s):
118k -> 213k d/s, 14.5KB -> 4.0KB per op, 264 -> 94 allocs per op.

Signed-off-by: Luke Van Drie <lukevandrie@google.com>
@LukeAVanDrie
LukeAVanDrie requested review from a team and shmuelk as code owners July 21, 2026 17:22
@LukeAVanDrie
LukeAVanDrie requested review from liu-cong and vMaroon July 21, 2026 17:22
@github-actions github-actions Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. labels Jul 21, 2026
…nality

- Verify the request-metric family bounds fairness_id like the flow
  control family, and that RecordFlowControlRequestQueueDuration bounds
  its model labels.
- Verify flow GC prunes the collected flow's metric series end to end.
- Document the client-derived label caps and GC pruning in
  docs/metrics.md.

Signed-off-by: Luke Van Drie <lukevandrie@google.com>
A flow whose client-chosen fairness ID equals the overflow value must not
delete the shared series that aggregates every capped-out tenant, so
DeleteFlowControlFlowSeries now no-ops on the overflow value.

Also document the revive-during-GC gauge race as an accepted limitation and
clarify in docs/metrics.md that the model labels share one cap pool and that
caps apply over the process lifetime.

Signed-off-by: Luke Van Drie <lukevandrie@google.com>
@LukeAVanDrie

Copy link
Copy Markdown
Contributor Author

cc: @liu-cong

@github-actions github-actions Bot added kind/bug Categorizes issue or PR as related to a bug. and removed kind/bug Categorizes issue or PR as related to a bug. labels Jul 21, 2026
@liu-cong
liu-cong merged commit cfa8a91 into llm-d:main Jul 21, 2026
31 checks passed
elevran pushed a commit that referenced this pull request Jul 22, 2026
Signed-off-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com>
Co-authored-by: llm-d-router-release-notes[bot] <287676111+llm-d-router-release-notes[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants