fix(otel): cap metric attributes so series count does not grow with traffic

[yassin-berriai]

TLDR

Problem this solves:

• GenAIMetricRecorder._common_attributes dumped the whole hidden_params object onto every metric datapoint as one safe_dumps label value. That object is per-request by construction, so the label value is unique on essentially every call, and a unique label value is a new time series
• All six GenAI instruments share those attributes, so one request minted up to six series that will never be written to again. This is the steady-state behavior of the feature, not an abuse case needing an attacker
• It is wrong twice over. Hosted backends bill on series count, so recommending that operators enable metrics would have meant a bill proportional to traffic. And a histogram whose every datapoint sits in its own series cannot be aggregated, so dashboards would look populated while answering nothing
• The same attribute set also carried client-supplied metadata (requester_metadata, spend_logs_metadata, user_api_key_end_user_id, requester_ip_address), so a caller could mint series out of a field they control

How it solves it:

• Both the success and the failure path now cap their attributes at METRIC_ATTRIBUTE_CEILING. That constant replaces the failure-only allowlist this PR is stacked on, so the two paths cannot drift apart
• The cap runs BEFORE the operator's otel.attributes filter, so an operator can narrow it and never widen it back to an unbounded label
• Client-supplied and per-request metadata is metric-ineligible and stays on the span, which already carries it and where cardinality is free
• hidden_params survives as a label but carries only model_id, the router's own deployment id, which is bounded by the deployment list and is what a per-deployment panel joins on. api_base is excluded despite naming the same deployment: it is a documented per-call parameter, so in SDK use it is caller-chosen, and a caller varying it would put the per-request cardinality straight back
• Because the shared validator accepts every span attribute name, a filter naming a metric-ineligible one now logs a warning once when the filter resolves, instead of silently emitting nothing for it

Relevant issues

Stacked on #35152, which established the bounded-attribute pattern on the failure path. Review that one first; this PR's own diff is the last commit.

Linear ticket

Resolves LIT-4967

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

A live proxy against real OpenAI (gpt-5.6, real spend), OTel v2 metrics on with the console exporter, three IDENTICAL requests each time. Series count is read off what the exporter actually emitted, parsed out of the exported JSON rather than eyeballed.

$ cat config.yaml
model_list:
  - model_name: probe-model
    litellm_params:
      model: openai/gpt-5.6
      api_key: os.environ/OPENAI_API_KEY
litellm_settings:
  callbacks: ["otel"]
general_settings:
  master_key: sk-1234

$ export LITELLM_OTEL_V2=1 LITELLM_OTEL_INTEGRATION_EXPORTER=console \
         LITELLM_OTEL_INTEGRATION_ENABLE_METRICS=true LITELLM_OTEL_INTEGRATION_METRIC_EXPORT_INTERVAL_MS=3000
$ python litellm/proxy/proxy_cli.py --config config.yaml --port 4967

$ for i in 1 2 3; do curl -s -o /dev/null -w "call $i http=%{http_code}\n" \
    -X POST http://127.0.0.1:4967/v1/chat/completions \
    -H 'Content-Type: application/json' -H 'Authorization: Bearer sk-1234' \
    -d '{"model":"probe-model","messages":[{"role":"user","content":"say ok"}],"max_completion_tokens":16}'; done
call 1 http=200
call 2 http=200
call 3 http=200

Before, on the base commit

Three identical requests, three separate series on every instrument. One request in, one series per instrument out.

instrument                               distinct series
  gen_ai.client.operation.duration       3
  gen_ai.client.response.duration        3
  gen_ai.client.token.usage              3
  gen_ai.usage.cost                      3

TOTAL distinct series across the GenAI instruments: 12

label keys present:
   gen_ai.framework
   gen_ai.operation.name
   gen_ai.request.model
   gen_ai.system
   hidden_params
   metadata.applied_guardrails
   metadata.requester_ip_address
   metadata.requester_metadata
   metadata.user_api_key_hash
   metadata.user_api_key_user_id

The label doing it, read off the first real call:

  request 1 hidden_params keys: ['additional_headers', 'api_base', 'batch_models', 'cache_key',
                                 'litellm_model_name', 'litellm_overhead_time_ms', 'model_id',
                                 'response_cost', 'usage_object']

response_cost, cache_key, litellm_overhead_time_ms, usage_object and the provider's additional_headers counters all move per call, so no two requests could ever share a series.

After, on this branch

Same three requests, one series per instrument, and it stays one however much traffic runs.

instrument                               distinct series
  gen_ai.client.operation.duration       1
  gen_ai.client.response.duration        1
  gen_ai.client.token.usage              1
  gen_ai.usage.cost                      1

TOTAL distinct series across the GenAI instruments: 4

label keys present:
   gen_ai.framework
   gen_ai.operation.name
   gen_ai.request.model
   gen_ai.system
   hidden_params
   metadata.user_api_key_hash
   metadata.user_api_key_user_id

hidden_params survives as the deployment identity and nothing else:

"hidden_params": "{\"model_id\": \"440e73889bf5117804b2cac8dcc4e4863cb4add7c322168ff51d685f14129a65\",
                  \"api_base\": \"https://api.openai.com\"}"

Four of the six instruments appear because these calls are non-streaming; the two streaming-only timing histograms are not recorded, which is pre-existing behaviour and unrelated to this change.

The ceiling really is a ceiling

An operator explicitly asking for the caller-controlled labels does not get them back, because the cap runs before the filter. Only the one requested attribute that is inside the ceiling survives.

$ cat config-include.yaml   # the relevant part
litellm_settings:
  callback_settings:
    otel:
      attributes:
        include_list:
          - "gen_ai.request.model"
          - "metadata.requester_ip_address"
          - "metadata.requester_metadata"

$ # two real calls, then read the exported series
TOTAL distinct series across the GenAI instruments: 4

label keys present:
   gen_ai.request.model

Test-level evidence

Five tests fail against the previous behaviour. The load-bearing one is that two requests differing only in per-request fields have to land in ONE series rather than two, which is the defect stated as a property rather than as an attribute list:

$ python -m pytest -q tests/test_litellm/integrations/otel/test_otel_v2_metrics.py   # pre-fix behaviour restored
FAILED test_two_calls_differing_only_per_request_share_one_series
  -> gen_ai.client.token.usage split into 2 series across 2 requests
FAILED test_hidden_params_label_carries_only_bounded_deployment_fields
FAILED test_success_attributes_are_capped_at_the_ceiling
FAILED test_failure_attributes_are_a_bounded_allowlist
FAILED test_a_metric_ineligible_filter_name_is_reported_not_silently_dropped
5 failed, 21 passed

With the fix:

$ python -m pytest -q tests/test_litellm/integrations/otel/
290 passed, 1 warning in 67.16s

Type

🐛 Bug Fix

Changes

METRIC_ATTRIBUTE_CEILING replaces FAILURE_ATTRIBUTE_KEYS and gains hidden_params; _bounded_attributes applies it and both record and record_failure go through it. _common_attributes builds the hidden_params label from BOUNDED_HIDDEN_PARAM_KEYS only.

Two judgement calls a reviewer should weigh rather than take on faith.

The first is that this removes attributes an operator may be reading today, and it does so deliberately rather than behind a flag. The alternative was to leave the default unbounded and let operators opt into the cap, which gets the incentives backwards: the failure mode is a bill and a set of unusable histograms, both of which show up long after the config choice, and the data being removed cannot be aggregated anyway. An operator who wants per-request detail already has it on the span. If a reviewer disagrees, the cheap compromise is to keep the ceiling as the default and add an explicit unsafe-widening escape hatch, which I would rather not add until someone asks for it.

The second is keeping hidden_params as a label name at all. Its contents are now a single field, so litellm.deployment.model_id would be the honest name. I kept the existing key so that queries already grouping on hidden_params keep working, and because renaming a label is a separate mechanical change that reviews better on its own. Flagging it because the name now badly undersells what the label is.

A review bot caught two things worth recording. api_base was in the bounded set in the first version of this PR and should not have been: I had reasoned about it as the deployment's base URL, which is true on a proxy reading it from config and false in SDK use where the caller passes it per call, and the general case is the one that decides whether a label is bounded. And the removal was originally silent, which is worse than the removal itself, hence the warning.

Worth noting what this does NOT fix. litellm/integrations/opentelemetry.py (the v1 integration) builds its metric attributes the same way and has the same defect. It is slated for deletion, and the v2 package is the live surface, so fixing it here and not there is deliberate; if v1's removal slips, that one needs the same cap.

QA runbook

1. Point a proxy at any real provider with OTel v2 metrics on: enable_metrics: true under the otel callback, plus an OTLP endpoint or the console exporter
2. Send the same /v1/chat/completions request three times
3. On staging, each request produces its own datapoint for each of the six gen_ai.client.* instruments, with a distinct hidden_params label; on this branch the three requests share one attribute set per instrument, so they aggregate into one series
4. Confirm the narrowing direction still works: set otel.attributes.exclude_list: ["metadata.user_api_key_hash"] and check the label is gone; then set include_list: ["hidden_params", "metadata.requester_ip_address"] and confirm the IP does NOT come back, because the ceiling runs first

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes default metric label sets operators may rely on for dashboards, but bounds cardinality and prevents unusable/billing-heavy series; behavior is covered by extensive tests.

Overview
OTel v2 GenAI metrics no longer attach unbounded labels that grew one time series per request. Success and failure recording both pass through METRIC_ATTRIBUTE_CEILING (replacing the failure-only allowlist) before otel.attributes include/exclude, so operators can narrow labels but cannot re-add client-controlled or per-request fields.

The hidden_params metric label is now a JSON dump of model_id only (BOUNDED_HIDDEN_PARAM_KEYS); full hidden_params, requester IP/metadata, spend logs metadata, and similar keys stay on spans only. Filter configs that name metric-ineligible span attributes log a one-time warning instead of failing silently.

Tests were updated and extended for shared ceiling behavior, single-series aggregation across differing per-request fields, bounded hidden_params, and the warning path.

^{Reviewed by Cursor Bugbot for commit 0376db4. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR bounds OpenTelemetry metric attributes consistently across success and failure paths.

• Restricts the hidden_params metric label to the router deployment identifier and removes caller-selectable api_base.
• Warns once when an operator’s attribute filter names a metric-ineligible dimension.
• Adds regression coverage for bounded attributes, stable series cardinality, filter behavior, and deployment-label contents.
• Documents the metric attribute ceiling and its interaction with operator filters.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported caller-selectable api_base dimension is excluded, and metric-ineligible filter names now produce an explicit warning.

Important Files Changed

Filename	Overview
litellm/integrations/otel/plumbing/metrics.py	Applies one bounded attribute ceiling to success and failure metrics, limits hidden parameters to model_id, and reports ineligible filter names.
tests/test_litellm/integrations/otel/test_otel_v2_metrics.py	Adds focused coverage proving per-request values no longer split metric series and both recording paths honor the ceiling.
litellm/integrations/otel/README.md	Documents the bounded-cardinality policy, excluded dimensions, and filter behavior.

_{Reviews (2): Last reviewed commit: "fix(otel): cap metric attributes so seri..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0376db4. Configure here.}