fix(otel): record the GenAI duration metric on failed requests - #35152
Conversation
`_record_metrics` ran only from `async_log_success_event`, so `gen_ai.client.operation.duration` counted only the requests that worked. Latency read off it during an incident was the latency of the surviving traffic, and with no error dimension anywhere there was no way to build a failure-rate panel or a success/failure split per model. A failed call now records the same duration histogram, tagged with the semconv `error.type` (the mapped provider exception's class name, bounded by construction; the message stays on the span). Success attributes are untouched, so an existing query can still isolate the old series with `error_type=""`. The other five instruments describe a completed generation and are skipped rather than filled with a fabricated zero: litellm hands the failure callback no `response_obj`, so there is no usage to split and no completion-token count, and it zeroes `response_cost` on failure. A proxy-gate rejection (auth / rate limit) records nothing, for the same reason it gets no span; no upstream call happened. `error.type` is stamped after the cardinality filter, like `gen_ai.token.type`, so an `otel.attributes` include/exclude list cannot strip the discriminator and silently merge failures into the success series. Resolves LIT-4955
|
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Greptile SummaryThis PR extends OpenTelemetry GenAI duration metrics to failed upstream requests.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| litellm/integrations/otel/logger.py | Routes eligible failure callbacks through the guarded GenAI metric recorder while retaining existing early-return behavior. |
| litellm/integrations/otel/plumbing/metrics.py | Adds failure-duration recording with bounded attributes and layered error-type fallback handling. |
| tests/test_litellm/integrations/otel/test_otel_v2_metrics.py | Adds focused regression coverage for failure metrics, cardinality controls, filtering, and unchanged success attributes. |
| litellm/integrations/otel/README.md | Documents the maintainer-facing behavior of failed-request metric recording. |
Reviews (2): Last reviewed commit: "fix(otel): bound the failure metric's at..." | Re-trigger Greptile
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The failure datapoint reused the success path's full attribute set, which carries client-supplied fields (`metadata.requester_metadata`, `metadata.spend_logs_metadata`, the end-user id taken from the request's `user` field) and per-request ones (the `hidden_params` blob holding the provider's response headers). A failed request needs no provider spend, so nothing rate-limits a caller who puts a unique value in a field they control and mints one histogram series per request. A failure now carries a bounded allowlist: the operation enum, provider, request model, framework, the key/alias/team/org/user identifiers, and `error.type`. Every entry is a fixed enum or an operator-provisioned identifier, so the failure series count is bounded by the deployment's own key, team and user count while the labels still answer which team on which model is failing and how. The user email is left out as PII duplicating the user id already on the series. The operator's `otel.attributes` filter layers on top, so it narrows the allowlist further and never widens it.
|
Pushed It acts on the cardinality finding: a failure datapoint no longer reuses the success attribute set, it carries a bounded allowlist of the operation enum, provider, request model, framework, the key/alias/team/org/user identifiers, and Live re-capture at The success series still carries the caller's Two tests cover the allowlist. One drives the same payload through the success path first, asserts those keys really were present there, then asserts the failure datapoint's attribute set exactly, so it is a proven removal and the exact-set form is the guard against refactoring back to |
|
The The sole failure is Evidence it is base:
Tracked as LIT-4975. Leaving it red rather than pushing a no-op to re-roll, since a re-run will fail the same way until the base is fixed. |
|
Root cause found for the
|
…raffic (#35166) `GenAIMetricRecorder._common_attributes` dumped the whole `hidden_params` object onto every metric datapoint as one label value. That object is per-request by construction: `response_cost`, `litellm_overhead_time_ms`, `cache_key`, `usage_object` and the provider's `additional_headers` rate-limit counters all move on every call. A unique label value is a new time series, and all six GenAI instruments share those attributes, so one request minted up to six series that would never be written to again That is the steady-state behavior of the feature rather than an abuse case, and it is wrong twice over. Hosted backends bill on series count, so recommending metrics be enabled would have meant a bill proportional to traffic. And a histogram whose every datapoint sits in its own series cannot be aggregated, so the dashboards would have looked populated while answering nothing Both paths now cap their attributes at METRIC_ATTRIBUTE_CEILING, which replaces the failure-only allowlist so the two paths cannot drift. The cap runs before the operator's `otel.attributes` filter, so an operator can narrow it and never widen it back to an unbounded label. Client-supplied and per-request metadata (`requester_metadata`, `spend_logs_metadata`, `user_api_key_end_user_id`, `requester_ip_address`) is metric-ineligible and stays on the span, which already carries it and where cardinality is free. `hidden_params` survives as a label but carries only `model_id` and `api_base`, which are bounded by the router's own deployment list and are the part a per-deployment panel reads Four tests fail against the previous behavior, the load-bearing one being that two requests differing only in per-request fields must land in one series rather than two
3 participants
TLDR
Problem this solves:
gen_ai.client.operation.durationmeasured only surviving trafficHow it solves it:
error.typeRelevant issues
Linear ticket
Resolves LIT-4955
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Live proxy on port 4955, no DB, real OpenAI calls against
gpt-5.5costing real money, plus a second deployment pointed at an unreachableapi_basefor a deterministic upstream failure.OTEL_EXPORTER=consoleso metric datapoints print;OTEL_ENDPOINT=""(exported empty rather than unset, otherwiseload_dotenvrestores the real collector endpoint from.envand the datapoints ship off-box instead of printing).Config used,
lit4955_config.yaml. Theexclude_listis there to keep the console output readable, and it doubles as proof that the cardinality filter cannot striperror.type:Both captures ran these exact commands:
Both runs answered:
The console exporter pretty-prints one JSON document per export, so the datapoints of the last export are read out of the log with:
Before, at
551e5d097c(this branch's merge base). The failure contributed to nothing; every series is the success:After, at
da00073979. The failure adds exactly one datapoint, on the duration histogram only, on its own series:Three things to read off the after capture.
error.typeis present on the failure series and absent on the success series, soerror_type=""still isolates the pre-existing series anderror_type!=""is a failure-rate numerator.error.typesurvived anexclude_listthat stripped six other attributes, because it is stamped after the filter. And no other instrument gained a failure series: token usage, cost and response duration each still carry exactly their one success datapoint, no fabricated zeros.Third capture, at
858f3eab94, for the bounded failure label set. Same proxy, config with theexclude_listremoved so nothing hides the difference, and both requests carrying caller-controlled fields ("user": "end-user-42"in the body and anx-litellm-spend-logs-metadata: {"ticket":"unique-per-request-value"}header). Printing the label names per datapoint ofgen_ai.client.operation.duration:The success series carries the caller's
spend_logs_metadata,requester_metadata, end-user id and thehidden_paramsblob, which is the pre-existing behavior this PR does not change. The failure series carries none of them: seven labels, all fixed enums or operator-provisioned identity, so a caller cannot mint a series per failed request.Type
🐛 Bug Fix
Changes
_record_metricswas called only fromasync_log_success_event, so a failed request contributed to no metric at all. Latency read offgen_ai.client.operation.durationduring an incident was the latency of the traffic that happened to succeed, and since no instrument carried an error dimension, a failure-rate panel or a success/failure split per model could not be written at all.async_log_failure_eventnow records too, through a newGenAIMetricRecorder.record_failure. The success and failure recordings share the existing best-effort wrapper (renamed to_guarded_recordand given a callable, since both need the same "never break the request, surface a bad attribute filter once at ERROR" behavior).Three decisions worth calling out.
One histogram, not two. A failed operation's duration goes into the same
gen_ai.client.operation.durationas a successful one, separated by an attribute rather than by a second instrument. That is what the GenAI semantic conventions specify:error.typeis a conditionally-required attribute on that instrument, present only when the operation ends in error. It keeps litellm's series where a consumer of GenAI telemetry looks for them, it makes the histogram's_countthe request counter for both outcomes (so no separate error counter is needed to express a failure rate), and it stays separable because the attribute is only ever added on the failure path. Pooling them with no attribute would have been the harmful option, since every current dashboard query would silently start including failures with no way to exclude them.error.typecarries the error. The key is the semconv-ownederror.typealready declared insemconv.Errorand already stamped on failure spans, not a new invented name. The value is the mapped provider exception's class name (error_information.error_class, the same value the span uses), falling back to the provider status code, then to the raw exception's class name, then to the semconv_OTHER. Every one of those is bounded. The exception message is unbounded and never becomes a label; it stays on the span and on thegen_ai.client.operation.exceptionevent, where high cardinality is free.error.typeis applied after the cardinality filter, exactly asgen_ai.token.typealready is, so an operator'sotel.attributesinclude or exclude list cannot strip the discriminator and silently merge failures back into the success series.The failure attribute set is a bounded allowlist, not the success set.
_common_attributescarries fields the client supplies, chieflymetadata.requester_metadata, along with per-request values such as thehidden_paramsblob (which holds the provider's response headers). A failed request costs the caller no provider spend, so nothing would rate-limit a caller who put a unique value in a field they control and minted one histogram series per request. A failure datapoint therefore carries exactly:gen_ai.operation.name,gen_ai.system,gen_ai.request.model,gen_ai.framework,metadata.user_api_key_hash,metadata.user_api_key_alias,metadata.user_api_key_team_id,metadata.user_api_key_team_alias,metadata.user_api_key_org_id,metadata.user_api_key_user_id, anderror.typeEvery one is a fixed enum or an operator-provisioned identifier, so the failure series count is bounded by the deployment's own key, team and user count, and the labels still answer the question an operator has during an incident: which team, on which model, is failing and how. Excluded are
metadata.requester_metadata,metadata.spend_logs_metadata,metadata.user_api_key_end_user_id(taken from the request'suserfield),metadata.requester_ip_address, andhidden_params.metadata.user_api_key_user_emailis left out as well; it is bounded, but it is PII duplicating the user id that is already on the series. The operator's ownotel.attributesfilter layers on top of the allowlist, so it can narrow the set further and never widen it.Duration is the only instrument recorded. The other five describe a completed generation. litellm hands the failure callback
response_obj=Noneby construction, so there is no usage to split into input/output tokens and no completion-token count to divide generation time by, and it zeroesresponse_coston failure; recording them would push fabricated zeros into series that dashboards average. Cost and usage recovered from a stream that broke mid-flight are still billed through SpendLogs and still stamped on the span, so nothing a dashboard needs is lost. A synthetic proxy-gate failure log (an auth or rate-limit rejection) records nothing at all, for the same reason it already gets no span: no upstream call happened, so its wall time is not a GenAI operation's duration and would pull the histogram toward the proxy's own latency.MCP is not covered by this.
async_log_failure_eventreturns early once the MCP tool-call ortools/listspan emitters have handled the event, so an MCP request never reaches the recorder on either the success or the failure path. This change gives MCP no error metric, andmcp_errors_totalon Grafana's MCP dashboard is still unbuildable; that is LIT-4951 and is deliberately out of scope here.A sibling PR (#35151, branch
litellm_otel_genai_metric_attributes) is reworking attribute construction in the sameplumbing/metrics.pyand the operation mapping inmodel/semconv.pyfor LIT-4954 and LIT-4959, so expect a small overlap there; this diff was kept minimal for that reason.Final Attestation