Skip to content

TRT-LLM additional metric children are absent until their first event #18169

Open

Description

@yifjiang

Problem

AdditionalMetricsCollector constructs labeled Prometheus families but does not create their children during initialization. A newly started TRT-LLM worker therefore omits supported counters such as trtllm_num_aborted_requests_total and trtllm_request_type_structured_output_total until each event first occurs. Dashboards and absent() checks cannot distinguish a supported zero from missing instrumentation.

Expected behavior

Create the already-known bounded label combination for each additional metric at collector startup. Keep all names, labels, event accounting, and engine pass-through metrics unchanged. A fresh registry should export zero-valued counters/gauges and zero-observation histograms before any request; subsequent events should accumulate normally.

This is a registration fix under the existing metrics guidelines (R1/L6), separate from cache-reuse accounting and new router funnel instrumentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions