You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Improve llm-d-router (EPP) observability documentation and operator-facing guidance so metrics, tracing, and embedded llm-d-kv-cache telemetry are discoverable and consistent with the current implementation.
Concrete deliverables:
Embedded llm-d-kv-cache: when precise-prefix-cache-scorer is enabled with indexerConfig.kvBlockIndexConfig.enableMetrics: true, kvcache_index_* (and related) metrics are registered on the same EPP registry—no separate kv-cache HTTP endpoint is required in standard deployments.
Cross-link related work (implement in separate PRs/issues where noted)
(TBD follow-up, separate issue or sub-task) Add an e2e or integration assertion that kvcache_index_* appears on EPP /metrics when enableMetrics: true and precise-prefix-cache-scorer is configured.
Operators and SREs cannot easily answer: what to scrape, which metric names to use in dashboards/alerts, and whether llm-d-kv-cache (separate repo) requires its own /metrics port when used via EPP.
Today, sample configs often set enableMetrics: false (e.g. deploy/config/sim-epp-kvcache-config.yaml), so kvcache_* series are easy to miss even though the integration path (shared ctrlmetrics.Registry) already exists in code.
Reduces duplicate work (e.g. opening a standalone kv-cache metrics endpoint) and aligns monitoring with the EPP Helm ServiceMonitor pattern (router.monitoring.prometheus).
What would you like to be added:
Improve llm-d-router (EPP) observability documentation and operator-facing guidance so metrics, tracing, and embedded llm-d-kv-cache telemetry are discoverable and consistent with the current implementation.
Concrete deliverables:
docs/metrics.mdllm_d_router_epp_*(introduced in Rename metrics and introduce a single subsystem for epp #851 / PR Add new metrics with single subsystem #1071).inference_*,llm_d_inference_scheduler_*) and point to replacements.:9090/metrics(serveMetrics/ controller-runtime metrics server).metricsDataSource).precise-prefix-cache-scoreris enabled withindexerConfig.kvBlockIndexConfig.enableMetrics: true,kvcache_index_*(and related) metrics are registered on the same EPP registry—no separate kv-cache HTTP endpoint is required in standard deployments.kvcache_index_*appears on EPP/metricswhenenableMetrics: trueandprecise-prefix-cache-scoreris configured.Why is this needed:
docs/metrics.mdstill states all metrics use thellm_d_inference_schedulersubsystem, which is outdated after the llm-d Router rename (docs: renameinference schedulertollm-d Routerand fix typos #1233) and metrics migration (Rename metrics and introduce a single subsystem for epp #851)./metricsport when used via EPP.enableMetrics: false(e.g.deploy/config/sim-epp-kvcache-config.yaml), sokvcache_*series are easy to miss even though the integration path (sharedctrlmetrics.Registry) already exists in code.router.monitoring.prometheus).