Skip to content

[Observability] improve EPP observability guide and metrics documentation #1394

Description

@gyliu513

What would you like to be added:

Improve llm-d-router (EPP) observability documentation and operator-facing guidance so metrics, tracing, and embedded llm-d-kv-cache telemetry are discoverable and consistent with the current implementation.
Concrete deliverables:

  1. Update docs/metrics.md
  2. Document scrape topology for operators
    • EPP Prometheus endpoint: default :9090/metrics (serveMetrics / controller-runtime metrics server).
    • Model-server metrics scrape path (data layer / Helm metricsDataSource).
    • Embedded llm-d-kv-cache: when precise-prefix-cache-scorer is enabled with indexerConfig.kvBlockIndexConfig.enableMetrics: true, kvcache_index_* (and related) metrics are registered on the same EPP registry—no separate kv-cache HTTP endpoint is required in standard deployments.
  3. Cross-link related work (implement in separate PRs/issues where noted)
  4. (TBD follow-up, separate issue or sub-task) Add an e2e or integration assertion that kvcache_index_* appears on EPP /metrics when enableMetrics: true and precise-prefix-cache-scorer is configured.

Why is this needed:

  • docs/metrics.md still states all metrics use the llm_d_inference_scheduler subsystem, which is outdated after the llm-d Router rename (docs: rename inference scheduler to llm-d Router and fix typos #1233) and metrics migration (Rename metrics and introduce a single subsystem for epp #851).
  • Operators and SREs cannot easily answer: what to scrape, which metric names to use in dashboards/alerts, and whether llm-d-kv-cache (separate repo) requires its own /metrics port when used via EPP.
  • Today, sample configs often set enableMetrics: false (e.g. deploy/config/sim-epp-kvcache-config.yaml), so kvcache_* series are easy to miss even though the integration path (shared ctrlmetrics.Registry) already exists in code.
  • Reduces duplicate work (e.g. opening a standalone kv-cache metrics endpoint) and aligns monitoring with the EPP Helm ServiceMonitor pattern (router.monitoring.prometheus).

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageIndicates an issue or PR lacks a triage label and requires one.observability

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions