diff --git a/docs/metrics.md b/docs/metrics.md index ba7106d732..fa84aafadf 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -1,165 +1,319 @@ # Metrics -The `llm-d-router` exposes the following Prometheus metrics to monitor its behavior and performance, particularly concerning Encode/Prefill/Decode disaggregation. +The `llm-d-router` Endpoint Picker (EPP) exposes Prometheus metrics to monitor its behavior and +performance. These are in addition to the Inference Gateway metrics; for how to view metrics, see the +Gateway API Inference Extension [metrics and observability guide](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/site-src/guides/metrics-and-observability.md). + +## Subsystems and naming -All metrics are in the `llm_d_inference_scheduler` subsystem. +A metric's full Prometheus name is `_`. The EPP uses two current subsystems: + +| Prefix | Scope | +|---|---| +| `llm_d_epp_` | Canonical, EPP-wide: request/latency, pool, scheduler, plugin, data layer, flow control, disaggregation, ext_proc, prefix indexer, multimodal, program-aware fairness, and predicted-latency metrics. | +| `llm_d_router_epp_` | The embedded llm-d-kv-cache metrics only (see [Embedded llm-d-kv-cache metrics](#embedded-llm-d-kv-cache-metrics)). | -## Scrape and see the metric +Earlier releases emitted metrics under `llm_d_inference_scheduler_`, `inference_objective_`, +`inference_pool_`, `inference_extension_`, and `kvcache_`. Those prefixes are **deprecated** but +still emitted: each recorder that has a deprecated predecessor writes both the legacy series and its +current twin (dual emission), so existing dashboards keep working during migration. See [Deprecated series](#deprecated-series). -Metrics defined by llm-d Router are in addition to Inference Gateway metrics. For more details of seeing metrics, see the [metrics and observability section](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/site-src/guides/metrics-and-observability.md). +## Scrape topology -## Metrics Details +### EPP metrics endpoint -### `disagg_decision_total` +The EPP serves its metrics from the controller-runtime metrics registry at `/metrics` on the metrics +port (default `9090`, configurable with `--metrics-port`). Every metric on this page is exposed on +this single endpoint. Metric authentication and TLS are configurable via `--metrics-endpoint-auth`. -* **Type:** Counter -* **Labels:** - * `model_name`: string (the target model name, or "unknown" if empty) - * `decision_type`: string - one of: - * `decode-only` - the request used the decode-only path (no disaggregation) - * `prefill-decode` - the request was split into prefill and decode stages (P/D or EP/D) - * `encode-decode` - the request used encode disaggregation with local prefill+decode (E/PD) - * `encode-prefill-decode` - the request used the full three-stage pipeline (E/P/D) -* **Release Stage:** ALPHA -* **Description:** Counts the number of requests processed, broken down by the disaggregation routing decision. -* **Usage:** Provides a high-level view of how many requests are utilizing each disaggregation topology. -* **Actionability:** - * Monitor the distribution across decision types to understand engagement rates for each disaggregation mode. - * Sudden changes in ratios might indicate configuration issues, changes in workload patterns, or problems with the decision logic. +### Model-server metrics (data layer) + +The `metrics-data-source` plugin scrapes each model-server pod's own `/metrics` endpoint (path +configurable) and feeds the results into the data layer for scorers. These are the model server's +metrics, distinct from the EPP metrics above; scrape the pods directly to collect them. + +### Embedded llm-d-kv-cache metrics + +When the precise prefix cache is enabled (`precise-prefix-cache-producer` / +`precise-prefix-cache-scorer`) with `indexerConfig.kvBlockIndexConfig.enableMetrics: true`, the +embedded llm-d-kv-cache index registers its `llm_d_router_epp_kv_cache_*` metrics on the **same** +controller-runtime registry the EPP `/metrics` endpoint already serves. No separate kv-cache HTTP +endpoint or scrape target is required. + +`enableMetrics` defaults to `false`, and shipped sample configs (e.g. +`deploy/config/sim-epp-kvcache-config.yaml`) leave it off, so these series are absent until an +operator opts in. + +## Deprecated series + +All deprecated series are still emitted as back-compat aliases alongside their current twins. Prefer +the current `llm_d_epp_*` (or `llm_d_router_epp_*`) names in new dashboards and alerts. + +| Deprecated prefix | Current replacement | +|---|---| +| `llm_d_inference_scheduler_*` (disagg, data-layer errors) | `llm_d_epp_*` | +| `inference_objective_*` (request/latency, predicted latency) | `llm_d_epp_*` | +| `inference_pool_*` (pool averages, queue size) | `llm_d_epp_*` | +| `inference_extension_*` (scheduler, plugin, info, flow control, prefix indexer) | `llm_d_epp_*` | +| `kvcache_index_*`, `kvcache_kvevents_*` | `llm_d_router_epp_kv_cache_*` | + +## Metrics catalog + +Names below omit the subsystem prefix. Unless a section states otherwise, the prefix is `llm_d_epp_` +and the release stage is ALPHA. Request and latency metrics share the label set +`{model_name, target_model_name, fairness_id, priority}`. + +### Request and latency + +| Name | Type | Notes | +|---|---|---| +| `request_total` | Counter | Total requests. | +| `request_error_total` | Counter | Errored requests; adds label `error_code`. | +| `request_duration_seconds` | Histogram | End-to-end request latency. | +| `request_size_bytes` | Histogram | Request body size. | +| `response_size_bytes` | Histogram | Response body size. | +| `request_input_tokens` | Histogram | Input token count. | +| `request_output_tokens` | Histogram | Output token count. | +| `request_cached_tokens` | Histogram | Prompt tokens served from cache. | +| `request_running` | Gauge | Requests currently in flight. | +| `request_ntpot_seconds` | Histogram | Normalized time per output token. | +| `request_ttft_seconds` | Histogram | Time to first token; adds label `streaming`. | +| `request_streaming_tpot_seconds` | Histogram | Time per output token (streaming). | +| `request_streaming_itl_seconds` | Histogram | Inter-token latency (streaming). | + +### Inference pool + +Label `{name}` (the pool name). + +| Name | Type | Notes | +|---|---|---| +| `average_kv_cache_utilization` | Gauge | Mean KV-cache utilization across the pool. | +| `average_queue_size` | Gauge | Mean queue depth. | +| `average_running_requests` | Gauge | Mean in-flight requests. | +| `std_dev_kv_cache_utilization` | Gauge | Spread of KV-cache utilization. | +| `std_dev_queue_size` | Gauge | Spread of queue depth. | +| `std_dev_running_requests` | Gauge | Spread of in-flight requests. | +| `ready_endpoints` | Gauge | Ready endpoints in the pool. | +| `per_endpoint_queue_size` | Gauge | Per-endpoint queue depth; labels `{name, model_server_endpoint}`. | + +### Scheduler + +| Name | Type | Notes | +|---|---|---| +| `scheduler_e2e_duration_seconds` | Histogram | End-to-end scheduling latency. | +| `scheduler_attempts_total` | Counter | Scheduling attempts; labels `{status, target_model_name, endpoint_name, namespace, port}`. | + +### Plugin, info, and model rewrite + +| Name | Type | Notes | +|---|---|---| +| `plugin_duration_seconds` | Histogram | Per-plugin execution time; labels `{extension_point, plugin_type, plugin_name}`. | +| `info` | Gauge | Build info; labels `{commit, build_ref}`. | +| `model_rewrite_decisions_total` | Counter | Model-rewrite decisions; labels `{model_rewrite_name, model_name, target_model}`. | -### `pd_decision_total` (deprecated) +### Data layer errors -> **Deprecated:** Use `disagg_decision_total` instead. +| Name | Type | Notes | +|---|---|---| +| `datalayer_poll_errors_total` | Counter | Data-source poll failures; label `{source_type}`. | +| `datalayer_extract_errors_total` | Counter | Extractor failures; labels `{source_type, extractor_type}`. | + +### Prefix cache indexer (approximate) + +Labels `{plugin_name, plugin_type}`. + +| Name | Type | Notes | +|---|---|---| +| `prefix_indexer_size` | Gauge | Entries in the approximate prefix index. | +| `prefix_indexer_hit_ratio` | Histogram | Prefix-match hit ratio. | +| `prefix_indexer_hit_bytes` | Histogram | Bytes matched per lookup. | + +### Multimodal encoder cache + +| Name | Type | Notes | +|---|---|---| +| `encoder_cache_queries_total` | Counter | Encoder-cache lookups; labels `{plugin_type, plugin_name, modality}`. | +| `encoder_cache_hits_total` | Counter | Encoder-cache hits; labels `{plugin_type, plugin_name, pod, modality}`. | +| `encoder_cache_hit_ratio` | Histogram | Hit ratio; labels `{plugin_type, plugin_name}`. | + +### Program-aware fairness + +| Name | Type | Notes | +|---|---|---| +| `program_aware_jains_fairness_index` | Gauge | Jain's fairness index across programs. | +| `program_aware_avg_wait_time_milliseconds` | Gauge | Mean wait time; label `{program_id}`. | +| `program_aware_attained_service_tokens` | Gauge | Attained service; label `{program_id}`. | + +### Predicted latency and SLO + +Labels `{plugin_name, plugin_type, model_name, target_model_name}` (some add `type`). + +| Name | Type | Notes | +|---|---|---| +| `inference_request_metric` | Gauge | Observed request metric; adds label `type`. | +| `request_predicted_ttft_seconds` | Histogram | Predicted time to first token. | +| `request_ttft_prediction_duration_seconds` | Histogram | Time spent computing the TTFT prediction. | +| `request_predicted_tpot_seconds` | Histogram | Predicted time per output token. | +| `request_tpot_prediction_duration_seconds` | Histogram | Time spent computing the TPOT prediction. | +| `request_slo_violation_total` | Counter | SLO violations; adds label `type`. | + +### Disaggregation + +Both metrics carry labels `{plugin_name, plugin_type, model_name, decision_type}`. The current names +are under `llm_d_epp_`; each has a deprecated `llm_d_inference_scheduler_*` twin. + +#### `disagg_decision_total` * **Type:** Counter * **Labels:** - * `model_name`: string (the target model name, or "unknown" if empty) - * `decision_type`: string ("decode-only" or "prefill-decode") -* **Release Stage:** ALPHA -* **Description:** Counts the number of requests processed, broken down by the Prefill/Decode disaggregation decision. This metric only covers P/D disaggregation and does not account for encode disaggregation. + * `model_name`: the target model name, or "unknown" if empty + * `decision_type`: one of + * `decode-only` - decode-only path (no disaggregation) + * `prefill-decode` - split into prefill and decode stages (P/D or EP/D) + * `encode-decode` - encode disaggregation with local prefill+decode (E/PD) + * `encode-prefill-decode` - full three-stage pipeline (E/P/D) +* **Description:** Counts requests processed, broken down by the disaggregation routing decision. +* **Actionability:** Monitor the distribution across decision types to understand engagement per + disaggregation mode. Sudden ratio changes may indicate configuration issues, workload shifts, or + problems in the decision logic. + +#### `pd_decision_total` (deprecated handler) + +> Prefer `disagg_decision_total`. This metric is maintained for the deprecated `pd-profile-handler` +> and covers only P/D disaggregation, not encode disaggregation. -> [!NOTE] -> This metric is maintained for backward compatibility with the deprecated -> `pd-profile-handler`. New deployments should use `disagg_decision_total`. +* **Type:** Counter +* **Labels:** `model_name`; `decision_type` (`decode-only` or `prefill-decode`). +* **Description:** Counts requests by the Prefill/Decode disaggregation decision. -## Flow Control Metrics +### Flow control -Exposed when the `flowControl` feature gate is enabled. All carry the `llm_d_epp_` prefix. +Exposed when the `flowControl` feature gate is enabled. -### `flow_control_request_queue_duration_seconds` +#### `flow_control_request_queue_duration_seconds` * **Type:** Histogram -* **Labels:** - * `fairness_id`: string (the tenant or flow identifier for fairness rotation) - * `priority`: string (the priority band, e.g., "0", "10") - * `outcome`: string (`Dispatched`, `RejectedCapacity`, `RejectedOther`, `EvictedTTL`, `EvictedContextCancelled`, `EvictedOther`) - * `inference_pool`: string - * `model_name`: string - * `target_model_name`: string -* **Release Stage:** ALPHA -* **Description:** Total time a request spends in the Flow Control layer, from enqueue to final outcome. -* **Usage:** Primary latency signal for flow control. Rising p99 indicates backends are saturated or capacity limits are too tight. - -### `flow_control_dispatch_cycle_duration_seconds` +* **Labels:** `fairness_id`, `priority`, `outcome` (`Dispatched`, `RejectedCapacity`, + `RejectedOther`, `EvictedTTL`, `EvictedContextCancelled`, `EvictedOther`), `inference_pool`, + `model_name`, `target_model_name` +* **Description:** Total time a request spends in the Flow Control layer, from enqueue to final + outcome. +* **Usage:** Primary latency signal for flow control. Rising p99 indicates backends are saturated + or capacity limits are too tight. + +#### `flow_control_dispatch_cycle_duration_seconds` * **Type:** Histogram -* **Release Stage:** ALPHA * **Description:** Time taken for each internal dispatch cycle. -* **Usage:** Measures the overhead of the dispatch loop itself. Rising values indicate increasing cost per cycle from saturation detection, priority band iteration, or fairness evaluation. +* **Usage:** Measures the overhead of the dispatch loop itself. Rising values indicate increasing + cost per cycle from saturation detection, priority band iteration, or fairness evaluation. -### `flow_control_request_enqueue_duration_seconds` +#### `flow_control_request_enqueue_duration_seconds` * **Type:** Histogram -* **Labels:** - * `fairness_id`: string (the tenant or flow identifier) - * `priority`: string (the priority band) - * `outcome`: string -* **Release Stage:** ALPHA +* **Labels:** `fairness_id`, `priority`, `outcome` * **Description:** Time taken to enqueue a request into the Flow Control layer. * **Usage:** Measures the time spent in capacity checks and queue insertion within the processor. -### `flow_control_queue_size` +#### `flow_control_queue_size` * **Type:** Gauge -* **Labels:** - * `fairness_id`: string (the tenant or flow identifier) - * `priority`: string (the priority band) - * `inference_pool`: string - * `model_name`: string - * `target_model_name`: string -* **Release Stage:** ALPHA +* **Labels:** `fairness_id`, `priority`, `inference_pool`, `model_name`, `target_model_name` * **Description:** Current number of requests actively held in the Flow Control queue. -* **Usage:** Tracks queue depth per priority band and tenant. A steadily growing value indicates the dispatch rate is lower than the arrival rate. +* **Usage:** Tracks queue depth per priority band and tenant. A steadily growing value indicates + the dispatch rate is lower than the arrival rate. -### `flow_control_queue_bytes` +#### `flow_control_queue_bytes` * **Type:** Gauge -* **Labels:** - * `fairness_id`: string (the tenant or flow identifier) - * `priority`: string (the priority band) - * `inference_pool`: string - * `model_name`: string - * `target_model_name`: string -* **Release Stage:** ALPHA +* **Labels:** `fairness_id`, `priority`, `inference_pool`, `model_name`, `target_model_name` * **Description:** Current total size in bytes of requests actively held in the Flow Control queue. -* **Usage:** Tracks memory pressure from queued requests. Compare against the configured `maxBytes` capacity to gauge how close a band is to rejecting new requests. +* **Usage:** Tracks memory pressure from queued requests. Compare against the configured `maxBytes` + capacity to gauge how close a band is to rejecting new requests. -### `flow_control_pool_saturation` +#### `flow_control_pool_saturation` * **Type:** Gauge -* **Labels:** - * `inference_pool`: string -* **Release Stage:** ALPHA -* **Description:** Current saturation level of the inference pool (0.0 = empty, 1.0 = fully saturated). -* **Usage:** When saturation reaches the usage limit threshold, the dispatch cycle skips dispatching and requests remain queued. Sustained 1.0 indicates all backends are at capacity. +* **Labels:** `inference_pool` +* **Description:** Current saturation level of the inference pool (0.0 = empty, 1.0 = fully + saturated). +* **Usage:** When saturation reaches the usage limit threshold, the dispatch cycle skips + dispatching and requests remain queued. Sustained 1.0 indicates all backends are at capacity. -### `flow_control_requests_total` +#### `flow_control_requests_total` * **Type:** Counter * **Labels:** - * `outcome`: string — the terminal outcome of the request. One of: - * `Dispatched` — request was forwarded to a backend - * `RejectedCapacity` — request was rejected because the queue was at capacity - * `RejectedNoEndpoints` — request was rejected at the capacity boundary while the candidate pool had no endpoints (surfaces as HTTP 503 rather than 429) - * `RejectedOther` — request was rejected for another reason (e.g., controller shutdown) - * `EvictedTTL` — request exceeded its time-to-live while waiting in the queue - * `EvictedContextCancelled` — client disconnected before the request was dispatched - * `EvictedOther` — request was evicted for another reason - * `priority`: string (the priority band, e.g., `"0"`, `"10"`) - * `inference_pool`: string -* **Release Stage:** ALPHA -* **Description:** Total number of requests processed by the Flow Control layer, incremented once per request after its terminal outcome is determined. -* **Usage:** Provides a direct signal for rejection and eviction rates without log parsing. Unlike `flow_control_request_queue_duration_seconds_count`, this counter also captures controller-level early rejections where no queue item is created (e.g., rejection during controller shutdown), covering cases the histogram misses. + * `outcome`: terminal outcome, one of `Dispatched`, `RejectedCapacity`, `RejectedNoEndpoints` + (candidate pool had no endpoints at the capacity boundary; surfaces as HTTP 503 rather than + 429), `RejectedOther`, `EvictedTTL`, `EvictedContextCancelled`, `EvictedOther` + * `priority`, `inference_pool` +* **Description:** Total requests processed by the Flow Control layer, incremented once per request + after its terminal outcome is determined. +* **Usage:** Direct signal for rejection and eviction rates without log parsing. Unlike + `flow_control_request_queue_duration_seconds_count`, this counter also captures controller-level + early rejections where no queue item is created (e.g. rejection during controller shutdown). * **Actionability:** - * A rising rate of `outcome="RejectedCapacity"` indicates the queue capacity limits are too tight or backends are persistently saturated — consider tuning `maxBytes`/`maxRequests` or scaling backends. - * A rising rate of `outcome="RejectedNoEndpoints"` indicates the inference pool has scaled to zero or all endpoints are unregistered — investigate pool health and scaling configuration. - * A rising rate of `outcome="EvictedTTL"` indicates requests are waiting longer than their TTL allows — investigate backend throughput or tighten admission. - * `outcome="Dispatched"` is the healthy baseline; compare it against total request rate to derive the acceptance ratio. + * Rising `outcome="RejectedCapacity"`: queue limits too tight or backends persistently + saturated - tune `maxBytes`/`maxRequests` or scale backends. + * Rising `outcome="RejectedNoEndpoints"`: the pool scaled to zero or all endpoints unregistered + - investigate pool health and scaling. + * Rising `outcome="EvictedTTL"`: requests waiting longer than their TTL - investigate backend + throughput or tighten admission. + * `outcome="Dispatched"` is the healthy baseline; compare against total request rate for the + acceptance ratio. +### ext_proc streams -## Opt-in ext_proc Stream Metrics +Three metrics covering the ext_proc gRPC stream lifecycle. Disabled by default; enable with +`--enable-grpc-stream-metrics`. -Three metrics covering ext_proc gRPC stream lifecycle. Disabled by default; enable with `--enable-grpc-stream-metrics`. These metrics are emitted under the `llm_d_epp_` prefix (separate from `llm_d_inference_scheduler_*`). - -### `extproc_streams_inflight` +#### `extproc_streams_inflight` * **Type:** Gauge -* **Release Stage:** ALPHA * **Description:** Number of ext_proc gRPC streams currently open. -* **Usage:** Sized at one stream per Envoy worker per EPP backend. A persistent increase under steady load indicates streams are being opened faster than they close. +* **Usage:** Sized at one stream per Envoy worker per EPP backend. A persistent increase under + steady load indicates streams are being opened faster than they close. -### `extproc_stream_duration_seconds` +#### `extproc_stream_duration_seconds` * **Type:** Histogram -* **Release Stage:** ALPHA * **Description:** Duration an ext_proc gRPC stream stays open, in seconds. -* **Usage:** Long-lived streams are normal; the histogram surfaces the distribution. A sudden shift toward short durations can indicate Envoy reconnecting due to handler errors. +* **Usage:** Long-lived streams are normal; the histogram surfaces the distribution. A sudden shift + toward short durations can indicate Envoy reconnecting due to handler errors. -### `extproc_streams_total` +#### `extproc_streams_total` * **Type:** Counter -* **Labels:** - * `code`: string — the gRPC status code at stream close (`OK`, `Canceled`, `DeadlineExceeded`, `Internal`, ...). Bare `context.Canceled` and `context.DeadlineExceeded` are classified to their canonical codes rather than collapsing into `Unknown`. -* **Release Stage:** ALPHA +* **Labels:** `code` - the gRPC status code at stream close (`OK`, `Canceled`, `DeadlineExceeded`, + `Internal`, ...). Bare `context.Canceled` and `context.DeadlineExceeded` are classified to their + canonical codes rather than collapsing into `Unknown`. * **Description:** Total ext_proc gRPC streams completed, by gRPC status code. -* **Usage:** Rate of `code="OK"` is the healthy stream-completion rate. A rising rate of `code="Internal"` or `code="Unknown"` indicates handler errors. `code="Canceled"` is expected on Envoy restarts and rolling EPP updates. +* **Usage:** Rate of `code="OK"` is the healthy completion rate. Rising `code="Internal"` or + `code="Unknown"` indicates handler errors. `code="Canceled"` is expected on Envoy restarts and + rolling EPP updates. + +### KV-cache index + +Prefix `llm_d_router_epp_`. Registered only when the embedded llm-d-kv-cache metrics are enabled (see +[Embedded llm-d-kv-cache metrics](#embedded-llm-d-kv-cache-metrics)). Unlabeled. + +| Name | Type | Notes | +|---|---|---| +| `kv_cache_index_admissions_total` | Counter | Blocks admitted to the index. | +| `kv_cache_index_evictions_total` | Counter | Blocks evicted from the index. | +| `kv_cache_index_lookup_requests_total` | Counter | Index lookups performed. | +| `kv_cache_index_lookup_hits_total` | Counter | Lookups that matched at least one block. | +| `kv_cache_index_max_pod_hit_count_total` | Counter | Best per-pod hit count observed per lookup. | +| `kv_cache_index_lookup_latency_seconds` | Histogram | Index lookup latency. | +| `kv_cache_events_dedup_removed_hashes_suppressed_total` | Counter | Deduplicated removal hashes suppressed. | +| `kv_cache_events_dedup_removed_hashes_forwarded_total` | Counter | Deduplicated removal hashes forwarded. | + +## Related work + +Broader observability work tracked separately (not part of this document): + +- Metrics naming / plugin labels: [#1243](https://github.com/llm-d/llm-d-router/issues/1243) +- Deprecate/remove legacy metrics: [#1070](https://github.com/llm-d/llm-d-router/issues/1070), [#962](https://github.com/llm-d/llm-d-router/issues/962) +- EPP operations guide: [#1291](https://github.com/llm-d/llm-d-router/issues/1291) +- E2E metrics stability: [#1192](https://github.com/llm-d/llm-d-router/issues/1192)