What happened:
- The
flow_control_pool_saturation help text claims a 0.0-1.0 range, but both detectors legitimately return values above 1: the utilization detector's per-pod score is max(qRatio, kvRatio) (saturationdetector/utilization/detector.go:128-133) and the concurrency detector returns totalInflight/totalCapacity (concurrency/detector.go:157-161). Values above 1 are meaningful -- the magnitude of oversubscription past the set point -- and dispatch gating is unaffected (usage-limit ceilings are <= 1.0, so anything >= 1 gates regardless). The bug is the help text, not the values: operators reading "1.0 = fully saturated" cannot interpret a 1.7 reading, and dashboards built on the documented range mis-render.
- The default (utilization) detector scores any pod 1.0 when its metrics are nil or older than
MetricsStalenessThreshold (utilization/detector.go:123-126), and both detectors return 1.0 for an empty endpoint list. At saturation 1.0 the dispatch cycle halts entirely (controller/internal/processor.go:377-383). A fleet-wide failure of the EPP-to-model-server metrics collection (the datalayer collectors polling model-server metrics -- dependent on pod networking, ports, TLS configuration, and auth) -- which legacy admission survives by shedding only sheddable traffic -- therefore becomes a 100% dispatch outage with no distinct signal. Note stale pods score exactly 1.0 while genuine oversubscription typically reads above 1, one more reason the two must be distinguishable.
What you expected to happen:
The gauge stays within its documented range, and saturation caused by missing/stale data is distinguishable (log + metric) from genuine saturation, so operators can tell a scrape outage from real overload.
How to reproduce it (as minimally and precisely as possible):
Enable flow control with the default detector; break metrics scraping for the model-server pods (e.g. bad TLS config). Observe flow_control_pool_saturation pinned at 1.0, dispatch halted, and no log or metric attributing the halt to staleness.
Anything else we need to know?:
Part of #1187; blocks enabling the gate by default.
Proposed fix: emit a staleness signal (rate-limited WARN plus a metric such as a stale-endpoints gauge -- keep pool_saturation itself label-stable); fix the help text to document the actual semantics (1.0 is the gating set point; values above 1 indicate oversubscription magnitude; empty-pool and stale-metrics conditions read as 1.0, fail-closed); document the operational dependency of dispatch on model-server metrics collection health; record a decision on fail-open vs fail-closed for stale metrics on a NON-empty pool (proposal: keep fail-closed, make it loud; empty-pool fail-closed is correct and already test-covered). Deliberately NOT clamping: >1 is the demand-magnitude half of the autoscaling composite (paired with the queue utilization gauges).
Release note: new staleness signal; pool_saturation help text corrected (values above 1.0 indicate oversubscription).
Environment:
- Kubernetes version: n/a (code analysis)
- llm-d-router version: main @
6cc1dac7
- Cloud provider or hardware configuration: n/a
- Install tools: n/a
- Others: n/a
What happened:
flow_control_pool_saturationhelp text claims a 0.0-1.0 range, but both detectors legitimately return values above 1: the utilization detector's per-pod score ismax(qRatio, kvRatio)(saturationdetector/utilization/detector.go:128-133) and the concurrency detector returnstotalInflight/totalCapacity(concurrency/detector.go:157-161). Values above 1 are meaningful -- the magnitude of oversubscription past the set point -- and dispatch gating is unaffected (usage-limit ceilings are <= 1.0, so anything >= 1 gates regardless). The bug is the help text, not the values: operators reading "1.0 = fully saturated" cannot interpret a 1.7 reading, and dashboards built on the documented range mis-render.MetricsStalenessThreshold(utilization/detector.go:123-126), and both detectors return 1.0 for an empty endpoint list. At saturation 1.0 the dispatch cycle halts entirely (controller/internal/processor.go:377-383). A fleet-wide failure of the EPP-to-model-server metrics collection (the datalayer collectors polling model-server metrics -- dependent on pod networking, ports, TLS configuration, and auth) -- which legacy admission survives by shedding only sheddable traffic -- therefore becomes a 100% dispatch outage with no distinct signal. Note stale pods score exactly 1.0 while genuine oversubscription typically reads above 1, one more reason the two must be distinguishable.What you expected to happen:
The gauge stays within its documented range, and saturation caused by missing/stale data is distinguishable (log + metric) from genuine saturation, so operators can tell a scrape outage from real overload.
How to reproduce it (as minimally and precisely as possible):
Enable flow control with the default detector; break metrics scraping for the model-server pods (e.g. bad TLS config). Observe
flow_control_pool_saturationpinned at 1.0, dispatch halted, and no log or metric attributing the halt to staleness.Anything else we need to know?:
Part of #1187; blocks enabling the gate by default.
Proposed fix: emit a staleness signal (rate-limited WARN plus a metric such as a stale-endpoints gauge -- keep
pool_saturationitself label-stable); fix the help text to document the actual semantics (1.0 is the gating set point; values above 1 indicate oversubscription magnitude; empty-pool and stale-metrics conditions read as 1.0, fail-closed); document the operational dependency of dispatch on model-server metrics collection health; record a decision on fail-open vs fail-closed for stale metrics on a NON-empty pool (proposal: keep fail-closed, make it loud; empty-pool fail-closed is correct and already test-covered). Deliberately NOT clamping: >1 is the demand-magnitude half of the autoscaling composite (paired with the queue utilization gauges).Release note: new staleness signal; pool_saturation help text corrected (values above 1.0 indicate oversubscription).
Environment:
6cc1dac7