Is this related to a problem? Please describe.
#1272 added a configurable startup warmup (--readiness-warmup-ms / STARGATE_READINESS_WARMUP_MS, default 60000) that keeps /readyz unavailable for a fixed window so Pylons have time to register and establish reverse tunnels before a new Stargate replica receives API gateway traffic. That fixed window is a worst-case bound — it always waits out the full duration even when Pylon registration finishes and stabilizes within a few seconds, which unnecessarily slows scale-up and rolling restarts in the common case.
Describe the solution you'd like
Let the warmup complete early, once it's clear the replica is already healthy, instead of always waiting out the fixed timer:
- Sample
StargateState's total active backend count once per second (sample interval configurable).
- Once that count is greater than zero and has stayed unchanged for N consecutive samples (default 5 seconds), treat the replica as ready immediately rather than waiting for the remainder of the fixed warmup window.
- Keep the existing fixed warmup as the upper bound / fallback: if the backend count never stabilizes (or stays at zero), readiness still flips at the current timeout, so today's worst-case behavior is preserved when nothing improves.
- Make both the sample interval and the stabilization window configurable (new CLI flags/env vars, and Helm chart values for
llm-request-router), independent of the existing readiness-warmup-ms knob.
Naming note: please don't describe this as "exiting" or an "early exit" from warmup in flags, code, or docs — that phrasing reads like a failure/abort path. This is a success path: the replica became healthy sooner than the worst-case timeout allows. Prefer language like "early readiness", "warmup stabilization detection", or "fast-path readiness promotion".
Describe alternatives you've considered
- Just lowering the default
readiness-warmup-ms — doesn't work well since it's a fixed worst-case bound and would either be too aggressive for replicas that register slowly, or still too conservative for replicas that register fast.
- Polling faster than once per second — the cadence should be configurable, but a 1s default keeps sampling cheap and avoids reacting to single-sample noise.
Additional context
By submitting this issue, you agree to follow our code of conduct and our contributing guidelines.
Is this related to a problem? Please describe.
#1272 added a configurable startup warmup (
--readiness-warmup-ms/STARGATE_READINESS_WARMUP_MS, default60000) that keeps/readyzunavailable for a fixed window so Pylons have time to register and establish reverse tunnels before a new Stargate replica receives API gateway traffic. That fixed window is a worst-case bound — it always waits out the full duration even when Pylon registration finishes and stabilizes within a few seconds, which unnecessarily slows scale-up and rolling restarts in the common case.Describe the solution you'd like
Let the warmup complete early, once it's clear the replica is already healthy, instead of always waiting out the fixed timer:
StargateState's total active backend count once per second (sample interval configurable).llm-request-router), independent of the existingreadiness-warmup-msknob.Naming note: please don't describe this as "exiting" or an "early exit" from warmup in flags, code, or docs — that phrasing reads like a failure/abort path. This is a success path: the replica became healthy sooner than the worst-case timeout allows. Prefer language like "early readiness", "warmup stabilization detection", or "fast-path readiness promotion".
Describe alternatives you've considered
readiness-warmup-ms— doesn't work well since it's a fixed worst-case bound and would either be too aggressive for replicas that register slowly, or still too conservative for replicas that register fast.Additional context
/readyzunavailable for a configurable startup warmup that defaults to 60 seconds"), which itself closed Delay Stargate readiness while Pylons connect #1271.StargateStateinsrc/libraries/rust/stargate/crates/stargate/src/routing_state/mod.rs, which needs to expose a total active-backend count for the stabilization sampler to read.readiness_*unit tests,health_lifecycleintegration test instargate_integration).By submitting this issue, you agree to follow our code of conduct and our contributing guidelines.