feat: enable the flowControl feature gate by default#2180
Open
LukeAVanDrie wants to merge 4 commits into
Open
Conversation
Flow control becomes the default admission path. Under saturation, requests now queue in the EPP and dispatch by priority as capacity frees, rather than piling into each model server's local queue; queue wait is bounded by a TTL. Sheddable (negative-priority) requests buffer under the same TTL and capacity rules instead of being rejected immediately. Opt out with featureGates: ["flowControl=false"]. - Flip the registered default for the flowControl gate to true and drop the "experimental" wording from the admission-control init logs. - Warn when a flowControl config section is present while the gate is explicitly disabled; everything in it except saturationDetector (which the legacy admission path also uses) is silently ignored otherwise. - Note that ENABLE_EXPERIMENTAL_FLOW_CONTROL_LAYER is now a no-op in its deprecation log and comment (removal tracked with llm-d#2065). - Flip the loader tests' registered gate default and pin the hermetic "do not shed" cases to the explicit flowControl=false opt-out, which they were implicitly relying on; keep the lora-affinity case on the default (gate-on) path by using pod queue depths below the saturation threshold. - Document the default, the opt-out, the metrics scrape health dependency, and the tuning knobs in the flow control README and metrics docs. Part of llm-d#1187. Fixes llm-d#2104. Signed-off-by: Luke Van Drie <lukevandrie@google.com>
- operations.md: flow control queues buffer request bodies in EPP memory under saturation, bounded by per-band maxRequests/maxBytes; note that queues, fairness, and the saturation view are per replica in Active-Active mode. - architecture.md: note the flowControl gate is on by default and how to disable it. - apix: the FeatureGates doc claimed omitted gates are always disabled; gates carry per-gate defaults and support the name=false syntax. - metrics.md: request_processing_duration_seconds note updated for default-on. Signed-off-by: Luke Van Drie <lukevandrie@google.com>
github-actions
Bot
added
size/L
Flow control treats an endpointless pool as a scale-from-zero waiting room: a request holds in queue until its queue-wait budget expires rather than failing fast. At the 60s default the client's 10s timeout and the 30s probe window both expire first, so the scale-to-zero specs never observe the 503 they assert. Give those two specs a 5s budget. They assert prompt failure under disruption, not cold-start queueing. Signed-off-by: Luke Van Drie <lukevandrie@google.com>
ENABLE_EXPERIMENTAL_FLOW_CONTROL_LAYER=false no longer disables the layer: the variable only ever appended the gate, so a false value falls through to the registered default, which is now enabled. Log the observed value and the featureGates opt-out so the operator who set it can see it is ignored. The 5000 / 1G defaults belong to priorityBands, not to the global flowControl caps, which default to unlimited; sizing advice that names the global knob sends operators to the wrong field and implies a bound that is not configured. The priority-holdback policy lowers admission ceilings, holding gated traffic in queue. It is not a shedding knob, and listing it beside the two that do shed reads as though it were. Also assert the registered gate default, which nothing covered, and the silent cases of the ignored-settings warning, which guard the saturationDetector exclusion that keeps it from firing on every legacy-path startup. Signed-off-by: Luke Van Drie <lukevandrie@google.com>
This was referenced Jul 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind feature
/kind deprecation
What this PR does / why we need it:
Flips the registered default of the
flowControlfeature gate totrue, making flow control the default admission path (#1187). The flip itself is one line; the rest of the PR is the fallout:flowControl:config section is present while the gate is explicitly disabled. Everything in the section exceptsaturationDetector(which the legacy admission path also uses) was silently ignored before.ENABLE_EXPERIMENTAL_FLOW_CONTROL_LAYERis now a no-op; its deprecation log and comment say so. The env var was already broken on main (ENABLE_EXPERIMENTAL_FLOW_CONTROL_LAYERdoesn't actually enable Flow Control #2065), and removal vs landing fix: sync deprecated flow control env var into feature gate map #2076 is resolved there.flowControl=false, which doubles as opt-out coverage. The lora-affinity case keeps running on the default gate-on path, with pod queue depths lowered below the saturation threshold so its scheduling assertion is unchanged.FeatureGatesAPI docs are updated for per-gate defaults and thename=falsesyntax.This intentionally lands ahead of the remaining #1187 hardening items to maximize soak time in nightly CI before the release cut. If open issues remain (or new ones surface) before the cut, reverting this single PR restores the previous default. Benchmark and e2e numbers from this branch will be attached to #2104.
Which issue(s) this PR fixes:
Fixes #2104
Release note (write
NONEif no user-facing change):