[Flow Control] Production readiness: enable the flowControl feature gate by default

[LukeAVanDrie]

Tracks enabling the flowControl feature gate by default (#1270). Code references pinned to 6cc1dac7.

Dates. The flip lands ~Jul 28 to soak in nightlies before the end-of-July llm-d-router cut. Go/no-go is Jul 27. The flip itself is one line and is not release-coupled.

Scope. This issue owns the router: blockers, the flip, router perf numbers, and router-side observability and reliability. Guide posture, deployment docs, and guide-level validation are llm-d/llm-d#2114, which binds on the Jul 30 code freeze for llm-d v0.9.0.

Blocking the flip

Boxes are checked on merge, not on PR open. Each item is a small, independently revertible PR; they are independent and can be in flight concurrently.

• A [Flow Control] [Graduation Blocker] DefaultRequestTTL and per-band MaxRequests have no effective defaults, allowing unbounded queueing #2095 -- DefaultRequestTTL and per-band MaxRequests have no effect
• B [Flow Control] [Graduation Blocker] Pre-admission TTL/cancel returns 500; registry lookup failure mislabeled as saturation (429) #2097 -- pre-admission TTL/cancel returns 500; registry lookup mislabeled as 429 (PR fix(flowcontrol): map pre-admission TTL/cancel to eviction statuses i… #2162, in review)
• C [Flow Control] [Graduation Blocker] Log quality pass; request drops visible at default verbosity #2101 [@ishwar170695] -- log quality pass; request drops visible at default verbosity
• E [Flow Control] [Graduation Blocker] Stale-metrics saturation silently halts all dispatch; pool_saturation help text contradicts its semantics #2100 -- stale-metrics saturation silently halts all dispatch; pool_saturation semantics (PR fix(flowcontrol): make stale-metrics saturation observable and correct pool_saturation semantics #2165, in review)
• F [Flow Control] [Graduation Blocker] Per-band and global queue utilization ratio gauges #2102 [@sudoalok] -- per-band and global queue utilization gauges (PR feat(flowcontrol): add per-band and global queue utilization gauges #2184, in review)
• G [Flow Control] [Graduation Blocker] Test feature-gate wiring in both directions and an admission-to-controller seam test #2103 -- feature-gate wiring tested in both directions
• H [Flow Control] Decide a cardinality policy for caller-controlled flow control labels #2106 -- cardinality policy for caller-controlled flow control labels
• I fix(datastore): retry dropped endpoint registrations instead of silently untracking pods #2092 -- datastore retries dropped endpoint registrations
• J -- router numbers for the flip PR: pkg/epp/flowcontrol benchmarks and the e2e suite with the gate on. Not started. The guide-level A/B at saturating load, which is what audit goal 3 actually turns on, is [Flow Control] v0.9.0: guide posture, docs, and validation for flow control on by default llm-d#2114 section 2 and does not gate the router cut.
• K [Flow Control] [Graduation Blocker] Enable the flowControl feature gate by default #2104 -- the flip (PR feat: enable the flowControl feature gate by default #2180, in review)

Go/no-go, Jul 27: if B or E has not merged, the router ships with the gate off and the flip moves to the next cut. B is a wrong-status-code correctness bug on the default path; E is the failure mode where a stale signal halts all dispatch. F and J gate the evidence and observability we need to debug the flip in nightlies, but neither is a correctness risk on its own.

Landing risk: make presubmit fails on clean main at check-latest-tags-strict (#2181). Six PRs have to clear the pre-merge gate this week, and a failure unrelated to the change under test is how the gate gets skipped. Worth fixing first; it is a one-line pin.

Not blocking

• D [Flow Control] [Graduation Blocker] Recover posture and invariant guards for flow-control-owned goroutines #2098 -- recover posture and invariant guards. Demoted to post-v0.10 (see [Flow Control] [Graduation Blocker] Recover posture and invariant guards for flow-control-owned goroutines #2098). The panics are removed by de-duplicating queue statistics rather than by guarding them, which makes this a cleanup rather than a gate. PRs refactor(flowcontrol): make queue statistics single-owner and remove the invariant panic #2170 and refactor(flowcontrol): UsageLimitPolicy fills a framework-owned ceilings buffer #2171.
• L [Flow Control] Request submitted during shutdown can strand until TTL with no finalization #2099 -- done.
• M ENABLE_EXPERIMENTAL_FLOW_CONTROL_LAYER doesn't actually enable Flow Control #2065 -- ENABLE_EXPERIMENTAL_FLOW_CONTROL_LAYER does not enable flow control. Becomes a no-op after the flip; resolve with [Flow Control] [Graduation Blocker] Enable the flowControl feature gate by default #2104's env-var decision.

The flip reaches llm-d the night it merges

Both llm-d-router-standalone charts render EPP as ghcr.io/llm-d/llm-d-router-endpoint-picker[-dev]:main with pullPolicy: Always, so every guide and every nightly already runs EPP built from router main. There is no chart release or llm-d change between this flip and the whole guide fleet running with flow control on.

Two consequences for the landing plan:

• The rollout cannot be staged from this side. Anything a guide needs must be committed to llm-d before the flip, decided from pre-flip evidence.
• Merging on Jul 28 leaves one night of post-flip nightly data before the Jul 30 llm-d code freeze. Slipping the flip past Jul 28 removes that night entirely.

Guide posture, the defaultRequestTTL sizing for cold-start guides, and the docs and CI work are tracked on llm-d/llm-d#2114. Help from guide owners is wanted there.

Landed since this issue opened

Shutdown-drain 503 mapping with outcome-to-status table tests (#1612); flow_control_requests_total{outcome,priority,inference_pool} (#1636); gate-disable syntax (#1728); --drain-timeout with readiness flip during drain (#1885); real-controller integration suite under -race in CI (#1545); capacity-rejection logs with used-vs-limit context; Admit docstring fix (#2009); concurrency-detector saturation fix via dynamic attributes (#1478).

#1478 is not on release-0.8, which explains field reports like #1474 and part of #2060.

Audit goals

The full six-goal breakdown with deployment-side coverage is on llm-d/llm-d#2114 section 2. The router-side half:

1. Feature completeness -- P/D is not ready; P/D saturation gating is [Flow Control / Disaggregation] Stage-Aware Saturation Gating for Prefill/Decode Pools #1186.
2. Change of behavior -- sheddable requests buffer until TTL or a capacity limit instead of shedding on saturation. Legacy behavior remains only in LegacyAdmissionController; the priority-holdback policy (feat: add priority based holdback strategies in flow control #1592) gates low-priority traffic earlier. Confirm no other deltas.
3. Performance -- J covers dispatch-path overhead.
4. Observability -- C and F. Metrics have nowhere to surface in the guides yet (Add monitoring for llm-d-router to monitoring section in all well lit path guides llm-d#1777).
5. Reliability -- E is the most important new failure mode.
6. Operations -- feat: enable the flowControl feature gate by default #2180 adds active-active limitations and queueing-induced latency to docs/operations.md.

Fast-follow

Filed:

• [Flow Control] Merge the controller and processor into a single engine (continues #889) #2169 -- merge the controller and processor into a single engine (continues Simplification and refactoring of the Flow Control layer #889). Absorbs the registry stats cleanup: Stats() is on the enqueue hot path via hasCapacity, and the utilization gauges ([Flow Control] [Graduation Blocker] Per-band and global queue utilization ratio gauges #2102) are instrumented off it deliberately so this removal does not churn metrics.
• [Flow Control] Derive QueueOutcome from the finalization error instead of storing both #2168 -- derive QueueOutcome from the finalization error instead of storing both.
• [Flow Control] Queue-wait TTL eviction should return 429, not 503, when the pool has endpoints #2046 -- queue-wait TTL eviction should return 429, not 503, when the pool has endpoints.
• [Flow Control] Queue-wait TTL should split by unavailability regime, like the status code in #2046 #2183 -- split queue-wait TTL by unavailability regime.
• [Flow Control] Eviction of in-flight dispatched requests #1119 -- in-flight eviction. flowcontrol/eviction/ is unwired in production today (EvictN is test-only); its hardening items activate when this lands.
• probabilistic-admitter-epp-config.yaml header no longer describes how admission composes #2182 -- probabilistic-admitter-epp-config.yaml documents its layering in terms of the legacy path. The flip makes the header wrong: the config neuters utilization-detector, so flow control never queues and the admitter sheds as before, which works for a reason the comment does not state. Flip-triggered, so it should not drift past the cut by much.

Needs an issue:

• Drain-window semantics: the FlowController shuts down on the SIGTERM context, so evictAll 503s queued work while ext_proc keeps serving for --drain-timeout. Decide dispatch-until-deadline vs documented immediate-evict, and add an e2e asserting SIGTERM behavior (e2e runs --drain-timeout=0). Includes Retry-After on 429/503 or a recorded decision to omit. [Guides] update all guides to properly configure request draining llm-d#1525 depends on the answer.
• Backlog-drain ceiling: dispatch is one item per cycle on a 1ms ticker, giving a ~1k req/s drain ceiling, and a finalized-but-unswept item at a queue head consumes a full cycle. Document or add bounded batch drain, decided with benchmarks.
• Fallback observability: fallback-admitted requests report the fallback priority in some metrics and the original in others. Add a dedicated fallback counter.
• Flow-control Prometheus rules. The base alert pack shipped (Ship default Prometheus alerting rules (PrometheusRule) for the EPP #1635); the flow-control rules need the utilization gauges ([Flow Control] [Graduation Blocker] Per-band and global queue utilization ratio gauges #2102) first. Include promtool check rules in CI, goroutine-leak checks, and a soak test.
• MaxBytes vs max-payload validation: oversized requests reject permanently, masked as saturation. Relevant for multi-modal.
• Deprecation timeline for the duplicated inference_extension_* metric twins.
• Queue-wait tracing child span (claimed in comments; follows [Observability] Standardize opentelemetry service, instrument and span naming #1519/observability: standardize OTel service, scope, and span naming #1565 conventions).
• Stale comments left by the sharding collapse: registry_helpers.go:75, registry/config.go:98, managedqueue.go:183-185, processor.go:534-535.

Benchmark harness in CI coordinates with PR #1638.

[LukeAVanDrie]

I think we should prioritize a logging audit and memory validation. In addition to a scale test, I would like to monitor a long-running EPP instance to verify there are no memory leaks under sustained load.

[sudoalok]

@LukeAVanDrie can I take the tracing item from this list? the queue_wait child span around the enqueue phase, ending when the request gets dispatched or evicted. will follow the #1565 naming conventions and keep it as its own small PR.

can also do the silent expiration logging as a separate small PR after that. btw while reading the code I noticed the requestID -> reqID rename is already done in controller.go, but a few log lines in controller/internal/processor.go still use requestID, so I'd just fold that into the logging PR.

[LukeAVanDrie]

Yes, much appreciated! There should be plenty of preexisting patterns for the spans in the other layers (e.g., scheduling layer).