docs: cover flow control default in sizing, architecture, and API docs

- operations.md: flow control queues buffer request bodies in EPP memory under
  saturation, bounded by per-band maxRequests/maxBytes; note that queues,
  fairness, and the saturation view are per replica in Active-Active mode.
- architecture.md: note the flowControl gate is on by default and how to
  disable it.
- apix: the FeatureGates doc claimed omitted gates are always disabled; gates
  carry per-gate defaults and support the name=false syntax.
- metrics.md: request_processing_duration_seconds note updated for default-on.

Signed-off-by: Luke Van Drie <lukevandrie@google.com>

Author: LukeAVanDrie

apix/config/v1alpha1/endpointpickerconfig_types.go

@@ -34,8 +34,9 @@ type EndpointPickerConfig struct {
 	metav1.TypeMeta `json:",inline"`
 
 	// +optional
-	// FeatureGates is a set of flags that enable various experimental features with the EPP.
-	// If omitted none of these experimental features will be enabled.
+	// FeatureGates is a set of flags that toggle optional EPP features. Each entry is a gate name,
+	// optionally suffixed with "=true" or "=false" (a bare name means "=true"). Gates carry
+	// per-gate defaults that apply when omitted; some (such as "flowControl") default to enabled.
 	FeatureGates FeatureGates `json:"featureGates,omitempty"`
 
 	// +required
@@ -186,7 +187,8 @@ func (sp SchedulingPlugin) String() string {
 	return "{" + strings.Join(parts, ", ") + "}"
 }
 
-// FeatureGates is a set of flags that enable various experimental features with the EPP
+// FeatureGates is a set of flags that toggle optional EPP features ("name", "name=true", or
+// "name=false"); omitted gates use their registered defaults.
 type FeatureGates []string
 
 func (fg FeatureGates) String() string {

docs/architecture.md

@@ -195,6 +195,8 @@ RequestHandler:
 - When no parsers are configured, `openai-parser`, `anthropic-parser`, and `vllmhttp-parser` are used.
 
 FlowControl:
+- The flow control admission layer itself is on by default (the `flowControl` feature gate);
+  disable it with `featureGates: ["flowControl=false"]`.
 - `fcfs-ordering-policy`, `global-strict-fairness-policy`, and `static-usage-limit-policy` are configured when absent.
 - `utilization-detector` is configured as the saturation detector when none is set.
 

docs/metrics.md

@@ -116,7 +116,7 @@ Unlabelled.
 
 | Name | Type | Notes |
 |---|---|---|
-| `request_processing_duration_seconds` | Histogram | Time from request receipt until the request body has been handled. Includes admission control, so under the flow control feature gate this covers queue wait; `flow_control_request_queue_duration_seconds` separates it out. |
+| `request_processing_duration_seconds` | Histogram | Time from request receipt until the request body has been handled. Includes admission control, so with flow control on (the default) this covers queue wait; `flow_control_request_queue_duration_seconds` separates it out. |
 | `response_processing_duration_seconds` | Histogram | Sum of the per-chunk handler slices for a streamed response, so model-server generation time between chunks is excluded. For a non-streaming response, the interval from response headers to completion. |
 
 ### Plugin, info, and model rewrite

docs/operations.md

@@ -19,6 +19,12 @@ The EPP acts as the routing intelligence engine. Its resource usage scales prima
 #### Memory Allocation
 - **Base Memory**: EPP memory usage is relatively low and stable with small output token requests, but scales with the number of concurrent inflight requests.
 - **Inflight Requests Impact**: Memory usage increases with the number of concurrent inflight requests and the output (decode) token length.
+- **Flow Control Queues**: With flow control enabled (the default), requests that cannot dispatch
+  under saturation are buffered in EPP memory, including their request bodies. The buffered volume
+  is bounded per priority band by `flowControl` `maxRequests` (default 5000) and `maxBytes`
+  (default 1G); budget for the sum of the per-band `maxBytes` limits of the priority levels your
+  traffic actually uses, on top of the inflight-request sizing above. Lower these limits (or set a
+  shorter `defaultRequestTTL`) to trade queueing for earlier shedding.
 - **Sizing Guidelines**:
   - For a request rate of 50 to 100 requests/second with 1k output tokens, EPP requires between **4 GiB and 6 GiB** of memory.
   - For workloads with longer output lengths (such as 5k output tokens), memory usage can reach **20+ GiB** due to the accumulation of state for concurrent inflight requests.
@@ -37,6 +43,10 @@ The EPP's scaling behavior and effectiveness are highly dependent on the configu
   | 3 | 2.7x |
   | 4 | 3.5x |
 
+  - **Note (Flow Control)**: Flow control state (queues, fairness accounting, and the saturation
+    view) is per replica and not shared. In Active-Active mode, priority and fairness are enforced
+    only within each replica's share of the traffic, and per-band capacity limits apply per
+    replica, so the fleet-wide queued volume scales with the replica count.
   - **Warning (Prefix Routing)**: **Active-Active mode should be avoided when using approximate prefix routing.** Because EPP replicas do not share prefix state, each replica only has visibility into the prefix state of the requests it has individually handled. This partition of state significantly degrades prefix cache hit rates, making prefix caching highly inefficient.
   - For more technical details and context on EPP replica state sync and scaling limitations, see [Issue #1290](https://github.com/llm-d/llm-d-router/issues/1290).
 

pkg/epp/flowcontrol/README.md

@@ -32,11 +32,12 @@ high load.
 
 ### Enabling, disabling, and tuning
 
-Flow control is enabled by default via the `flowControl` feature gate. With it on, requests queue
-under saturation and are admitted by priority instead of being rejected outright; sheddable
-(negative-priority) requests buffer until TTL or capacity limits are hit rather than shedding
-immediately on saturation. To restore the legacy saturation-only admission behavior, opt out in the
-EPP config:
+Flow control is enabled by default via the `flowControl` feature gate. With it on, saturation no
+longer pushes excess requests into each model server's local queue: they wait in the EPP and
+dispatch by priority as capacity frees, with queue wait bounded by a TTL. Sheddable
+(negative-priority) requests buffer under the same TTL and capacity rules instead of being
+rejected immediately on saturation. To restore the legacy saturation-only admission behavior, opt
+out in the EPP config:
 
 ```yaml
 featureGates: ["flowControl=false"]
@@ -46,8 +47,12 @@ Setting a `flowControl:` config section while the gate is disabled logs a warnin
 except `saturationDetector` (which the legacy admission path also uses) is ignored.
 
 Operationally, dispatch decisions depend on fresh endpoint metrics: saturation detection reads the
-scraped model-server metrics, so a broken or stale scrape path degrades or halts dispatch. Keep
-metrics scrape health monitored when running with flow control on.
+model-server metrics that the EPP's own data layer scrapes from pods (this is independent of
+cluster monitoring such as Prometheus). Endpoints whose metrics are older than the detector's
+`metricsStalenessThreshold` count as fully saturated, so if the EPP loses its scrape path to all
+pods (network policy, metrics port change, a starved refresh loop), dispatch halts and queued
+requests eventually shed at their TTL. Keep the refresh interval comfortably inside the staleness
+threshold and monitor scrape health when running with flow control on.
 
 Tuning knobs, all under the `flowControl:` config section: