[Flow Control] Priority-Based Progressive Holdback (Adaptive Capacity Reserves)

[LukeAVanDrie]

Flow control currently enforces binary backpressure: when the backend pool reaches 100% saturation, EPP stops dispatching and queues incoming requests.

To protect critical real-time latency SLOs under heavy load, we need a progressive holdback mechanism. This allows EPP to proactively gate/buffer lower-priority traffic (e.g., best-effort batch at 75% load, standard at 90% load) as pool saturation rises, thereby maintaining a reserve of available compute capacity to absorb sudden high-priority bursts.

Since priority levels are dynamically derived from InferenceObjective CRDs at runtime, hardcoded mappings are brittle. We will implement a parameter-driven priority-holdback-policy* plugin. This plugin will implement EPP's pluggable flowcontrol.UsageLimitPolicy` interface and support multiple gating algorithms to dynamically distribute admission ceilings across the $[0, 1]$ saturation range.

---

1. Pluggable Progressive Gating Strategies

The plugin will compute an admission ceiling $c_i \in [0.0, 1.0]$ for each active priority. In the EPP dispatchCycle, traffic is gated if the pool's current saturation exceeds the priority's ceiling ($S > c_i$).

The plugin may support the following gating strategies (seeking feedback here):

A. static

• Algorithm: Fixed, explicit mapping.
• Formula:
  $$c_i = \begin{cases}
  \text{StaticMap}[p_i] & \text{if } p_i \in \text{StaticMap} \
  \text{DefaultStaticCeiling} & \text{otherwise}
  \end{cases}$$

B. stepwise-spread

• Algorithm: Automatically divides the range between a configured minCeiling ($c_{min}$) and maxCeiling ($c_{max}$) into equal steps based strictly on the number $N$ of currently active priorities, ignoring their absolute numerical difference.
• Formula (for index $i$ in sorted descending active priorities):
  $$c_i = c_{max} - i \times \frac{c_{max} - c_{min}}{N - 1}$$

C. linear-proportional

• Algorithm: Scales ceilings linearly between $c_{min}$ and $c_{max}$ based on the numerical value of active priorities relative to the observed active range $[p_{min}, p_{max}]$.
• Formula:
  $$r_i = \frac{p_i - p_{min}}{p_{max} - p_{min}}$$
  $$c_i = c_{min} + r_i \times (c_{max} - c_{min})$$

D. sigmoid-smoothed

• Algorithm: Maps priorities to ceilings along a normalized logistic sigmoid curve to create "soft floors" and "soft ceilings", grouping similar priorities together.
• Formula:
  $$\sigma(x) = \frac{1}{1 + e^{-k(x - x_0)}}$$
  $$S_{norm}(x) = \frac{\sigma(x) - \sigma(0)}{\sigma(1) - \sigma(0)}$$
  $$c_i = c_{min} + S_{norm}(x_i) \times (c_{max} - c_{min})$$
  (Support mapping over either the absolute priority value domain or the relative index rank domain)

---

2. Safety Guarantees & Corner Cases

• Single Active Priority ($N = 1$): To respect flow control's work-conserving nature, if only a single priority level is active, EPP will bypass progressive holdback and assign it $c_{max}$ (usually 1.0), preventing artificial limits when the pool has spare capacity.
• Dynamic Priority Inversion Prevention: To guarantee that a higher priority is never gated before a lower priority (regardless of mathematical edge cases), the plugin will run a right-to-left monotonizing sweep over the calculated ceilings:

  // Ensure ceilings[i] >= ceilings[i+1] (highest first)
  for i := n - 2; i >= 0; i-- {
      if ceilings[i] < ceilings[i+1] {
          ceilings[i] = ceilings[i+1]
      }
  }

• Boot-Time Validation: The plugin factory will perform local priority-inversion validation on startup when using the static mapping. EPP will fail to boot on invalid configurations.

---

3. Configuration Schema

The policy is configured inside the existing EPP plugins block and referenced inside flowControl:

apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: EndpointPickerConfig
metadata:
  name: default-config
featureGates:
- flowControl
plugins:
- type: priority-holdback-policy
  name: dynamic-holdback
  parameters:
    strategy: "sigmoid-smoothed"
    minCeiling: 0.6
    maxCeiling: 1.0
    sigmoidSteepness: 8.0
    sigmoidMidpoint: 0.5
    sigmoidDomain: "value"
flowControl:
  usageLimitPolicyPluginRef: dynamic-holdback

[LukeAVanDrie]

@evacchi and @lioraron, I would love to get your thoughts on the implementation proposed here. Note, this doesn't address the starvation requirement in kubernetes-sigs/gateway-api-inference-extension#1861. I will create a separate issue to address that as it demands a separate mechanism (likely eviction) IMO.

Are the strategies here useful, if so, what to prioritize? Probably need to land static and stepwise first as an MVP.

[LukeAVanDrie]

/assign @loicmarchal

[loicmarchal]

@LukeAVanDrie I started a first implementation with two strategies (stepwise spread and linear proportional). This gives a first version of the holdback policy without going too complex in the configuration.

The other strategies have their merits of course. The sigmoid would give the ability to group priorites closer to minCeiling and maxCeiling for the bottom and top priorities respectively, but that might become useful if we have more that 2-3 priorities. And the static allows the user to define the ceilings explicitly (although that one is quite weak if the InferenceObjectives change at runtime).

I've also considered other approaches, we could decide to add them if we think they add useful flexibility to reflect plausible usecases. But maybe we can start with a first implementation with the stepwise spread and linear proportional strategies?

Something to consider: I believe the difference between stepwise spread and linear proportional is the same idea as the value vs rank domain in the sigmoid smoothed. If we decide to add the sigmoid strategy, we could have two strategies: linear and sigmoid, both with rank or value domain as parameter to spread the ceiling accross the line/curve evenly or proportionally to the priorities value. If we decide to go for that approach, having released a version that supports "stepwise-spread" and "linear-proportional" could introduce a breaking change if we change it to "linear" with value/rank domain. We could decide to already go for one strategy with value/rank parameter to be future proof. Unless we decide now we won't include the sigmoid strategy.

Another reflection I had: It could be interesting to explore an even more progressive holdback (at least as an optional strategy) as opposed to only stepped holdback strategies, by taking the current saturation into account to calculate the ceilings. The saturation is a parameter to the ComputeLimit function. We could use it, or introduce metrics reflecting the current saturation or the state of the system (queue depth, KV cache utilization, etc.) to calculate the ceilings.
For example, the ceilings could move down more significantly for lower priorities as saturation increases, while ceilings for top priorities move slightly up. This would be a more dynamic (and truly progressive) as opposed to the current strategies where ceilings values are static and don't change as saturation varies. I'll explore that option. That could be for a next round of implementation of strategies.

[lioraron]

This capability is critical for batch-gateway's flow control integration.

We've been running benchmarks (batch-gateway#491) with the current static-usage-limit-policy and confirmed that flow control cannot differentiate between interactive and batch traffic without per-band ceilings:

• The dispatch cycle stops entirely when saturation >= ceiling for the highest-priority band (processor.go:dispatchCycle returns false). Since the static policy returns the same ceiling (1.0) for all bands, either ALL bands are blocked or NONE are — no selective gating.
• With queueDepthThreshold tuned to match the backend's actual queue depth (~200), saturation is permanently >= 1.0 during burst. Both interactive (priority 100) and batch (priority -1) are equally HoL-blocked. Our benchmark scenario with flow-control shows identical interactive latency to the benchmark scenarios without flow-control.
• Setting the threshold higher makes saturation permanently < 1.0, meaning no blocking at all. There is no threshold value that creates meaningful priority differentiation with a uniform ceiling.

The static strategy from this proposal would unblock us immediately — e.g., interactive ceiling = 1.0 (never blocked), batch ceiling = 0.6 (blocked when saturation > 60%). The more sophisticated strategies (stepwise-spread, sigmoid) would be valuable longer term.

What is the current status of this issue?

cc @LukeAVanDrie @loicmarchal

[loicmarchal]

@lioraron the issue is ready to be merged in #1592. Waiting for a maintainer to take a look (cc @liu-cong @ahg-g @elevran @nirrozenbaum)

At the moment it supports computed ceilings, distributed proportionally to the priority values on the ceilings range (linear-proportional strategy) or evenly distributed on the ceilings range (stepwise-spread). We could add support for explicit ceilings definition as a follow-up.

[lioraron]

@loicmarchal Is this issue fully implemented?