feat(epp): add soft-reflective-ceiling usage limit policy (#1956)

* feat(epp): add soft-reflective-ceiling usage limit policy

The static usage limit policy applies one ceiling uniformly across
priorities. This policy graduates the ceiling by band rank so the
highest-priority band is never gated and lower bands are gated first
as pool saturation rises, reserving headroom for critical traffic
without hard-classifying any priority as reservable.

Signed-off-by: Michael Kalantar <kalantar@us.ibm.com>

* docs(epp): clarify when to use soft-reflective-ceiling policy

Refocus the "Why choose" section on continuous rate control,
saturation-adaptive ceilings, and lack of parameters. These are the
properties that distinguish this policy from priority-holdback-policy,
rather than the priority-awareness both plugins share.

Signed-off-by: Michael Kalantar <kalantar@us.ibm.com>

* refactor(epp): key soft-reflective-ceiling counters by priority

The flow-control registry provisions and retires priority bands at
runtime, so a rank-indexed counter follows the rank rather than the
band when the priority set changes. Keying counters by priority value
via sync.Map preserves each band's alternation state when new
priorities appear between existing ones.

Signed-off-by: Michael Kalantar <kalantar@us.ibm.com>

* fix(epp): share tick across gated bands in soft-reflective-ceiling

Per-band counters desynchronize under rising saturation and, combined
with the dispatch loop's short-circuit at the first gated band, starve
lower gated bands. Use a single shared atomic tick so all gated bands
open together on the same call.

Signed-off-by: Michael Kalantar <kalantar@us.ibm.com>

* docs(epp): permit bounded dispatch-spreading state in UsageLimitPolicy

Soften the interface docstring from MUST be stateless to SHOULD be
stateless and explicitly permit small bounded dispatch-spreading state
(e.g. tick counters). Signal conditioning stays out of usage-limit
policies. Update the soft-reflective-ceiling README's Not-Stateless
bullet to match.

Signed-off-by: Michael Kalantar <kalantar@us.ibm.com>

---------

Signed-off-by: Michael Kalantar <kalantar@us.ibm.com>
Co-authored-by: Abdullah Gharaibeh <40361897+ahg-g@users.noreply.github.com>

Author: kalantar

cmd/epp/runner/runner.go

@@ -85,6 +85,7 @@ import (
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/flowcontrol/saturationdetector/utilization"
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/flowcontrol/usagelimits"
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/flowcontrol/usagelimits/priorityholdback"
+	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/flowcontrol/usagelimits/softreflectiveceiling"
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/admitter/latencyslo"
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/admitter/probabilisticadmitter"
 	reqdataprodprefix "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/dataproducer/approximateprefix"
@@ -610,6 +611,7 @@ func (r *Runner) registerInTreePlugins() {
 	fwkplugin.Register(slodeadline.SLODeadlineOrderingPolicyType, slodeadline.SLODeadlineOrderingPolicyFactory)
 	fwkplugin.Register(usagelimits.StaticUsageLimitPolicyType, usagelimits.StaticPolicyFactory)
 	fwkplugin.Register(priorityholdback.PolicyType, priorityholdback.PolicyFactory)
+	fwkplugin.Register(softreflectiveceiling.PolicyType, softreflectiveceiling.Factory)
 
 	// Register Request level data producer plugins as defaults for their respective data keys.
 	fwkplugin.RegisterAsDefaultProducer(reqdataprodprefix.ApproxPrefixCachePluginType, reqdataprodprefix.ApproxPrefixCacheFactory, attrprefix.PrefixCacheMatchInfoDataKey)

deploy/config/soft-reflective-ceiling-epp-config.yaml

@@ -0,0 +1,26 @@
+# EPP configuration for the soft-reflective-ceiling usage limit policy.
+#
+# The policy applies a graduated, priority-aware dispatch ceiling: the
+# highest-priority band is never gated, and progressively lower-priority
+# bands are gated first as pool saturation rises. Gated items remain in
+# their queues; the ceiling controls dispatch, not admission.
+apiVersion: llm-d.ai/v1alpha1
+kind: EndpointPickerConfig
+featureGates:
+- flowControl
+plugins:
+- type: queue-scorer
+- type: kv-cache-utilization-scorer
+- type: prefix-cache-scorer
+- type: soft-reflective-ceiling-policy
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: queue-scorer
+    weight: 2.0
+  - pluginRef: kv-cache-utilization-scorer
+    weight: 2.0
+  - pluginRef: prefix-cache-scorer
+    weight: 3.0
+flowControl:
+  usageLimitPolicyPluginRef: soft-reflective-ceiling-policy

pkg/epp/framework/interface/flowcontrol/plugins.go

@@ -134,11 +134,12 @@ type SaturationDetector interface {
 // Saturation represents resource usage as a fraction of total capacity (0.0 = idle, 1.0 = fully saturated)
 // as described in [/pkg/epp/flowcontrol/contracts.SaturationDetector]
 //
-// Architecture (Stateless Singleton):
+// Architecture (Mostly Stateless Singleton):
 // UsageLimitPolicy plugins are Singletons. A single instance handles limit computation for all priority bands.
-// The plugin MUST be stateless: it is a pure function that maps the current saturation and
-// active priority domain to a set of ceilings. Any signal conditioning (trend detection, smoothing) belongs in
-// the SaturationDetector layer, not here.
+// The plugin SHOULD be stateless -- a pure function mapping the current saturation and active priority
+// domain to a set of ceilings. Small bounded dispatch-spreading state (e.g. a tick counter used to fold
+// successive calls into a proportional duty cycle) is permitted; signal conditioning (trend detection,
+// smoothing) is not, and belongs in the SaturationDetector layer.
 //
 // Integration:
 // This policy is called during dispatch decision-making, before a request is allowed to proceed. For each

pkg/epp/framework/plugins/flowcontrol/usagelimits/softreflectiveceiling/README.md

@@ -0,0 +1,69 @@
+# Soft Reflective Ceiling Usage Limit Policy
+
+**Type:** `soft-reflective-ceiling-policy`
+
+A usage limit policy that applies a graduated, priority-aware dispatch ceiling. Where the [static usage limit policy](../README.md) applies one ceiling uniformly across all priorities, this policy tightens the ceiling for lower-priority bands as pool saturation rises, reserving headroom for higher-priority traffic.
+
+The ceiling controls **dispatch**, not admission. Requests continue to be enqueued as they arrive; a gated band simply is not drawn from for dispatch on that call, so its items remain in their queues until the ceiling opens.
+
+## Why choose this policy?
+
+- **Continuous Rate Control**: Once a band's ceiling is reached, the policy alternates open and closed across calls so that the effective dispatch rate degrades continuously with saturation. This is unlike step-function gating (`static-usage-limit-policy`, `priority-holdback-policy`), which is either fully open or fully closed at a fixed threshold.
+- **Saturation-Adaptive Ceilings**: The per-band ceilings shift with the observed saturation itself. Under light load, lower bands see permissive ceilings; under heavy load, ceilings compress toward zero. Fixed-threshold policies (`priority-holdback-policy`, `static-usage-limit-policy`) hold their thresholds regardless of current load.
+- **No Policy Parameters**: Behavior is derived entirely from saturation and the active priority count. Suits deployments that prefer a single well-defined schedule over per-environment tuning.
+
+Deployments that need to bound the lowest band's gating aggressiveness explicitly should use `priority-holdback-policy` instead.
+
+## What it does
+
+For each call, the policy receives the current pool `saturation` (from the configured `SaturationDetector`) and an ordered list of `priorities` where `priorities[0]` is the highest. It returns one ceiling per band. Each ceiling is compared against saturation by the Flow Controller: when saturation exceeds the ceiling, the band is gated for that call and its items are held in queue.
+
+**Reflective ceiling per band:**
+
+    ceiling[i] = 1 - i * saturation / (N - 1)
+
+where `N = len(priorities)`. Band 0 receives `ceiling[0] = 1.0` and dispatches while saturation is below 1.0. At `saturation >= 1.0` the Flow Controller halts every band, including band 0, because it gates on `saturation >= ceiling`.
+
+**Per-band decision:**
+
+- If `saturation < ceiling[i]`: the band is fully open (`1.0`); items dispatch normally.
+- If `saturation >= 1.0`: non-critical bands are fully gated (`0.0`); their items remain queued until saturation drops.
+- Otherwise the band is at or past its reflective ceiling. It alternates open (`1.0`) and closed (`0.0`) across calls with
+
+        period = round(saturation / (1 - saturation))
+
+    so that, on average, each gated band's ceiling is open on 1 out of every `period` calls. Higher saturation lengthens the gated intervals; the effective dispatch rate approximates `(1 - saturation) / saturation`.
+
+All gated bands share a single policy-wide tick, so on any given call either every gated band's ceiling is open or every gated band's ceiling is closed. That monotonicity across ranks is required because the Flow Controller's dispatch loop aborts at the first band whose ceiling gates. The shared tick is bounded state used only to spread dispatch evenly across calls; signal conditioning (smoothing, hysteresis, trend detection) is delegated to the Saturation Detector layer per the `UsageLimitPolicy` contract.
+
+## Inputs consumed
+
+This policy consumes runtime signals passed by the Flow Controller:
+
+- **Pool Saturation**: The current saturation value from the configured `SaturationDetector` plugin.
+- **Priority Bands**: The ordered list of active priorities. Only the number of bands and their rank order matter, not the numeric priority values.
+
+## Configuration
+
+This policy takes no parameters. Any non-empty parameters block is rejected at load time.
+
+```yaml
+plugins:
+  - type: soft-reflective-ceiling-policy
+flowControl:
+  usageLimitPolicyPluginRef: soft-reflective-ceiling-policy
+```
+
+Unlike the static usage limit policy, this policy is **not** framework-injected. Declare it explicitly to activate it.
+
+## Trade-offs
+
+- **Not Stateless**: The alternation pattern is only observable across calls, so the policy keeps one policy-wide atomic tick counter. This is small bounded state that the `UsageLimitPolicy` contract permits for dispatch spreading, but it means the policy is not a pure function of its inputs: two calls with the same `(saturation, priorities)` will not always return the same result.
+- **Coarse Rate Control**: The effective dispatch rate is `1 / period` for integer `period`, so the gated dispatch rate transitions in discrete steps rather than smoothly with saturation.
+- **Rank-Only, Not Magnitude**: The ceiling depends only on the rank of each priority, not the numeric spacing between them. Two adjacent bands with a wide gap in priority values are treated the same as two bands with a small gap.
+- **Requires Multiple Bands**: With a single band the policy degenerates to always-open. Differentiation begins at two or more bands.
+- **Queue Growth Under Sustained Saturation**: Because gated items remain queued rather than being rejected, lower-priority queues can grow while saturation is high. Bounding queue size is the responsibility of the eviction plugins, not this policy.
+
+## Related Documentation
+- [Static Usage Limit Policy](../README.md)
+- [Flow Control User Guide](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/v1.5.0/site-src/guides/flow-control.md)

pkg/epp/framework/plugins/flowcontrol/usagelimits/softreflectiveceiling/policy.go

@@ -0,0 +1,155 @@
+/*
+Copyright 2026 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Package softreflectiveceiling implements a UsageLimitPolicy that gates
+// lower-priority bands proportionally as saturation rises. For each band i
+// (priorities ordered highest first) it computes a reflective ceiling
+//
+//	ceiling[i] = 1 - i*saturation/(N-1)
+//
+// When saturation reaches a band's ceiling the policy alternates ceiling=1.0
+// and ceiling=0.0 across calls so that, on average, each gated band's
+// ceiling is open on 1/period of calls, where
+// period = round(saturation/(1-saturation)).
+//
+// A single policy-wide tick counter drives the alternation. All gated bands
+// share it, so on any given call either every gated band's ceiling is open
+// or every gated band's ceiling is closed. This monotonicity is required by
+// the dispatch loop, which stops at the first band whose ceiling is at or
+// below current saturation. Per-band counters would drift out of phase as
+// bands enter gating at different saturations, so a lower gated band's
+// open ticks would repeatedly land on a higher gated band's closed ticks
+// and the lower band would starve.
+//
+// The single tick is small bounded state used only for dispatch spreading,
+// which the UsageLimitPolicy contract permits. Signal conditioning (trend
+// detection, smoothing) is not permitted here; it belongs in the
+// SaturationDetector layer.
+package softreflectiveceiling
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"math"
+	"sync/atomic"
+
+	"github.com/go-logr/logr"
+	"sigs.k8s.io/controller-runtime/pkg/log"
+
+	logutil "github.com/llm-d/llm-d-router/pkg/common/observability/logging"
+	"github.com/llm-d/llm-d-router/pkg/epp/framework/interface/flowcontrol"
+	fwkplugin "github.com/llm-d/llm-d-router/pkg/epp/framework/interface/plugin"
+)
+
+// PolicyType is the registration string. The YAML "type:" field in
+// pluginsCustomConfig must equal this value for the loader to find the
+// factory.
+const PolicyType = "soft-reflective-ceiling-policy"
+
+// Factory creates a soft-reflective ceiling policy instance. The algorithm
+// has no tunable parameters, so any provided parameters block must be empty.
+// The framework's strict decoder (DisallowUnknownFields) surfaces config
+// typos at load time rather than silently ignoring them.
+func Factory(name string, rawConfig *json.Decoder, handle fwkplugin.Handle) (fwkplugin.Plugin, error) {
+	if rawConfig != nil {
+		var empty struct{}
+		if err := rawConfig.Decode(&empty); err != nil {
+			return nil, fmt.Errorf("soft-reflective-ceiling-policy takes no parameters: %w", err)
+		}
+	}
+	logger := logr.Discard()
+	if handle != nil {
+		logger = log.FromContext(handle.Context())
+	}
+	return newPolicy(name, logger), nil
+}
+
+type policy struct {
+	name string
+
+	// tick advances once per ComputeLimit call in which at least one band is
+	// in the alternating regime. All gated bands read the same value, so the
+	// per-call gating decision is monotone across ranks -- required because
+	// the dispatch loop stops at the first band whose ceiling has gated.
+	tick atomic.Int64
+}
+
+var _ flowcontrol.UsageLimitPolicy = (*policy)(nil)
+
+func newPolicy(name string, logger logr.Logger) *policy {
+	p := &policy{name: name}
+	logger.WithName(p.TypedName().String()).V(logutil.DEFAULT).Info("Creating new SoftReflectiveCeilingPolicy")
+	return p
+}
+
+func (p *policy) TypedName() fwkplugin.TypedName {
+	return fwkplugin.TypedName{Type: PolicyType, Name: p.name}
+}
+
+// ComputeLimit returns per-band ceilings. priorities[0] is the highest
+// priority band and receives ceiling=1.0. Lower bands whose reflective
+// ceiling has been reached alternate between 1.0 and 0.0 with period
+// round(saturation/(1-saturation)), producing a proportional duty cycle.
+func (p *policy) ComputeLimit(_ context.Context, saturation float64, priorities []int) []float64 {
+	n := len(priorities)
+	ceilings := make([]float64, n)
+	if n == 0 {
+		return ceilings
+	}
+	if n == 1 {
+		ceilings[0] = 1.0
+		return ceilings
+	}
+
+	// Ceilings decrease monotonically with rank, so any band is at or past
+	// its reflective ceiling iff the lowest band is (whose ceiling is
+	// 1 - saturation). Advance the shared tick only when at least one band
+	// is in the alternating regime; below that saturation the policy is a
+	// pure function of its inputs.
+	inAlternatingRegime := saturation >= 1.0-saturation && saturation < 1.0
+
+	var open bool
+	if inAlternatingRegime {
+		// 1e-9 guards against round-off when saturation is very near 1.0.
+		period := int64(math.Max(1, math.Round(saturation/(1.0-saturation+1e-9))))
+		open = p.tick.Add(1)%period == 0
+	}
+
+	for i := range priorities {
+		if i == 0 {
+			ceilings[i] = 1.0
+			continue
+		}
+
+		reflectiveCeiling := 1.0 - float64(i)*saturation/float64(n-1)
+
+		switch {
+		case saturation < reflectiveCeiling:
+			ceilings[i] = 1.0
+		case saturation >= 1.0:
+			ceilings[i] = 0.0
+		default:
+			if open {
+				ceilings[i] = 1.0
+			} else {
+				ceilings[i] = 0.0
+			}
+		}
+	}
+
+	return ceilings
+}