Skip to content

feat(flowcontrol): demand-driven in-flight eviction#2093

Draft
LukeAVanDrie wants to merge 5 commits into
llm-d:mainfrom
LukeAVanDrie:feat/fc-eviction-controller
Draft

feat(flowcontrol): demand-driven in-flight eviction#2093
LukeAVanDrie wants to merge 5 commits into
llm-d:mainfrom
LukeAVanDrie:feat/fc-eviction-controller

Conversation

@LukeAVanDrie

@LukeAVanDrie LukeAVanDrie commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind feature

What this PR does / why we need it:

Implements the reclamation controller from the in-flight eviction design (#2061, docs/flow-control-eviction.md): flow control can now evict in-flight sheddable requests (priority < 0) when higher-priority requests are queued behind pool saturation, converting the priority-inversion window from unbounded natural churn time to a bounded revocation round-trip.

The merged victim filter/ordering policies and the ext_proc eviction channel are used as-is; the merged RequestEvictor is reused with small tracker extensions (commit 1), and the wiring is adapted from draft PR #1360; the wiring commit carries a Co-authored-by trailer for @RishabhSaini. Maintainers, please retain it in the squash-merge message. This PR adds the piece #1119 left open, deciding when and how much to evict:

  • Trigger: on Head-of-Line blocking, scan blocked bands for queued demand strictly above the victim head's priority; bands with unattainable (zero) ceilings are excluded. The demand band's priority bounds victim priority for the whole decision, so multi-revocation decisions cannot churn the demand band itself.
  • Sizing: a saturation-gauge deficit against a mean-footprint credit, with pending-reclaim debits so capacity in the revoking/reclaiming stages is never counted twice.
  • Pacing: confirmation-gated. New decisions wait until every evicted request's stream has terminated plus a grace covering the sensor's confirmation-to-visibility lag, with a timeout so a hung stream cannot wedge reclamation.
  • Configuration: one public field, flowControl.enableEviction (default false). The grace is derived at wiring time from the selected saturation detector (dispatch-tick scale for the concurrency detector; refresh interval + staleness threshold + an engine reclaim budget for scraped sensors); remaining parameters are internal. The design doc's Configuration section carries the rationale.
  • Observability: revocations issued and terminal outcomes (confirmed, timed_out), reclaim target, pending reclaim, and issue-to-confirmation latency.

Review guide: the diff is +1365/-38 across 22 files, but the decision logic is one new file, pkg/epp/flowcontrol/controller/internal/reclamation.go (~330 lines, half of it the invariants as comments); review it against the design doc's Trigger/Sizing/Pacing sections. The rest splits into tracker extensions on the merged RequestEvictor (commit 1), metrics (mechanical), and wiring (commit 3, adapted from #1360). Tests are ~45% of the diff. The three commits are ordered to be read in sequence.

A closed-loop benchmark reproducing the design doc's Validation results is published alongside this PR as a companion draft (necessarily based on main, since GitHub cannot base a PR on a fork branch; its reviewable diff is its final commit). Per the design doc, an end-to-end measurement of issue-to-abort latency is part of the acceptance criteria; eviction remains experimental (within the experimental flow control layer) until that measurement exists.

Which issue(s) this PR fixes:

Part of #1119 (partial eviction and retry orchestration are deferred per the design doc's non-goals; close the issue here if we consider the core ask delivered)

Release note:

Flow control can evict in-flight sheddable requests (priority < 0) to reclaim pool capacity when higher-priority requests are blocked by saturation. Disabled by default; enable with flowControl.enableEviction. When enabled, sheddable requests may be terminated mid-stream with HTTP 429.

Test plan (new tests; functionality verified):

  • Reclamation sizing across the decision table, including the exact-boundary (zero-deficit) case and zero-ceiling demand exclusion
  • Pacing gate lifecycle: closed while outstanding, cooling through the grace, timeout retirement with late-confirmation ignored
  • Trigger through the real dispatch cycle: HoL break fires a decision, empty bands and non-dominated victims do not, repeat cycles against a stale gauge are suppressed
  • Tracker confirmation listener fires at most once, only for evicted requests; failed evictions re-track without notifying; victim priority bound stops a decision at the demand band
  • Config translation and validation
  • Race detector clean on the eviction and controller packages

Not covered here (tracked as acceptance criteria in the design doc): live e2e measurement of the ImmediateResponse-to-engine-abort chain.

Known edge, tolerated with a comment at the site: re-tracking a victim (bound hit or failed eviction) races a concurrent completion in a sub-microsecond window; the re-inserted dead entry costs at most one spurious ConfirmationTimeout stall on a later decision.

LukeAVanDrie and others added 3 commits July 20, 2026 16:04
…he tracker

RequestEvictor gains the confirmation signal for reclamation pacing
(an at-most-once listener fired when an evicted request's stream
terminates), a victim-priority peek, and a strict per-decision victim
priority bound on EvictN so multi-revocation decisions cannot churn
the demand band itself.

Signed-off-by: Luke Van Drie <lukevandrie@google.com>
… eviction

Implements the controller from docs/flow-control-eviction.md (PR
llm-d#2061): on HoL blocking, scan blocked bands for queued demand strictly
above the victim head, size revocations as a saturation deficit
against a mean-footprint credit with pending-reclaim debits, and pace
by confirmation (stream termination) plus a grace covering the
sensor's visibility lag, with a timeout so a hung stream cannot wedge
the gate. Bands with unattainable ceilings are excluded from demand.
Adds the flowControl.enableEviction API field and its translation.
New metrics: revocations issued and terminal outcomes, reclaim target,
pending reclaim, confirmation latency.

Signed-off-by: Luke Van Drie <lukevandrie@google.com>
The confirmation grace is derived at wiring time from the selected
saturation detector (dispatch-tick scale for the concurrency detector;
refresh interval + staleness + engine reclaim budget for scraped
sensors); remaining controller parameters are internal. The Director
tracks dispatches and stream termination; the ext_proc server receives
the evict-channel lookup.

Signed-off-by: Luke Van Drie <lukevandrie@google.com>

Co-authored-by: Rishabh Saini <rishabhsaini01@gmail.com>
Signed-off-by: Luke Van Drie <lukevandrie@google.com>
@github-actions github-actions Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. kind/feature Categorizes issue or PR as related to a new feature. labels Jul 20, 2026
Signed-off-by: Luke Van Drie <lukevandrie@google.com>
Operators tuning holdback and priorities need to see which band's
demand drives destruction; the terminal-outcome counter stays
unlabeled because it measures actuator health, not band economics.
Adds an emission test covering the eviction metrics.

Signed-off-by: Luke Van Drie <lukevandrie@google.com>
@github-actions github-actions Bot added kind/feature Categorizes issue or PR as related to a new feature. and removed kind/feature Categorizes issue or PR as related to a new feature. labels Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/feature Categorizes issue or PR as related to a new feature. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant