feat(prefixcacheaffinity): derive the load gate bar from the request

[Lucas-Fernandes-Martins]

Opt-in penaltyMode: matchedTokens. Default behaviour is unchanged.

Why

The TTFT load gate is the only thing that can release a cache pin once the affinity filter has made one:

bestStickyTTFT − bestNonStickyTTFT > maxTTFTPenaltyMs
TTFT_ms = inFlightTokens / peakPrefillThroughput × 1000

Both inputs are constants chosen at deploy time, and both are hard to choose:

The right threshold is per-deployment and moves. Measured across 8 llm-d deployments, the break-even value spans 143 ms → 4,092 ms. On glm-5-2 it shifted ~16× within a day as mean prompt length went from ~5k to ~78k tokens.

peakPrefillThroughput compounds it. Measured peak spans 743 → 27,261 tok/s against a 15928 default whose own comment says "Calibrated for Qwen 32B on 2x H100 80GB (TP=2), vLLM 0.19". It is a straight multiplier on the effective token threshold, so a wrong value silently rescales a threshold someone else picked.

In practice the default barely fires. On glm-5-2 the gate released 0.6% of requests (4 of 702 sampled scheduling decisions) while 17.2% stayed pinned to an endpoint 5–18 s of prefill more loaded than an available alternative. Reconstructed gate input on that deployment: p50 40 ms, p90 6,459 ms, p99 15,148 ms — against a bar of 18,000 ms.

What this does

Ask the question the constant stands in for. Staying pinned costs the extra queued work; leaving costs re-prefilling whatever the sticky endpoint already holds:

inFlight(bestSticky) − inFlight(bestNonSticky)  >  MatchBlocks × BlockSizeTokens

Both sides are token counts, so no throughput constant enters the decision. Both come from PrefixCacheMatchInfo and InFlightLoad, which the filter already consumes — MatchBlocks() a few lines earlier for the affinity score — so there is no new plumbing and no new dependency.

The bar now scales with how much the request would actually forfeit, which is what makes it survive a workload shift.

Design choices worth reviewing

• maxTTFTPenaltyMs is retained as an absolute ceiling in this mode. A request with a large enough cached prefix would otherwise justify an unbounded queue. Set it to 0 to disable the ceiling.
• Rejected at startup with ttftSource: latencyPredictor. That path returns milliseconds from a model with no token interpretation, so the two sides are not commensurable. Failing fast beats silently comparing unlike units.
• A missing match attribute yields 0 matched tokens and skips the test, rather than treating "no signal" as "cache worth nothing, always release".
• bestTTFT refactored onto bestByTTFT, which also returns the winning endpoint — the break-even test needs per-endpoint state, not a scalar. bestTTFT is kept as a thin wrapper so existing call sites are untouched.

Test plan

go build ./... clean; go test ./pkg/... ./apix/... 0 failures.

Nine new tests in plugin_test.go:

test	asserts
KeepsPinWhenGapBelowMatchedTokens	gap 1000 < matched 1440 → hold
ReleasesPinWhenGapExceedsMatchedTokens	gap 4000 > matched 1440 → release
ScalesWithCacheValue	same gap, larger prefix → now worth holding
CeilingStillApplies	ceiling fires even when the prefix outvalues the gap
NoMatchInfoIsNoOp	missing block size does not release
StaticModeUnchanged	regression on the default path, both directions
DefaultPenaltyModeIsStatic	factory default
MatchedTokensRejectsLatencyPredictor	validation
RejectsUnknownPenaltyMode	validation

Status

Draft, on the fork, not proposed upstream yet. Two things I would want settled first:

1. Whether the ceiling should stay or the mode should be purely break-even.
2. Whether upstream would prefer this to eventually replace maxTTFTPenaltyMs rather than sit beside it. The mode exists as opt-in specifically to avoid changing anyone's behaviour on upgrade, but the long-term win is deleting two calibration knobs, not adding a third.

Measurements come from ~/mistral/sync_perf/ (gate_dist.py reconstructs the gate input from EPP scorer logs; recommend.py measures PPT and break-even per deployment).