burst prefix-cache affinity for prompt-sharing requests

cmd/epp/runner/runner.go

@@ -83,6 +83,7 @@ import (
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/admitter/latencyslo"
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/admitter/probabilisticadmitter"
 	reqdataprodprefix "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/dataproducer/approximateprefix"
+	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/dataproducer/burstprefix"
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/dataproducer/inflightload"
 	mmproducer "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/dataproducer/multimodal"
 	preciseproducer "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/dataproducer/preciseprefixcache"
@@ -520,6 +521,7 @@ func (r *Runner) registerInTreePlugins() {
 	fwkplugin.Register(nohitlru.NoHitLRUType, nohitlru.Factory)
 	fwkplugin.Register(activerequest.ActiveRequestType, activerequest.Factory)
 	fwkplugin.Register(preciseprefixcache.PrecisePrefixCachePluginType, preciseprefixcache.PluginFactory)
+	fwkplugin.Register(burstprefix.PluginType, burstprefix.Factory)
 	fwkplugin.Register(mmcacheaffinity.Type, mmcacheaffinity.Factory)
 	fwkplugin.Register(preciseproducer.PluginType, preciseproducer.PluginFactory)
 

deploy/config/epp-burst-prefix-cache-config.yaml

@@ -0,0 +1,35 @@
+# Sample EPP config: burst prefix cache pipeline
+#   token-producer -> burst-prefix-cache-producer -> prefix-cache-scorer
+#
+# Co-locates bursts of prompt-sharing requests (e.g. RL rollout group samples)
+# onto shared replicas so a shared prefix is prefilled once instead of scattered
+# across replicas on a cold cache.
+apiVersion: llm-d.ai/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+  - type: token-producer
+    parameters:
+      modelName: hf-repo/model-name           # set to the model used in the vLLM deployment
+      vllm:
+        url: http://<vllm-service>.<namespace>.svc.cluster.local:8000
+  - type: single-profile-handler
+  - type: decode-filter
+  - type: burst-prefix-cache-producer
+    parameters:
+      windowDurationMs: 100       # batch window T
+      maxPerReplica: -1           # k; -1 sends all samples of a group to one replica
+      minColocateBlocks: 0        # >0 co-locates distinct groups sharing >= N leading blocks; 0 = load-balanced
+  - type: prefix-cache-scorer
+    parameters:
+      prefixMatchInfoProducerName: burst-prefix-cache-producer
+  - type: load-aware-scorer
+  - type: max-score-picker
+schedulingProfiles:
+  - name: default
+    plugins:
+      - pluginRef: decode-filter
+      - pluginRef: prefix-cache-scorer
+        weight: 2.0
+      - pluginRef: load-aware-scorer
+        weight: 1.0
+      - pluginRef: max-score-picker

pkg/epp/framework/plugins/requestcontrol/dataproducer/README.md

@@ -17,6 +17,7 @@ Producers may also implement additional lifecycle hooks:
 | `token-producer` | [`tokenizer`](tokenizer/) | `TokenizedPrompt` | Tokenizes the request prompt via vLLM `/render`; required by precise-prefix-cache-producer and context-length-aware scorers. |
 | `approx-prefix-cache-producer` | [`approximateprefix`](approximateprefix/) | `PrefixCacheMatchInfo` | Hashes the prompt into blocks and matches against a per-pod LRU index for approximate prefix-cache affinity. |
 | `precise-prefix-cache-producer` | [`preciseprefixcache`](preciseprefixcache/) | `PrefixCacheMatchInfo` | Maintains a precise KV-block index by subscribing to vLLM KV-events; requires `token-producer` upstream. |
+| `burst-prefix-cache-producer` | [`burstprefix`](burstprefix/) | `PrefixCacheMatchInfo` | Batches requests within a time window and co-locates prompt-sharing samples (e.g. RL rollout groups) onto shared replicas; requires `token-producer` upstream. |
 | `inflight-load-producer` | [`inflightload`](inflightload/) | `InFlightLoad` | Tracks real-time in-flight request and token counts per endpoint across the full request lifecycle. |
 | `predicted-latency-producer` | [`predictedlatency`](predictedlatency/) | `LatencyPredictionInfo` | Trains XGBoost models via a sidecar and generates per-endpoint TTFT/TPOT predictions. |
 | `session-id-producer` | [`sessionid`](sessionid/) | `SessionID` | Extracts a session identifier from a request header or cookie and publishes it for affinity-aware plugins. |
@@ -27,6 +28,7 @@ Producers may also implement additional lifecycle hooks:
 The framework resolves a DAG from each plugin's `Produces` and `Consumes` declarations and runs producers in dependency order. Explicit dependencies to be aware of:
 
 - `precise-prefix-cache-producer` **requires** `token-producer` upstream (it consumes `TokenizedPrompt`).
+- `burst-prefix-cache-producer` **requires** `token-producer` upstream (it consumes `TokenizedPrompt`).
 - `mm-embeddings-cache-producer` **optionally** consumes `TokenizedPrompt`; configure `token-producer` first when multimodal features need tokenizer-derived hashes.
 - `inflight-load-producer` **optionally** consumes `PrefixCacheMatchInfo` from an approx or precise prefix producer; prefix-discounting is applied automatically when the attribute is present.
 - `predicted-latency-producer` **optionally** consumes `PrefixCacheMatchInfo`; set `prefixMatchInfoProducerName` in its config to the name of the prefix producer instance.
@@ -35,6 +37,7 @@ The framework resolves a DAG from each plugin's `Produces` and `Consumes` declar
 
 - [Approximate Prefix Cache Producer](approximateprefix/README.md)
 - [Precise Prefix Cache Producer](preciseprefixcache/README.md)
+- [Burst Prefix Cache Producer](burstprefix/README.md)
 - [Token Producer](tokenizer/README.md)
 - [In-Flight Load Producer](inflightload/README.md)
 - [Predicted Latency Producer](predictedlatency/README.md)

pkg/epp/framework/plugins/requestcontrol/dataproducer/approximateprefix/plugin.go

@@ -32,6 +32,7 @@ import (
 	fwksched "github.com/llm-d/llm-d-router/pkg/epp/framework/interface/scheduling"
 	attrprefix "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/datalayer/attribute/prefix"
 	approxprefixconstants "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/dataproducer/approximateprefix/constants"
+	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/dataproducer/prefixhash"
 	tokenproducer "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/dataproducer/tokenizer"
 )
 
@@ -179,7 +180,7 @@ func (p *dataProducer) Produce(ctx context.Context, request *fwksched.InferenceR
 	if p.config.MaxPrefixTokensToMatch > 0 && blockSize > 0 {
 		maxBlocks = p.config.MaxPrefixTokensToMatch / blockSize
 	}
-	perPromptHashes := getBlockHashes(ctx, request, blockSize, maxBlocks)
+	perPromptHashes := prefixhash.GetBlockHashes(ctx, request, blockSize, maxBlocks)
 
 	prefixCacheServers := make(map[ServerID]int)
 	totalBlocks := 0

pkg/epp/framework/plugins/requestcontrol/dataproducer/approximateprefix/plugin_test.go

@@ -32,6 +32,7 @@ import (
 	fwkrh "github.com/llm-d/llm-d-router/pkg/epp/framework/interface/requesthandling"
 	fwksched "github.com/llm-d/llm-d-router/pkg/epp/framework/interface/scheduling"
 	attrprefix "github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/datalayer/attribute/prefix"
+	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/dataproducer/prefixhash"
 )
 
 func testHandle() plugin.Handle {
@@ -138,7 +139,7 @@ func TestPreRequest(t *testing.T) {
 		p.wg.Wait()
 
 		// 4. Verify indexer was updated
-		perPromptHashes := getBlockHashes(context.Background(), req1, config.BlockSizeTokens, defaultMaxPrefixBlocks)
+		perPromptHashes := prefixhash.GetBlockHashes(context.Background(), req1, config.BlockSizeTokens, defaultMaxPrefixBlocks)
 		for _, promptHashes := range perPromptHashes {
 			for _, hash := range promptHashes {
 				pods := p.indexer().Get(hash)
@@ -182,7 +183,7 @@ func TestPreRequest(t *testing.T) {
 			p.PreRequest(context.Background(), req, res)
 			p.wg.Wait()
 
-			perPromptHashes := getBlockHashes(context.Background(), req, config.BlockSizeTokens, defaultMaxPrefixBlocks)
+			perPromptHashes := prefixhash.GetBlockHashes(context.Background(), req, config.BlockSizeTokens, defaultMaxPrefixBlocks)
 			allHashes = append(allHashes, perPromptHashes[0])
 		}
 

pkg/epp/framework/plugins/requestcontrol/dataproducer/approximateprefix/types.go

@@ -22,6 +22,7 @@ import (
 	k8stypes "k8s.io/apimachinery/pkg/types"
 
 	"github.com/llm-d/llm-d-router/pkg/epp/framework/interface/plugin"
+	"github.com/llm-d/llm-d-router/pkg/epp/framework/plugins/requestcontrol/dataproducer/prefixhash"
 )
 
 // indexerInterface maintains an LRU cache of prompt prefix hashes and the server(s) that might have that
@@ -36,8 +37,9 @@ type indexerInterface interface {
 // podSet holds a set of pods that may have a specific prefix hash.
 type podSet map[ServerID]struct{}
 
-// blockHash is a hash of a block of request data.
-type blockHash uint64
+// blockHash is a hash of a block of request data. It aliases prefixhash.BlockHash
+// so this package and other prefix-aware producers share one block-hash type.
+type blockHash = prefixhash.BlockHash
 
 // server contains information about a specific server/pod and its cache capacity.
 type server struct {

pkg/epp/framework/plugins/requestcontrol/dataproducer/burstprefix/README.md

@@ -0,0 +1,73 @@
+# Burst Prefix Cache Producer
+
+**Type:** `burst-prefix-cache-producer`
+
+A request-level data producer that co-locates bursts of prompt-sharing requests
+so a shared prefix is prefilled once instead of scattered across replicas on a
+cold cache.
+
+## Problem
+
+When many requests that share a prompt arrive at the same instant (for example
+the `n` group samples of an RL rollout step), every replica's prefix cache is
+still cold, so a cache-state scorer scores them all zero and load balancing
+spreads them. The shared prompt is then prefilled redundantly on several
+replicas and the prefix-cache benefit is lost.
+
+## What it does
+
+Requests arriving within a configurable window are assigned jointly:
+
+1. Each request's prompt is hashed into prefix blocks (shared `prefixhash`).
+2. Requests with an identical prompt prefix are grouped. An identical group of
+   more than one member is always a placement unit; a single request becomes a
+   unit only when `minColocateBlocks > 0` and it shares at least that many leading
+   blocks with some other request in the batch.
+3. Units are steered onto a replica (or a bounded set of replicas), filling one
+   replica up to `maxPerReplica` before spilling to the next least-loaded replica.
+   Identical groups are placed first and kept whole, so the proven same-prompt
+   co-location is the firm structure; prefix-sharing units then attach to it. A
+   unit prefers a replica that already holds a unit sharing at least
+   `minColocateBlocks` leading blocks and is still under its fair share of the
+   batch (prefix co-location bounded by balance, so a shared prefix is prefilled
+   once without stampeding prefix-sharing units onto one replica); otherwise units
+   are balanced across replicas. Longer-prefix units are placed first so shorter
+   units match against the richest set of already-placed prefixes.
+4. The producer emits `PrefixCacheMatchInfo` with a full match on the assigned
+   replica and zero elsewhere.
+
+A request that overlaps no other in the batch, and any prefix-less request,
+receives no affinity (scored zero everywhere), leaving it to other scorers.
+With `minColocateBlocks == 0` only identical groups are placed.
+
+## Scoring
+
+This producer emits `PrefixCacheMatchInfo`; it does not score. Reuse the
+`prefix-cache-scorer` and point it at this producer:
+
+```yaml
+- type: prefix-cache-scorer
+  parameters:
+    prefixMatchInfoProducerName: burst-prefix-cache-producer
+```
+
+## Configuration
+
+| field | default | meaning |
+|---|---|---|
+| `windowDurationMs` | 100 | batch window T in milliseconds; must be in `1..10000`. Every request waits up to this long, so keep it small relative to other request processing times. |
+| `maxPerReplica` | -1 | max samples of one group per replica (k); -1 = unlimited (whole group to one replica) |
+| `blockSizeTokens` | 64 | token block size for prefix hashing |
+| `maxPrefixTokensToMatch` | 0 | cap on matched prefix tokens; 0 uses the default block cap. A positive value must be >= `blockSizeTokens`, otherwise it yields zero prefix blocks. |
+| `minColocateBlocks` | 0 | min shared leading blocks for inter-unit co-location and for a single request to gain an affinity; 0 disables both (only identical groups are placed, purely load-balanced) |
+| `maxBatchSize` | 1000 | max requests one window may accumulate; Produce returns an error once reached. -1 = unlimited. |
+
+## Operational notes
+
+- The producer adds up to `windowDurationMs` of latency per request while a
+  window fills; its producer timeout is extended to cover the window.
+- Co-location is decided within a single window from the request prompts. Identity
+  (the samples of one prompt) is matched exactly; partial overlap between distinct
+  prompts is matched to `minColocateBlocks` leading blocks. Cross-step warm-cache
+  reuse is left to the persistent (approximate or precise) prefix cache producer,
+  which can run alongside this one.