Add support for length-based scoring in prefix cache plugin by azamikram · Pull Request #1459 · llm-d/llm-d-router

azamikram · 2026-06-02T19:21:16Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

The current prefix scorer only considers the match ratio when scoring endpoints, treating all matched prefixes equally regardless of their absolute length.

This PR introduces matchLengthWeight and matchLengthScaleTokens parameters to the prefix scorer. The updated scoring function computes a weighted sum of the match ratio and the match length ratio. The match length ratio is non-linear to prioritize KV cache reuse for longer requests relative to the matchLengthScaleTokens.

To maintain backward compatibility, matchLengthWeight defaults to 0.0, preserving the current scoring behavior.

Benchmark results and detailed report

Macro Results (RatioOnlyScorer is the baseline and CostAwareScorer is the proposed function)

Use case 1: RatioOnlyScorer never migrates to a cold pod even when the load increases. CostAwareScorer migrates the short requests to a cold pod.

Use case 2: Even when RatioOnlyScorer migrates to a cold pod, it places both short and long request on the cold pod.

Use case 3: When the load increases so much that long requests also need to be migrates to a cold pod, both RatioOnlyScorer and CostAwareScorer converge.

Which issue(s) this PR fixes:

Fixes #1095

Release note (write NONE if no user-facing change):

Prefix cache scorer optionally considers absolute prefix match length.

github-actions · 2026-06-02T19:21:30Z

🚨 Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

liu-cong

can you also add your benchmark results?

liu-cong · 2026-06-02T23:51:23Z

 	// The name of the data producer that produces PrefixCacheMatchInfo.
-	PrefixMatchInfoProducerName string `json:"prefixMatchInfoProducerName,omitempty"`
+	PrefixMatchInfoProducerName string   `json:"prefixMatchInfoProducerName,omitempty"`
+	PrefixLengthWeight          *float64 `json:"prefixLengthWeight,omitempty"`


can we use float and int instead of the pointers?

Pointers are useful for differentiating between fields not in the JSON and fields in the JSON with the default value (0 or 0.0 in this case).

I decided to go with float and int to keep things simple. Let me know what you think.

shmuelk · 2026-06-04T11:52:33Z


- higher score: more of the request prefix is expected to be reusable from cache
- lower score: less prefix cache reuse is expected
+If `prefixLengthWeight` is `0.0` (the default), the score simplifies to just the `matchRatio`.


This comment should be included in the initial description of how the plugin computes the score.

shmuelk · 2026-06-04T11:53:23Z

 ## Configuration

-This plugin does not define any plugin-specific parameters.
+This plugin supports the following configuration parameters:


Suggested change

This plugin supports the following configuration parameters:

This plugin supports the following optional configuration parameters:

liu-cong · 2026-06-05T16:31:59Z

/hold until we get good benchmarking results

shmuelk · 2026-06-09T06:00:17Z

@azamikram Please rebase with main to solve the CI issues

ahg-g

I think another approach is to set a minimum for the denominator (8k tokens for example, but we need to do some testing), this will make it simpler since it is just one parameter, not two.

ahg-g · 2026-06-05T12:35:43Z

 	// The name of the data producer that produces PrefixCacheMatchInfo.
-	PrefixMatchInfoProducerName string `json:"prefixMatchInfoProducerName,omitempty"`
+	PrefixMatchInfoProducerName string   `json:"prefixMatchInfoProducerName,omitempty"`
+	PrefixLengthWeight          *float64 `json:"prefixLengthWeight,omitempty"`


pls clearly document the parameters

ahg-g · 2026-06-05T12:39:14Z

-	PrefixMatchInfoProducerName string `json:"prefixMatchInfoProducerName,omitempty"`
+	PrefixMatchInfoProducerName string   `json:"prefixMatchInfoProducerName,omitempty"`
+	PrefixLengthWeight          *float64 `json:"prefixLengthWeight,omitempty"`
+	MaxModelLen                 *int     `json:"maxModelLen,omitempty"`


MaxModelLen is confusing (I know vllm uses this name, but it is not accurate), probably MaxContextLength? also, what is the unit? please prefix it with the unit, if tokens, it would be MaxContextLengthTokens

I agree that MaxModelLen is a bit confusing and its purpose isn't set to a limit on the maximum number of tokens. The goal is to configure the point at which the requests become most important for KV cache usage. I ended up renaming it to prefillSaturationTokens, let me know what you think.

ahg-g · 2026-06-05T12:41:06Z

-This produces a normalized score in the range `[0, 1]`:
+Where:
+- `matchRatio = matchBlocks / totalBlocks` (the fraction of the request prefix matched)
+- `matchLengthRatio = min(1.0, matchedBlocks * blockSize / maxModelLen) ^ 2` (square of normalized matched blocks relative to `maxModelLen`)


explain why this is applying a square

ahg-g · 2026-06-05T12:47:57Z

+| Parameter | Type | Description | Default |
+| :--- | :--- | :--- | :--- |
+| `prefixLengthWeight` | float | Weight of the absolute prefix length in the score. Must be between `0.0` and `1.0`. | `0.0` |
+| `maxModelLen` | integer | The maximum context length (in tokens) supported by the model. Required if `prefixLengthWeight` > `0.0`. | `8192` |


where is this default being applied?

ahg-g · 2026-06-05T12:50:08Z

@@ -115,8 +138,19 @@ func (p *Plugin) Score(ctx context.Context, _ *fwksched.InferenceRequest, endpoi
 		}

 		if prefixMatchInfo, ok := info.(*attrprefix.PrefixCacheMatchInfo); ok {


to reduce indentation, pls do the following:

if prefixMatchInfo, ok := info.(*attrprefix.PrefixCacheMatchInfo); !ok { logger.V(logutil.DEFAULT).Error(nil, "PrefixCacheMatchInfo has unexpected type, assigning score 0", "endpoint", endpoint) return scores } // reset of the scores calculations return scores

hexfusion · 2026-06-12T14:16:55Z

@azamikram we would like to get this into the release do you have cycles to address feedback?

liu-cong · 2026-07-09T05:52:21Z

 ```text
-score = matchBlocks / totalBlocks
+score = 0
+score += matchLengthWeight * matchLengthRatio


pls also update the doc according with the renamed matchLengthScore and matchRatioScore.

Let's also have the formula in one line, score += add unnecessary cognitive overhead IMO.

I updated the doc and the PR description as well. Also add main results in the PR from the doc.

Signed-off-by: Azam Ikram <azamikram@google.com>

github-project-automation Bot added this to llm-d-router Jun 2, 2026

github-actions Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 2, 2026

github-actions Bot requested a review from ahg-g June 2, 2026 19:21

github-actions Bot added the kind/bug Categorizes issue or PR as related to a bug. label Jun 2, 2026

github-actions Bot requested a review from liu-cong June 2, 2026 19:21

github-actions Bot added kind/cleanup kind/documentation Categorizes issue or PR as related to documentation. kind/feature Categorizes issue or PR as related to a new feature. labels Jun 2, 2026

azamikram force-pushed the prefix-cache-scorer branch 2 times, most recently from da55286 to 33f74a1 Compare June 2, 2026 19:40

zetxqx reviewed Jun 2, 2026

View reviewed changes

Comment thread pkg/epp/framework/plugins/scheduling/scorer/prefix/plugin.go Outdated

azamikram force-pushed the prefix-cache-scorer branch from c6b69f5 to c528524 Compare June 2, 2026 21:21

liu-cong reviewed Jun 2, 2026

View reviewed changes

shmuelk reviewed Jun 4, 2026

View reviewed changes

Comment thread pkg/epp/framework/plugins/scheduling/scorer/prefix/README.md Outdated

shmuelk reviewed Jun 4, 2026

View reviewed changes

github-actions Bot added the hold PRs that are blocked on design, other features, release cycle, etc. label Jun 5, 2026

ahg-g reviewed Jun 9, 2026

View reviewed changes

zetxqx previously approved these changes Jul 2, 2026

View reviewed changes

Comment thread pkg/epp/framework/plugins/scheduling/scorer/prefix/plugin.go

liu-cong mentioned this pull request Jul 9, 2026

[Agentic] Optimized router config for agentic traces #1867

Open

liu-cong reviewed Jul 9, 2026

View reviewed changes

liu-cong removed kind/bug Categorizes issue or PR as related to a bug. hold PRs that are blocked on design, other features, release cycle, etc. kind/cleanup labels Jul 9, 2026

azamikram dismissed zetxqx’s stale review via 0752452 July 9, 2026 21:59

azamikram force-pushed the prefix-cache-scorer branch from bf0865c to 0752452 Compare July 9, 2026 21:59

azamikram requested a review from liu-cong July 9, 2026 22:03

liu-cong approved these changes Jul 9, 2026

View reviewed changes

github-actions Bot removed kind/feature Categorizes issue or PR as related to a new feature. kind/bug Categorizes issue or PR as related to a bug. labels Jul 9, 2026

Madhumasa84 mentioned this pull request Jul 10, 2026

feat: optimize router config guidelines for agentic workloads #1958

Open

Copilot AI mentioned this pull request Jul 10, 2026

handlers: guard latency metrics when timestamps are absent #1963

Closed

Add support for length-based scoring in prefix cache plugin

9914597

Signed-off-by: Azam Ikram <azamikram@google.com>

llm-d-router-release-notes Bot mentioned this pull request Jul 10, 2026

docs: add release-note fragment for PR #1459 #1967

Merged

	This plugin supports the following configuration parameters:
	This plugin supports the following optional configuration parameters:

		@@ -115,8 +138,19 @@ func (p Plugin) Score(ctx context.Context, _ fwksched.InferenceRequest, endpoi
		}

		if prefixMatchInfo, ok := info.(*attrprefix.PrefixCacheMatchInfo); ok {

Uh oh!

Conversation

azamikram commented Jun 2, 2026 • edited by liu-cong Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 2, 2026

Uh oh!

Uh oh!

liu-cong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liu-cong commented Jun 5, 2026

Uh oh!

shmuelk commented Jun 9, 2026

Uh oh!

ahg-g left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hexfusion commented Jun 12, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

azamikram commented Jun 2, 2026 •

edited by liu-cong

Loading