Skip to content

Commit 0752452

Browse files
committed
Fix variable names
Signed-off-by: Azam Ikram <azamikram@google.com>
1 parent bed40b8 commit 0752452

2 files changed

Lines changed: 13 additions & 17 deletions

File tree

pkg/epp/framework/plugins/scheduling/scorer/prefix/README.md

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -14,24 +14,22 @@ The scorer computes a score between `0.0` and `1.0` for each endpoint using a we
1414
### Scoring Formula
1515

1616
```text
17-
score = 0
18-
score += matchLengthWeight * matchLengthRatio
19-
score += (1.0 - matchLengthWeight) * matchRatio
17+
score = matchLengthWeight*matchLengthScore + (1.0-matchLengthWeight)*matchRatioScore
2018
```
2119

2220
Where:
23-
* **`matchRatio`** = `matchBlocks` / `totalBlocks`
24-
* **`matchLengthRatio`** = `min(1.0, matchedTokens / matchLengthScaleTokens) ^ 2`
21+
* **`matchRatioScore`** = `matchBlocks` / `totalBlocks`
22+
* **`matchLengthScore`** = `min(1.0, matchedTokens / matchLengthScaleTokens) ^ 2`
2523
* `matchedTokens` = `matchBlocks` * `blockSize`
2624
* **`matchLengthScaleTokens`** is the scaling factor.
2725

28-
If `matchLengthWeight` is `0.0` (default), the score simplifies to just the `matchRatio`.
26+
If `matchLengthWeight` is `0.0` (default), the score simplifies to just the `matchRatioScore`.
2927

3028
If the attribute is missing, has the wrong type, or `totalBlocks` is zero, the endpoint receives score `0`.
3129

3230
### Why the Quadratic Term?
3331

34-
The squaring of `matchLengthRatio` introduces a non-linearity motivated by the fact that attention computation grows quadratically as a function of prompt length. This ensures that the scoring function assigns a higher priority to longer requests, where the computational and latency savings of KV-cache reuse are significantly more critical.
32+
The squaring of `matchLengthScore` introduces a non-linearity motivated by the fact that attention computation grows quadratically as a function of prompt length. This ensures that the scoring function assigns a higher priority to longer requests, where the computational and latency savings of KV-cache reuse are significantly more critical.
3533

3634
## Inputs Consumed
3735

@@ -44,7 +42,7 @@ The attribute is typically produced by the approximate prefix cache data produce
4442
| Parameter | Type | Description | Default |
4543
| :--- | :--- | :--- | :--- |
4644
| `matchLengthWeight` | float | Weight of the absolute match length in the score. Must be between `0.0` and `1.0`. | `0.0` |
47-
| `matchLengthScaleTokens` | integer | The number of tokens used to normalize `matchLengthRatio`. | `8192` |
45+
| `matchLengthScaleTokens` | integer | The number of tokens used to normalize `matchLengthScore`. | `8192` |
4846

4947
### Example
5048

pkg/epp/framework/plugins/scheduling/scorer/prefix/plugin.go

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -154,18 +154,16 @@ func (p *Plugin) Score(ctx context.Context, _ *fwksched.InferenceRequest, endpoi
154154
continue
155155
}
156156

157-
matchLengthRatio := 0.0
158-
matchRatio := float64(matchBlocks) / float64(totalBlocks)
157+
matchRatioScore := float64(matchBlocks) / float64(totalBlocks)
159158
blockSize := prefixMatchInfo.BlockSizeTokens()
160-
if blockSize > 0 && p.matchLengthScaleTokens > 0 {
159+
matchLengthScore := 0.0
160+
// Calculate matchLengthScore when match length is considered
161+
if p.matchLengthWeight > 0.0 && blockSize > 0 {
161162
// (matchBlocks * blockSize / matchLengthScaleTokens) ^ 2
162-
// Capped at 1.0 as the normalized score term cannot be greater than 1.
163-
ratio := float64(matchBlocks) * float64(blockSize) / float64(p.matchLengthScaleTokens)
164-
matchLengthRatio = math.Min(1.0, ratio)
165-
matchLengthRatio *= matchLengthRatio
163+
normalizedMatchLength := math.Min(1.0, float64(matchBlocks)*float64(blockSize)/float64(p.matchLengthScaleTokens))
164+
matchLengthScore = normalizedMatchLength * normalizedMatchLength
166165
}
167-
scores[endpoint] += p.matchLengthWeight * matchLengthRatio
168-
scores[endpoint] += (1.0 - p.matchLengthWeight) * matchRatio
166+
scores[endpoint] += p.matchLengthWeight*matchLengthScore + (1.0-p.matchLengthWeight)*matchRatioScore
169167
}
170168
return scores
171169
}

0 commit comments

Comments
 (0)