You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
***`matchLengthScaleTokens`** is the scaling factor.
27
25
28
-
If `matchLengthWeight` is `0.0` (default), the score simplifies to just the `matchRatio`.
26
+
If `matchLengthWeight` is `0.0` (default), the score simplifies to just the `matchRatioScore`.
29
27
30
28
If the attribute is missing, has the wrong type, or `totalBlocks` is zero, the endpoint receives score `0`.
31
29
32
30
### Why the Quadratic Term?
33
31
34
-
The squaring of `matchLengthRatio` introduces a non-linearity motivated by the fact that attention computation grows quadratically as a function of prompt length. This ensures that the scoring function assigns a higher priority to longer requests, where the computational and latency savings of KV-cache reuse are significantly more critical.
32
+
The squaring of `matchLengthScore` introduces a non-linearity motivated by the fact that attention computation grows quadratically as a function of prompt length. This ensures that the scoring function assigns a higher priority to longer requests, where the computational and latency savings of KV-cache reuse are significantly more critical.
35
33
36
34
## Inputs Consumed
37
35
@@ -44,7 +42,7 @@ The attribute is typically produced by the approximate prefix cache data produce
44
42
| Parameter | Type | Description | Default |
45
43
| :--- | :--- | :--- | :--- |
46
44
|`matchLengthWeight`| float | Weight of the absolute match length in the score. Must be between `0.0` and `1.0`. |`0.0`|
47
-
|`matchLengthScaleTokens`| integer | The number of tokens used to normalize `matchLengthRatio`. |`8192`|
45
+
|`matchLengthScaleTokens`| integer | The number of tokens used to normalize `matchLengthScore`. |`8192`|
0 commit comments