docs/configs: replace deprecated maxPrefixBlocksToMatch and blockSize with maxPrefixTokensToMatch and blockSizeTokens#2072
Open
liu-cong wants to merge 7 commits into
Open
Conversation
… with maxPrefixTokensToMatch and blockSizeTokens Update sample deployment configurations, test fixtures, and documentation to use non-deprecated parameter names. Signed-off-by: Cong Liu <conliu@google.com>
github-actions
Bot
added
the
size/M
…Match * blockSizeTokens Scale maxPrefixTokensToMatch accurately based on the block count and token block size. Signed-off-by: Cong Liu <conliu@google.com>
…ith maxPrefixTokensToMatch Signed-off-by: Cong Liu <conliu@google.com>
…cksToMatch with maxPrefixTokensToMatch Signed-off-by: Cong Liu <conliu@google.com>
…nd blockSizeTokens Signed-off-by: Cong Liu <conliu@google.com>
liu-cong
marked this pull request as ready for review
July 18, 2026 04:12
…TokensToMatch Signed-off-by: Cong Liu <conliu@google.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates llm-d Router documentation and configuration fixtures to stop using deprecated prefix-cache sizing fields (maxPrefixBlocksToMatch, blockSize) in favor of token-based equivalents (maxPrefixTokensToMatch, blockSizeTokens), aligning configs with the supported schema ahead of the planned deprecation removal.
Changes:
- Replace
maxPrefixBlocksToMatchwithmaxPrefixTokensToMatchacross deploy configs and e2e fixture YAML. - Update docs to reference token-based caps (including benchmark tables and config examples).
- Update precise/approx prefix-cache examples to use
blockSizeTokenswhere applicable.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 19 comments.
Show a summary per file
| File | Description |
|---|---|
| test/integration/epp/e2e_config_smoke_test.go | Mirrors e2e YAML fixtures for hermetic startup-parse smoke coverage, updated to token-based prefix caps. |
| test/e2e/configs_test.go | E2E fixture configs updated to token-based prefix caps. |
| docs/operations.md | Updates sizing guidance and benchmark tables to reference maxPrefixTokensToMatch. |
| docs/disaggregation.md | Updates disaggregation config examples to use token-based prefix cap field names. |
| docs/architecture.md | Updates architecture example config to token-based prefix cap/block sizing field names. |
| deploy/config/sim-pd-epp-config.yaml | Simulator P/D config updated to token-based prefix cap. |
| deploy/config/sim-epp-no-hit-lru.yaml | Simulator “no-hit-lru” config updated to token-based prefix cap/block sizing field names. |
| deploy/config/sim-epp-config.yaml | Simulator config updated to token-based prefix cap. |
| deploy/config/pd-epp-config.yaml | Sample P/D config updated to token-based prefix cap. |
Comment on lines
8
to
+12
| indexerConfig: | ||
| tokenProcessorConfig: | ||
| blockSize: 5 | ||
| blockSizeTokens: 5 | ||
| kvBlockIndexConfig: | ||
| maxPrefixBlocksToMatch: 256 | ||
| maxPrefixTokensToMatch: 1280 |
Comment on lines
153
to
+157
| indexerConfig: | ||
| tokenProcessorConfig: | ||
| blockSize: 5 | ||
| blockSizeTokens: 5 | ||
| kvBlockIndexConfig: | ||
| maxPrefixBlocksToMatch: 256 | ||
| maxPrefixTokensToMatch: 1280 |
Comment on lines
278
to
283
| - type: prefix-cache-scorer | ||
| parameters: | ||
| autoTune: false | ||
| blockSizeTokens: 5 | ||
| maxPrefixBlocksToMatch: 256 | ||
| maxPrefixTokensToMatch: 1280 | ||
| lruCapacityPerServer: 31250 |
Comment on lines
341
to
346
| - type: prefix-cache-scorer | ||
| parameters: | ||
| autoTune: false | ||
| blockSizeTokens: 5 | ||
| maxPrefixBlocksToMatch: 256 | ||
| maxPrefixTokensToMatch: 1280 | ||
| lruCapacityPerServer: 31250 |
| - **Rule of Thumb**: Allocate **0.5 to 1.0 CPU cores per request/second** of expected throughput for large agentic workloads (approximately 100k input / 1k output tokens). | ||
| - **Scaling Behavior**: CPU utilization scales linearly with the request rate, and increases with both the input prompt size and output token length. | ||
| - **Prefix Matching Overhead**: Increasing the `maxPrefixBlocksToMatch` parameter increases EPP CPU utilization. At lower throughputs, a large prefix block limit (such as 6250 blocks) can increase EPP CPU utilization by over 100% compared to a small limit (256 blocks) due to the overhead of searching and matching blocks. | ||
| - **Prefix Matching Overhead**: Increasing the `maxPrefixTokensToMatch` parameter increases EPP CPU utilization. At lower throughputs, a large prefix limit (such as 100,000 tokens / 6,250 blocks with `blockSizeTokens: 16`) can increase EPP CPU utilization by over 100% compared to a small limit (4,096 tokens / 256 blocks) due to the overhead of searching and matching prefix blocks. |
…-block cap with 64-token min block size Signed-off-by: Cong Liu <conliu@google.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace remaining usages of deprecated fields
maxPrefixBlocksToMatchandblockSizewith their supported token-based equivalents (maxPrefixTokensToMatchandblockSizeTokens) across documentation, deployment configs, and test fixtures.This prepares the repo for eventual removal of these fields in v0.11.0 (zsh.N + 2$ policy after v0.9.0 deprecation).