Hi team,
I briefly mentioned this idea in Slack earlier. I noticed that the estimate backend in token-producer (pkg/epp/framework/plugins/requestcontrol/dataproducer/tokenizer/estimate.go) hardcodes bytesPerToken = 4 to pack raw input bytes into pseudo-tokens for prefix hashing.
For ASCII-dominant languages this is reasonable (~4 chars ≈ 1 token), but for Chinese (UTF-8, 3 bytes/char), each pseudo-token spans only ~1.33 characters. This undercounts tokens by ~25%, and more critically, causes 3-byte characters to frequently misalign with the 4-byte boundary. As a result, minor prompt edits can shift the hash-block boundaries and severely degrade cache hit rates.
I’d love to work on this by introducing a configurable bytesPerToken or language-prefixed presets in the estimate backend. I'll use llm-d-inference-sim locally to prototype and validate the cache hit rate improvement.
Could you please assign this issue to me? I'd be glad to submit a PR once the local simulation tests pass. Thanks!
Hi team,
I briefly mentioned this idea in Slack earlier. I noticed that the estimate backend in token-producer (
pkg/epp/framework/plugins/requestcontrol/dataproducer/tokenizer/estimate.go) hardcodesbytesPerToken = 4to pack raw input bytes into pseudo-tokens for prefix hashing.For ASCII-dominant languages this is reasonable (~4 chars ≈ 1 token), but for Chinese (UTF-8, 3 bytes/char), each pseudo-token spans only ~1.33 characters. This undercounts tokens by ~25%, and more critically, causes 3-byte characters to frequently misalign with the 4-byte boundary. As a result, minor prompt edits can shift the hash-block boundaries and severely degrade cache hit rates.
I’d love to work on this by introducing a configurable
bytesPerTokenor language-prefixed presets in the estimate backend. I'll usellm-d-inference-simlocally to prototype and validate the cache hit rate improvement.Could you please assign this issue to me? I'd be glad to submit a PR once the local simulation tests pass. Thanks!