Motivation
BenchmarkMetrics (vllm/benchmarks/serve.py) reports output_throughput and total_token_throughput and nothing per-character. For English that distinction rarely matters. For scripts whose tokenizers are less efficient it inverts the ranking: tokens/s flatters exactly the models that make the reader wait longest.
Measured on one machine, one protocol, chat-scenario decode, Thai output (characters counted in U+0E00–U+0E7F only):
| served model |
Thai chars per output token |
| Qwen-SEA-LION-v4.5-27B |
3.18 |
| WangchanLION-v3-8B |
2.15 |
| OpenThaiGPT-1.6-72B |
1.85 |
| Typhoon-2.5-Qwen3-4B |
1.75 |
| OpenThaiGPT-ThaiLLM-8B |
1.36 |
That is a 2.3x spread across models a user might reasonably compare. Two models at the same tok/s can differ by more than a factor of two in delivered text; a benchmark table that shows only tok/s reports the faster-reading model as the slower one. The effect is not Thai-specific — it applies to any language whose tokenizer coverage is weaker than English, which is most of them.
Proposal
Add character-level counterparts to the existing throughput fields:
output_char_throughput and mean_chars_per_token in BenchmarkMetrics, computed from the decoded output text already available in the benchmark path.- Optional
--charset-range (or a named preset) to count only codepoints in a range, so mixed-language output can be scored on the target script rather than on incidental Latin/punctuation. Without it, count all non-whitespace characters.
- Print both in the summary table; include both in
--save-result JSON.
No change to existing fields or defaults — purely additive, so existing dashboards keep working.
Alternatives considered
- Leave it to users: possible today only by re-deriving from raw output text, which
bench serve does not save by default.
- Report a normalization factor instead (chars/token alone): less useful, because the reader wants delivered characters per second, not a ratio.
Prior art
bench serve already special-cases non-Latin behaviour elsewhere; and vLLM's own detokenizer tests carry a Burmese case precisely because multibyte scripts behave differently (#9625).
Willing to implement
Yes — I have a working implementation of this measurement in a downstream harness (Thai chars/s + chars/token, used to produce the table above) and can port it. Filing as an RFC first because the flag surface (--charset-range vs presets vs always-count-all) is a design choice I would rather have maintainers pick before I write the PR.
Measured on vLLM 0.25.1, NVIDIA GB10 (SM121); artifacts public at https://github.com/AGICAFET-LABS/thai-vllm
Disclosure: prepared with assistance from Claude (Anthropic); the numbers above are from real runs on the hardware named.
Motivation
BenchmarkMetrics(vllm/benchmarks/serve.py) reportsoutput_throughputandtotal_token_throughputand nothing per-character. For English that distinction rarely matters. For scripts whose tokenizers are less efficient it inverts the ranking: tokens/s flatters exactly the models that make the reader wait longest.Measured on one machine, one protocol, chat-scenario decode, Thai output (characters counted in U+0E00–U+0E7F only):
That is a 2.3x spread across models a user might reasonably compare. Two models at the same tok/s can differ by more than a factor of two in delivered text; a benchmark table that shows only tok/s reports the faster-reading model as the slower one. The effect is not Thai-specific — it applies to any language whose tokenizer coverage is weaker than English, which is most of them.
Proposal
Add character-level counterparts to the existing throughput fields:
output_char_throughputandmean_chars_per_tokeninBenchmarkMetrics, computed from the decoded output text already available in the benchmark path.--charset-range(or a named preset) to count only codepoints in a range, so mixed-language output can be scored on the target script rather than on incidental Latin/punctuation. Without it, count all non-whitespace characters.--save-resultJSON.No change to existing fields or defaults — purely additive, so existing dashboards keep working.
Alternatives considered
bench servedoes not save by default.Prior art
bench servealready special-cases non-Latin behaviour elsewhere; and vLLM's own detokenizer tests carry a Burmese case precisely because multibyte scripts behave differently (#9625).Willing to implement
Yes — I have a working implementation of this measurement in a downstream harness (Thai chars/s + chars/token, used to produce the table above) and can port it. Filing as an RFC first because the flag surface (
--charset-rangevs presets vs always-count-all) is a design choice I would rather have maintainers pick before I write the PR.Measured on vLLM 0.25.1, NVIDIA GB10 (SM121); artifacts public at https://github.com/AGICAFET-LABS/thai-vllm
Disclosure: prepared with assistance from Claude (Anthropic); the numbers above are from real runs on the hardware named.