Token-count estimation without a tokenizer. 22 character-class counters, plus a 23rd that the multilingual table prices, one left-to-right pass over a string, and a table of coefficients with one column per tokenizer generation.
tokens = max(1, round(sum over buckets of count[bucket] * coefficient[bucket]))
That is the whole algorithm. No vocabulary file, no dependency, no network, no cold start.
novocab.py is a reference implementation in stdlib Python, and the prose below is complete enough
that you can write your own in any language without reading it.
The tables cover 5 tokenizer generations: Claude 4.7 and later, Claude before 4.7, OpenAI
o200k_base, OpenAI cl100k_base, and the SentencePiece vocabulary shared by every generative
Gemini and by the Gemma open weights.
Two coefficient tables ship. The default is fitted on 808 chunks of source code, JSON, technical markdown and Wikipedia in twelve languages. Mean absolute percentage error, out of fold, on those same rows:
| chars/4 | 29 MB o200k BPE build | novocab | |
|---|---|---|---|
| install size | 0 MB | 29 MB | 0 MB |
| floats | 0 | 2 | 22 |
| Claude 4.7+ | 52.25% | 11.11% | 5.26% |
| Claude pre-4.7 | 43.14% | 20.35% | 5.29% |
GPT-4o, GPT-5.x (o200k) |
28.35% | 5.34% | 6.31% |
GPT-4, GPT-3.5 (cl100k) |
41.71% | 40.64% | 5.29% |
| Gemini 2.5 to 3.7 | 29.00% | 10.90% | 6.04% |
The 29 MB column is a real o200k_base BPE build plus two scalars. It wins on the one generation
where it is the tokenizer and loses on the other four.
The default table is not a general multilingual estimator. Its calibration corpus holds two Latin-script languages, English and Vietnamese. On the 24 Latin-script languages of a held-out 39-language Wikipedia set that the corpus does not contain, the default reads 10.4% to 42.0% against Claude 4.7+, median 27.1%, with a one-sided bias near -20% that does not cancel when you sum. German reads 28.50% and Welsh 42.03%. English on the same instrument reads 10.55%.
The multilingual table exists for that text. It adds nine continental European languages to the fit and prices a twenty-third bucket. Held out, on rows no fit has seen:
| held out, MAPE | Claude 4.7+ | Claude pre-4.7 | o200k | cl100k | Gemini |
|---|---|---|---|---|---|
| European prose, 20,895 chunks, default | 20.70 | 25.20 | 21.51 | 21.87 | 22.70 |
| European prose, 20,895 chunks, multilingual | 8.10 | 9.06 | 9.48 | 8.08 | 8.81 |
| Wikipedia in 39 languages, 94,274 chunks, default | 19.50 | 22.45 | 20.02 | 20.92 | 20.77 |
| Wikipedia in 39 languages, multilingual | 10.48 | 11.23 | 10.77 | 9.97 | 11.39 |
Neither table dominates the other. The trade, with the languages that lose it named, is under the two coefficient tables.
Every rung is a real implementation scored on the same 808 rows under one protocol. Third-party estimators have no fitting step on this corpus, so they are evaluated straight over all 808 rows with no train split, which is the more generous treatment. novocab is the only fitted rung and is quoted out of fold.
| rung | Claude 4.7+ | o200k | what it is |
|---|---|---|---|
chars/4 |
52.25 | 28.35 | the constant OpenAI's help centre prints for English |
chars/3.5 |
45.46 | 27.38 | the constant Anthropic's glossary prints for English |
| Claude Code | 52.25 | 28.35 | CHARS_PER_TOKEN = 4 |
| opencode | 52.19 | 28.32 | Math.max(0, Math.round(input.length / 4)) |
| pi | 52.02 | 28.19 | Math.ceil(text.length / 4) |
| OpenClaw | 38.94 | 16.76 | chars / 4 with CJK characters weighted by class |
| tokenx 2.1.0 | 40.63 | 13.66 | 2 kB segment-and-rule table, calibrated for o200k |
| ai-tokenizer 1.0.6 | 31.22 | 0.00 | 30 MB BPE tables, one vocabulary per provider |
| bpe-lite 0.5.1 | 29.08 | 0.00 | 494 kB BPE, Claude-2-era Anthropic vocabulary |
| novocab, default profile | 5.26 | 6.31 | 22 counters, 22 floats, 0 MB |
| ctok | 0.00 | exact Claude reconstruction, 808/808 | |
| tiktoken, gpt-tokenizer | 0.00 | o200k_base itself |
The same ladder against o200k_base is in
assets/benchmark-o200k.png, where two of the packages drop to 0.00%
because on that target they are the tokenizer. Every bar on every chart on this page is the default
profile.
Three notes on the comparands. The two BPE packages reproduce o200k_base exactly on the same
rows, which is the instrument check that makes their Claude numbers readable; both document their
Claude support as approximate, and bpe-lite's own accuracy report names its Anthropic provider as a
reverse-engineered Claude-2-era vocabulary. tokenx states in its README that it is calibrated
against o200k_base and never claims Claude, so its o200k row is its own target; tuned in its own
favour with the four custom language rules its README prescribes and defaultCharsPerToken swept
from 1.00 to 20.00, it reaches 9.04 on o200k. The coding-harness rows are budget heuristics that
normally sit behind provider-reported usage counts, so their figures bound the error when the
estimate is all there is.
from novocab import estimate, bucket_counts
estimate("hello world") # 3 (Claude 4.7+)
estimate("hello world", "o200k") # 2
estimate(open("README.md").read(), "gemini")
estimate(german_text, profile="multilingual")
bucket_counts("parseConfig();")["camel"] # 1$ python3 novocab.py notes.txt
1284
$ cat src/*.rs | python3 novocab.py -g o200k
$ python3 novocab.py -g claude-legacy --buckets notes.txt
$ python3 novocab.py --profile multilingual artikel.txt
Generation names are claude, claude-legacy, o200k, cl100k, gemini. Profile names are
default and multilingual. The empty string returns 0, the one exception to the max(1, ...)
floor.
Images are priced separately and exactly, by geometry instead of by fit:
from novocab import estimate_image_tokens
estimate_image_tokens(1170, 2532) # 3825 a phone screenshot
estimate_image_tokens(1170, 2532, "gemini") # 1078
estimate_image_tokens(1170, 2532, "openai") # 2304$ python3 novocab.py --image 2560x1440
4787
$ python3 novocab.py --image 2560x1440 -g gemini
1100
Walk the string one code point at a time. Not bytes, not UTF-16 units. Astral characters count once.
Define these as explicit code-point ranges. Do not use your language's isspace, isdigit, \s or
\p{Nd}. Those disagree across languages in ways that break the numbers quietly: Python's
isdigit() is true for U+00B2 SUPERSCRIPT TWO and for circled digits, and JavaScript's \s matches
U+FEFF where Python's isspace() does not. Name the code points and the question goes away.
| class | code points |
|---|---|
SPACE |
0009-000D, 0020, 0085, 00A0, 1680, 2000-200A, 2028-2029, 202F, 205F, 3000 |
DIGIT |
0030-0039, 0660-0669, 06F0-06F9 |
CJK |
2E80-2EFF, 3040-309F, 30A0-30FF, 3100-312F, 3130-318F, 31F0-31FF, 3400-4DBF, 4E00-9FFF, A000-A4CF, AC00-D7AF, F900-FAFF, 20000-2FA1F |
KANA |
3040-30FF, 31F0-31FF, a subset of CJK |
HANGUL |
3130-318F, AC00-D7AF, a subset of CJK |
CJK_PUNCT |
3000-303F, FE30-FE4F, FF00-FFEF |
CYRILLIC |
0400-052F, 2DE0-2DFF, A640-A69F |
ARABIC |
0600-06FF, 0750-077F, 08A0-08FF, FB50-FDFF, FE70-FEFF |
HEBREW |
0590-05FF, FB1D-FB4F |
OTHER_SCRIPT |
0370-03FF and 1F00-1FFF (Greek); 0E00-0E7F (Thai); 0900-097F and A8E0-A8FF (Devanagari) |
LATIN_ACCENTED |
00C0-00FF, 0100-017F, 0180-024F, 1E00-1EFF |
PUNCT_RUN |
0021-0026, 0028-002F, 003A-0040, 005B-0060, 007B-007E |
SYMBOL |
2190-21FF, 2200-22FF, 2300-23FF, 2460-24FF, 2500-25FF, 2600-27BF, 2900-297F, 2B00-2BFF |
EMOJI |
1F000-1FAFF, a range that already contains regional indicators and skin-tone modifiers |
VARIATION |
FE00-FE0F, E0100-E01EF |
ZWJ |
200D |
MARK |
Unicode general category Mn. Adding Mc was tested and is worse |
LETTER |
Unicode general category L |
UPPER |
Unicode general category Lu |
KANA and HANGUL carve CJK into three disjoint parts; whatever is in CJK and in neither
subset is Han proper. SPACE and CJK_PUNCT overlap at U+3000, and the whitespace test runs first,
so U+3000 is whitespace. Both orderings are part of the definition.
Write the ranges as escapes or as hex integers, never as literal characters. The literal form of the
CJK class contains U+F900, which NFC-normalizes to U+8C48. A pipeline that normalized such a
source would rewrite the range start and widen the class by 10,368 code points, the whole Private
Use Area included. Escapes cannot be normalized.
Eight distinct script classes break words, not four: Cyrillic, Arabic, Hebrew, Greek, Thai, Devanagari, CJK, and everything else. Greek, Thai and Devanagari share one coefficient and are still three separate classes, because a Greek run followed immediately by a Thai run has to count as two words.
At each position, take the first rule that matches.
- Whitespace. If the code point is in
SPACE, consume the whole run of consecutiveSPACEcode points. Add 1 tows, since a run of any length is one unit, and add the number of U+000A inside that run tonl. - CJK. If in
CJK: add 1 tokanaif it is inKANA, else 1 tohangif it is inHANGUL, else 1 tocjk. If inCJK_PUNCT, add 1 tocjk. One character, one unit. - Combining mark. If in
MARK, add 1 tomark. Marks stand alone. They never count toward a word's length. - Digits. If in
DIGIT, consume the whole run ofddigits and addceil(d / 3)tonum. - Word. If in
LETTER, consume the maximal run of letters in the same script class. A script change ends the word. LetLbe the run length.CYRILLICaddsLtocyr.ARABICaddsLtoarb.HEBREWaddsLtoheb.OTHER_SCRIPTaddsLtoother.- Otherwise it is a Latin-class word:
- Accent surcharge first, before anything else. Add 1 to
lataccfor every code point in the run that is inLATIN_ACCENTED. This happens whether or not the word turns out to be ALL-CAPS, and the letter still counts toward the word's length below.lataccis an extra charge, not a reassignment. - Marker surcharge, also across the whole run. Add to
latMarkonce for each of the six patterns in the marker table that the word matches. ALL-CAPS words included, same aslatacc. - ALL-CAPS if
L >= 2, at least one letter is cased, and every cased letter isUPPER. AddLtocaps. - Otherwise add 1 to exactly one of
word2(L <= 2),word35(L3 to 5),word68(L6 to 8), or to none of them whenL >= 9; and whenL >= 9addLtolongChars. - In the non-ALL-CAPS case, also add 1 to
camelfor every position where anUPPERletter follows a cased non-UPPERletter. That is thelCboundary inparseConfigand inHTTPServer.
- Accent surcharge first, before anything else. Add 1 to
- Two guards. A Latin-class run stops at U+2000, so an alphabetic code point at or above that boundary never joins a Latin word. And if the run comes out empty, as it does for U+2102 DOUBLE-STRUCK CAPITAL C, treat that code point as one wide punctuation character and advance one position. Without the second guard the pointer never moves.
- Symbols and emoji. If it is
ZWJ, or inVARIATION,EMOJIorSYMBOL, add 1 toemoji. One coefficient covers all of them. - ASCII punctuation, consumed as a run. If the code point is in
PUNCT_RUN, consume the whole run of consecutivePUNCT_RUNcode points. Add 1 topunctfor each code point in the run, and add 1 topunctRunfor the run as a whole, however long it is. EveryPUNCT_RUNcode point is below 128, so this rule never touchespunctWide. - Everything else is punctuation. Add 1 to
punct. If the code point is at or above 128, add 1 topunctWideas well.
PUNCT_RUN is every ASCII punctuation and symbol character except U+0027 APOSTROPHE, which is
exactly the gap between the first two ranges. The apostrophe is excluded on purpose. It lives inside
English words instead of between them, and folding it in makes every contraction look like a
punctuation boundary.
A punctRun is a maximal run of consecutive code points all in PUNCT_RUN. A run of nine
characters scores the same 1 as a run of one.
| input | punctRun |
punct |
|---|---|---|
)) |
1 | 2 |
); |
1 | 2 |
), |
1 | 2 |
=> |
1 | 2 |
}); |
1 | 3 |
""" |
1 | 3 |
), ( |
2 | 3 |
a.b.c |
2 | 2 |
don't |
0 | 1 |
), ( is two runs because the space between them is not in the class. a.b.c is two runs of one
character each, since the letters break them. don't is zero runs, and its single punct charge is
the apostrophe falling through to rule 8.
The counter exists because real tokenizers merge adjacent punctuation. =>, """, ), and });
are each fewer pieces than their character count, and a strictly per-character price cannot
represent that. Adding it is worth +0.41 to +1.16 MAPE points depending on generation, on all five
at once, and the per-character punctuation price collapses by half to three quarters when it goes
in.
b is the counts, c is the coefficient column.
tokens = max(1, round(
c.ws * b.ws
+ c.nl * b.nl
+ c.cjk * b.cjk // Han plus CJK punctuation
+ c.kana * b.kana
+ c.hang * b.hang
+ c.num * b.num
+ c.caps * b.caps
+ c.punct * (b.punct - b.punctWide)
+ c.punctWide * b.punctWide
+ c.punctRun * b.punctRun
+ c.emoji * b.emoji
+ c.word2 * b.word2
+ c.word35 * b.word35
+ c.word68 * b.word68
+ c.longChars * b.longChars
+ c.camel * b.camel
+ c.mark * b.mark
+ c.cyr * b.cyr
+ c.latacc * b.latacc
+ c.arb * b.arb
+ c.heb * b.heb
+ c.other * b.other
+ c.latMark * b.latMark // 0 in the default table, which has no entry for it
))
Four surcharges are the only places a code point is charged twice, and all four are deliberate:
punctWide on top of punct, punctRun on top of punct, and latacc and latMark on top of
the word buckets. Note the subtraction on the punct line. The counter puts non-ASCII punctuation
in both punct and punctWide; the fit saw two disjoint classes, so the sum has to take the wide
characters back out of the ASCII price. Everything else partitions, and the partition is asserted on
all 808 chunks by the parity harness.
Picking between them is the one configuration decision on this page that changes accuracy by more than a point.
estimate(text) # default, 22 buckets
estimate(text, profile="multilingual") # 23 buckets, nine more languagesThe default table:
| bucket | what it counts | Claude 4.7+ | Claude pre-4.7 | o200k | cl100k | Gemini |
|---|---|---|---|---|---|---|
ws |
whitespace runs | 0.15082 | 0.21966 | 0.20807 | 0.26741 | 0.05273 |
nl |
newlines inside those runs | 0.88951 | 1.09437 | 0.87552 | 0.98286 | 1.42431 |
cjk |
Han plus CJK punctuation | 1.14050 | 1.13777 | 0.85869 | 1.29798 | 0.70727 |
kana |
hiragana and katakana | 0.81266 | 0.81275 | 0.78964 | 1.03003 | 0.46900 |
hang |
Hangul | 1.33051 | 1.31089 | 0.73488 | 1.23908 | 0.70079 |
num |
digit groups of at most 3 | 1.64130 | 1.57510 | 1.27501 | 0.84841 | 2.04923 |
caps |
letters inside ALL-CAPS runs | 0.88050 | 0.39795 | 0.17197 | 0.17286 | 0.25771 |
punct |
ASCII punctuation characters | 0.30109 | 0.13921 | 0.10887 | 0.15759 | 0.11437 |
punctWide |
non-ASCII punctuation | 0.30750 | 0.24288 | 0.00000 | 1.68743 | 0.07056 |
punctRun |
one per ASCII punctuation run | 0.94150 | 0.97765 | 0.63549 | 0.56427 | 0.87915 |
emoji |
emoji and symbol code points | 5.67236 | 4.88216 | 3.90532 | 2.47126 | 3.58867 |
word2 |
Latin words of 1 to 2 letters | 0.42890 | 0.38281 | 0.91323 | 1.11275 | 0.85964 |
word35 |
Latin words of 3 to 5 letters | 1.25169 | 0.92090 | 0.88736 | 0.81344 | 0.89277 |
word68 |
Latin words of 6 to 8 letters | 2.06196 | 1.09653 | 0.82891 | 0.59397 | 1.29285 |
longChars |
letters in Latin words of 9+ | 0.32142 | 0.11396 | 0.14146 | 0.15955 | 0.09807 |
camel |
CamelCase internal boundaries | 1.81557 | 1.84319 | 0.90845 | 0.37484 | 1.15128 |
mark |
combining marks | 1.71751 | 1.73718 | 0.63148 | 1.03969 | 0.20149 |
cyr |
Cyrillic letters | 0.37246 | 0.36422 | 0.24104 | 0.47875 | 0.24213 |
latacc |
accented Latin letters | 1.09752 | 1.12890 | 0.11085 | 0.97388 | 0.09373 |
arb |
Arabic letters | 0.73599 | 0.72180 | 0.32115 | 0.76789 | 0.37441 |
heb |
Hebrew letters | 0.77411 | 0.76283 | 0.41437 | 1.12564 | 0.51904 |
other |
Greek, Thai, Devanagari letters | 0.72418 | 0.71339 | 0.37851 | 0.99794 | 0.40764 |
The multilingual table. Six floats marked F are not refitted and carry the default column's
value exactly; the reason is in where the algorithm came from.
| bucket | Claude 4.7+ | Claude pre-4.7 | o200k | cl100k | Gemini |
|---|---|---|---|---|---|
ws F |
0.15082 | 0.21966 | 0.20807 | 0.26741 | 0.05273 |
nl F |
0.88951 | 1.09437 | 0.87552 | 0.98286 | 1.42431 |
cjk |
1.14310 | 1.13996 | 0.85058 | 1.29043 | 0.70238 |
kana |
0.81086 | 0.81084 | 0.79129 | 1.03156 | 0.46972 |
hang |
1.33073 | 1.31086 | 0.73318 | 1.23761 | 0.69969 |
num |
1.52637 | 1.47540 | 1.49297 | 1.05524 | 2.15797 |
caps |
0.82205 | 0.38407 | 0.15776 | 0.13703 | 0.21536 |
punct F |
0.30109 | 0.13921 | 0.10887 | 0.15759 | 0.11437 |
punctWide F |
0.30750 | 0.24288 | 0.00000 | 1.68743 | 0.07056 |
punctRun F |
0.94150 | 0.97765 | 0.63549 | 0.56427 | 0.87915 |
emoji F |
5.67236 | 4.88216 | 3.90532 | 2.47126 | 3.58867 |
word2 |
0.73601 | 0.59266 | 0.50323 | 0.67849 | 0.70466 |
word35 |
1.34254 | 0.84269 | 0.36867 | 0.44266 | 0.43391 |
word68 |
1.82737 | 0.98384 | 1.69633 | 1.30123 | 1.96863 |
longChars |
0.34596 | 0.17420 | 0.18338 | 0.19689 | 0.15664 |
camel |
0.98601 | 1.05661 | 0.61205 | 0.00000 | 0.64534 |
mark |
1.71420 | 1.73205 | 0.62055 | 1.02896 | 0.19169 |
cyr |
0.37274 | 0.36438 | 0.23960 | 0.47736 | 0.24137 |
latacc |
0.92756 | 1.11371 | 0.54693 | 1.30033 | 0.44270 |
arb |
0.73575 | 0.72164 | 0.32059 | 0.76736 | 0.37394 |
heb |
0.77437 | 0.76303 | 0.41248 | 1.12379 | 0.51788 |
other |
0.72405 | 0.71322 | 0.37746 | 0.99686 | 0.40679 |
latMark |
3.43702 | 2.70713 | 1.30142 | 2.10752 | 1.33791 |
The default was fitted 2026-08-29, the multilingual table 2026-08-30. Store the date next to any estimate you keep. The floats expire, and an estimate with no record of which coefficients produced it cannot be audited later.
Five decimal places is a reproducibility artefact, not five significant digits. Copy the floats exactly, because that is what makes your implementation agree with this one bit for bit, and do not read the digits as measurement. A cluster bootstrap puts the standard error of the tightest coefficient at 0.0101 and the largest at 3.78, so none of the published floats is pinned past its first decimal place. The measurement is under earned precision.
A coefficient of 0.00000 is not a rounding artefact. Fitting is non-negative least squares, so zero
means the fit found no use for that bucket on that generation. Two land there across both tables,
punctWide on o200k and camel on cl100k. The near-zeros carry more information: latacc reads
0.09373 on Gemini and 0.11085 on o200k against 0.97 to 1.13 on the other three. Those two tokenizers
have real merges for accented Latin, and the other three pay roughly one token per accented
character, which is byte-level fallback with no merges at all.
| column | applies to |
|---|---|
claude |
claude-opus-4-7, claude-opus-4-8, claude-opus-5, claude-sonnet-5, claude-fable-5, measured token-identical on 81,744 tokens |
claude-legacy |
claude-sonnet-4-5, claude-haiku-4-5 |
o200k |
GPT-4o, GPT-4.1, GPT-5.x, o1/o3/o4-mini. tiktoken maps the gpt-5 prefix to o200k_base, and o200k_harmony produced byte-identical counts on 371,969 tokens, so there is no newer OpenAI encoding to target |
cl100k |
GPT-4, GPT-3.5-turbo, text-embedding-3-* |
gemini |
every generative Gemini the API serves: 2.5-flash, 3-flash-preview, 3.1-pro-preview, 3.1-flash-lite, 3.5-flash, 3.5-flash-lite, 3.6-flash, 3.7-flash, the -latest aliases, the omni, transcribe, robotics, computer-use, deep-research and image heads, and the Gemma 4 open weights served beside them. 21 model ids, byte-identical over a 123-probe battery. Not gemini-embedding-* |
One feature set serves all five columns and only the floats differ. That is measured. Giving each generation the feature list that helps that generation was scored against the control of one shared set. The shared set wins on both Claude columns by 0.31 and 0.74 points. It loses on the two OpenAI ones by 0.20 and 0.28, both under the 0.30 decision bar.
The twenty-third bucket. Six letter patterns, each with a minimum word length at which it is allowed to fire:
| pattern | fires when the word | minimum length |
|---|---|---|
#r |
contains an accented letter immediately followed by r |
1 |
^k |
starts with k |
6 |
a$ |
ends in a |
5 |
ei |
contains the pair ei anywhere |
6 |
i$ |
ends in i |
5 |
en$ |
ends in en |
6 |
^ and $ anchor the word. # is any letter the ASCII fold leaves outside a-z, which on
Latin-script text is an accented letter. Count the Latin-script words matching each pattern and sum,
so a word scores once per pattern it carries and at most 6 in total. Length is code points in the
run, the same length the word buckets are cut on, and a minimum of 1 admits every word. Case folding
is ASCII A-Z only, not str.lower(), which can change a string's length and can fold an accented
letter onto an ASCII one. The word run is the run every other Latin word bucket uses, ALL-CAPS words
included, ending at a combining mark exactly as they do.
Per 100 running Latin words, the six fire 16.7 times on held-out European Wikipedia, 13.9 on Dutch,
3.1 on English, 2.4 on Vietnamese, 1.3 on held-out source code and 1.2 on the calibration corpus.
Per pattern: en$ 11.6 on German and 8.8 on Dutch against 0.4 on English; a$ 13.7 on Italian and
11.4 on Polish; i$ 8.3 on Italian and on Turkish; ei 6.1 on German; ^k 5.2 on Turkish; #r
5.5 on Turkish.
The length gates are load-bearing. Ungated, i$ fires on 8.9% of running Vietnamese words against a
3.6% median across the eight European target languages, and a Vietnamese chunk carries 737 Latin
words against a 354 corpus average. Pin the whole default table, fit one marker float at Europe
scale, and an ungated set takes the Vietnamese stratum from 5.66 to 18.82 MAPE. The gated set takes
it to 6.50. A minimum of 5 code points removes 88% of the Vietnamese word-final-i mass and keeps
48% to 78% of the European, because Vietnamese syllables are short and the European target words are
not.
Held out, Claude 4.7+:
| default | multilingual | ||
|---|---|---|---|
| European Wikipedia, 20,895 chunks | 20.70 | 8.10 | multilingual by 12.6 |
| European literary prose, 192 chunks | 21.40 | 8.27 | multilingual by 13.1 |
| Wikipedia in 39 languages, 94,274 chunks | 19.50 | 10.48 | multilingual by 9.0 |
| the 1,287-chunk validation set | 11.82 | 6.72 | multilingual by 5.1 |
| English Wikipedia, 4,711 chunks | 10.56 | 8.17 | multilingual by 2.4 |
| Vietnamese Wikipedia, 2,670 chunks | 8.57 | 7.68 | multilingual by 0.9 |
| strings under 250 characters | 7.22 | 7.07 | multilingual by 0.15 |
| source code, 13,721 chunks | 7.77 | 7.75 | even |
| the calibration corpus, out of fold | 5.40 | 5.32 | even |
| Vietnamese, 64 validation chunks | 4.38 | 4.58 | default by 0.2 |
| Hungarian Wikipedia, 2,643 chunks | 10.41 | 15.81 | default by 5.4 |
For continental European prose and for mixed-language encyclopedic text, use the multilingual profile. On Claude 4.7+ it is at worst even on every published cut, and 29 of the 39 languages of the held-out Wikipedia set improve under it. On the other three generations the picture is a trade: o200k English goes 6.23 to 8.00, o200k source code 8.93 to 11.17, Gemini English 5.29 to 8.05. Keep the default for CJK-dominant text, for Hungarian, and on the non-Claude generations for English, source code and short strings.
One sentence, measured. "Physics is a natural science that investigates the fundamental phenomena of nature." is 83 characters and costs 26 Claude 4.7+ tokens; both profiles say 25 and 30. The same
sentence in German is 92 characters, costs 42, and the default says 32 where the
multilingual profile says 37.
CJK punctuation and fullwidth forms cost about 3 Claude tokens each against an ideograph's 1, so
splitting them out of cjk is mechanically justified, and the refitted cjkPunct lands 1.8x to
3.9x above the ideograph price on every generation.
| bucket | Claude 4.7+ | Claude pre-4.7 | o200k | cl100k | Gemini |
|---|---|---|---|---|---|
cjk (Han only) |
1.00494 | 1.01816 | 0.76445 | 1.23216 | 0.56341 |
cjkPunct (3000-303F, FE30-FE4F, FF00-FFEF) |
2.56856 | 2.39782 | 1.84569 | 1.99136 | 2.22289 |
It is off by default because it is worth +0.17 / +0.14 / -0.10 / -0.19 / +0.21 points, under the bar everywhere and negative on both OpenAI columns. Turn it on if your text is dense in fullwidth punctuation.
Image pricing is geometry at all three vendors, so the formulas here are exact and the only thing
that can go wrong with them is a vendor changing a constant. Measured 2026-08-29 against
count_tokens (Anthropic), countTokens (Google) and usage.prompt_tokens (OpenAI). Content,
codec, colour depth, alpha, EXIF orientation and GIF frame count moved no vendor's count by a single
token. A 12 kB blank PNG and a 7 MB noise PNG of the same pixel dimensions cost the same, and an
animated GIF is priced as one frame. Every formula is transpose-invariant.
Anthropic: a patch grid with two caps.
pw = ceil(w / 28)
ph = ceil(h / 28)
tokens = pw * ph + C
If the long edge exceeds 28 * axis_max pixels the image is resized to exactly that, both
dimensions rounded half-up to whole pixels. If the resulting grid still exceeds the total patch
budget, the price is the largest aspect-preserving downscale that fits, which always snaps one axis
onto a multiple of 28.
| generation | models | per axis | total budget | constant | ceiling |
|---|---|---|---|---|---|
anthropic |
opus-4-8, opus-5, sonnet-5, fable-5 | 92 (2576 px) | 4784 | +3 | 4787 |
anthropic-legacy |
sonnet-4-5, haiku-4-5 | 56 (1568 px) | 1568 | +4 | 1572 |
4784 patches is exactly the grid of a 2560x1440 frame, which is also the most expensive image that exists on this vendor. The widely cited 1568 px long-edge cap does not apply to the current generation, so a 2000x200 image is not resized and costs the full 579 tokens. And the price is not monotone in area: 4000x2000 is 8 megapixels and costs 4235, while 2560x1440 is 3.7 megapixels and costs 4787, because the long-edge cap crushes the short edge.
Gemini: a fixed budget, and no area term.
pw = floor(sqrt(N * w / h))
ph = floor(sqrt(N * h / w))
tokens = pw * ph with N = 1120
This is the patch grid of the image rescaled so its area is exactly N patches. Read what is
absent: w and h enter only as a ratio. The pixel dimensions do not appear in the price. A 1x1
image and an 8000x8000 image both cost 1089 tokens on Gemini 3.x, and downscaling before you send
saves nothing. If an extreme aspect ratio floors one axis to zero it clamps to one patch and the
other axis takes what is left of the budget, so 10000x1 costs 1120. Both floors use integer
arithmetic, not a float square root: k*k <= N*w/h is equivalent to k*k <= floor(N*w/h) because
k*k is an integer, so isqrt((N*w)//h) is exact.
N is set by the mediaResolution request field, and the ladder is exactly 1:2:4. LOW is 280,
MEDIUM is 560 and is the Gemini 2.5 default, HIGH is 1120 and is the Gemini 3.x default.
gemini-legacy is the Gemini 2.5 default path, which charges a flat 258 tokens for any image at
all, measured up to 8000x8000. Which path a model sits on does not follow the version number:
gemini-3-pro-image and gemini-3.1-flash-image are on the flat path, while gemini-3.5-transcribe
and gemini-robotics-er-2-preview are on the 1120 path. Sort by modality, not by version.
OpenAI: patches times a multiplier, or tiles plus a base.
patch models: tokens = floor(ceil(w / 32) * ceil(h / 32) * multiplier)
tile models: tokens = base + ceil(w / 512) * ceil(h / 512) * tile
Patch models fit the image inside a pixel limit, cover it with 32x32 patches and shrink proportionally if the grid exceeds the budget. Tile models fit a 2048 px square, bring the shortest side down to 768 px if it exceeds it, then count 512 px tiles.
| generation | shape | constants | limits | ceiling |
|---|---|---|---|---|
openai-gpt-5.4 |
patches | x 6/5 | 2048 px, 2500 patches | 3000 |
openai-gpt-5.2 |
patches | x 6/5 | 2048 px, 6144 patches | 4915 |
openai-gpt-4.1-mini |
patches | x 81/50 | 2048 px, 6144 patches | 6635 |
openai-gpt-4o |
tiles | base 85, tile 170 | 2048 px, short side 768 px | 1445 |
openai-gpt-5.1 |
tiles | base 70, tile 140 | 2048 px, short side 768 px | 1190 |
openai-gpt-4o-mini |
tiles | base 2833, tile 5667 | 2048 px, short side 768 px | 48169 |
openai is an alias for openai-gpt-5.4. The multiplier is floored, not rounded up: OpenAI's
published rule says round up and its published worked examples are one token high wherever the
product is not already a whole number, and fourteen of twenty-four live probes contradict the
published rule by flooring. The multiplier is stored as an exact fraction because at the floor
boundary a float rounds the wrong way. Three commonly quoted constants are now wrong. The
1536-patch cap is gone, since gpt-4.1-mini at 2048x2048 measures 6635 tokens. detail: low is 85
tokens only on gpt-4o and gpt-4.1, against 70 on gpt-5.1 and 2833 on gpt-4o-mini. And on
gpt-5.4, detail: low costs more than detail: high, because it carries a 6144-patch budget
against high's 2500.
Worked examples.
| generation | 1170x2532 phone | 2560x1440 desktop | 512x512 thumbnail |
|---|---|---|---|
anthropic |
3825 | 4787 (ceiling) | 364 |
anthropic-legacy |
1460 | 1564 | 365 |
gemini |
1078 | 1100 | 1089 |
gemini-legacy |
258 | 258 | 258 |
openai-gpt-5.4 |
2304 | 2764 | 307 |
openai-gpt-5.2 |
2304 | 2764 | 307 |
openai-gpt-4.1-mini |
3110 | 3732 | 414 |
openai-gpt-4o |
1445 (ceiling) | 1105 | 255 |
openai-gpt-5.1 |
1190 (ceiling) | 910 | 210 |
openai-gpt-4o-mini |
48169 (ceiling) | 36835 | 8500 |
Fourteen of those thirty cells are direct measurements: the whole gemini and openai-gpt-5.4
rows, the 512x512 column on every generation except openai-gpt-5.2 and openai-gpt-4o-mini, and
the 2560x1440 cell on both Anthropic rows.
The continuous form. For a square of side x below the caps the exact count is well
approximated by a * x^2, and for any rectangle by a * w * h, since the exact form factorises.
The exact count is a strict upper envelope, because ceil(w/p) * ceil(h/p) >= w*h/p^2 with equality
only when both edges are exact multiples of the patch size.
Anthropic: w*h/784 + C <= tokens <= (w+27)(h+27)/784 + C
OpenAI: m*w*h/1024 - 1 < tokens <= m*(w+31)(h+31)/1024
Gemini: N - sqrt(N)*(sqrt(a) + 1/sqrt(a)) + 1 < tokens <= N
for aspect ratio a = w/h, multiplier m, and N = 1120. Tokens per pixel squared are 1/784 for
both Anthropic generations, 1.2/1024 for openai-gpt-5.4 and -5.2, 1.62/1024 for
openai-gpt-4.1-mini, 170/262144 for openai-gpt-4o, 140/262144 for openai-gpt-5.1, 5667/262144
for openai-gpt-4o-mini, and zero for Gemini, which has no area term at all. The Anthropic bracket
was checked against 81 uncapped real-image measurements with zero violations. Fitting a free
coefficient to those 81 points returns 0.00129511 against 1/784 = 0.00127551, and the entire 1.54%
difference is the ceiling edge term. Relative error decays like 1/k at the worst case x = 28k+1:
65.5% at x=57, 6.0% at x=897, 2.1% at x=2549.
Provenance and grading.
| generation | evidence | exact rate |
|---|---|---|
anthropic |
138 real images, 438 synthetic points | 138/138 real, 437/438 synthetic |
anthropic-legacy |
157 synthetic points, no real images | 157/157 synthetic |
gemini |
275 probes, 70 of them held out | 70/70 held out, 275/275 overall |
gemini-legacy |
60 probes across nine model ids | 60/60 |
openai-* |
24 probes across six models, 30 API requests | 24/24 |
Re-scored against the shipped code over every logged row at every vendor, the estimator is exact on 1231 of 1232 points. The one miss is an Anthropic knife-edge at 5265x4615 where the correct grid is 73x64 and the estimator picks 74x64, a 1.37% overcount that affects roughly 0.2% of dimensions and never exceeds one patch row. A reference implementation in JavaScript agrees with this one on all 5060 cells of a 460-dimension by 11-generation cross-check.
anthropic is the strongest column, validated on real files across four image formats, six pixel
modes, EXIF orientation and animated GIFs, at 100%. anthropic-legacy is derived and not
re-validated: its total budget is bracketed to [1568, 1574] instead of pinned, because no probe in
the searched box discriminates within that range. gemini is measured on synthetic images only, and
N = 1120 is the unique integer in [1000, 1300] that produces zero misses over an 80-point aspect
sweep. openai-* is verified on gpt-5.4-nano, gpt-4.1-mini, gpt-4o, gpt-5.2, gpt-5.1 and
gpt-4o-mini, with gpt-5.4, gpt-5.4-mini, gpt-4.1 and gpt-5 sharing a probed model's
profile and not themselves probed. Deliberately absent because nothing measured backs them: the
gpt-5.5 and gpt-5.6 families, the -pro models, the deprecated patch models, every non-default
detail level except gpt-5.4 at low, and Gemini's 258-per-tile path.
One implementation detail is knowingly approximate. OpenAI's own calculator applies a float32 rounding step when computing the aspect ratio for its pixel-limit resize; this implementation uses exact integer round-half-up. No probe discriminates the two, and they could differ by one patch row on a knife-edge dimension.
There is also a transport limit that is not a pricing limit. Anthropic rejects base64 image payloads above exactly 10,485,760 characters, which is 10 MiB measured on the base64 string and not on the decoded image, capping the source file at 7.5 MiB on disk. It is independent of dimensions: a 10000x10000 solid PNG passes while a 2400x1600 noise PNG does not.
The coefficients price content only. Every measurement behind them isolates a string from its
request by subtraction, so what you get back is the cost of the text and nothing else. Not covered,
by construction: the per-request message frame, which an Anthropic count_tokens call puts at 7 for
claude-opus-4-8 and 9 for claude-sonnet-4-5 on a two-character message; system prompts, tool
definitions, response schemas and cache boundaries; multi-message conversations, where role markers
and turn structure add cost; audio and video.
One instrument per generation, with the isolation convention stated beside it.
- Claude.
POST /v1/messages/count_tokens, isolated asiso(X) = count(M + X) - count(M)with a fixed short markerM, so the request envelope cancels on both sides. The endpoint is free and rate-limited at 2,000 requests per minute and needs nomessagescall, so calibration costs nothing. Every later round usedctokoffline instead, gated at 808/808 exact against the recorded API column before use. - OpenAI. Local
tiktoken0.14.0,len(encoding.encode(text))foro200k_baseandcl100k_base. Exact, no marker, no isolation, because there is no envelope to subtract. - Gemini. Local
sentencepiece0.2.2 over thegemma3SentencePiece model file, pinned by the sha256 that Google's own loader pins, 262,144 pieces. Verified exact against Google'scountTokenson 418 of 418 strings, a battery that included astral emoji, ZWJ sequences, combining-mark traps, control characters, byte-fallback bait and the empty string. The Gemini column is the only one whose fitting target carries no measurement error at all.
Four corpora appear on this page. Every figure says which one it came from.
| corpus | chunks | characters | languages | what it is for |
|---|---|---|---|---|
| calibration | 808 | 2,160,395 | 12 | the rows the default table was fitted on. Every default figure labelled a number of record comes from here, cross-validated inside this set |
| multilingual fit | 1,096 | 2,898,907 | 21 | the calibration corpus plus 288 European Wikipedia chunks. The rows the multilingual table was fitted on |
| validation | 1,287 | 3,673,031 | 20 | fetched 2026-08-29 from sources sharing zero groups with the calibration corpus. Never used to fit anything |
| large evaluation | 592,284 | 1,263,262,482 | 40 plus code | built to answer the corpus-size question at a scale where the answer could not be an artefact. Nothing that ships was fitted on it |
Calibration corpus. 808 chunks in four length bands from 40 to 16,000 characters. 456 chunks of file content: TypeScript, Rust, JavaScript, dense JSON, English technical markdown, Chinese prose, mixed Chinese and English notes, plus Chinese, Japanese, Korean and classical Chinese chunks. 352 chunks of Wikipedia in Arabic, Chinese, English, Greek, Hebrew, Hindi, Japanese, Korean, Russian, Thai and Vietnamese. Those are content-matched: thirty English seed topics followed through interlanguage links, so the per-script figures are not confounded by subject. 179 articles, each recorded with page id and revision id, so that half is re-fetchable at the exact revision. Text is CC BY-SA 4.0. The file half is five private repositories and is not published, and the Wikipedia half is encyclopedic register only. That scope applies to every default-profile number of record on this page.
Multilingual fit corpus. The calibration corpus at full weight plus 288 Wikipedia chunks, 738,512 characters, in German, French, Spanish, Italian, Portuguese, Czech, Polish, Turkish and Dutch: 32 per language, 8 per length band, one article per chunk, at exactly the size of the eleven Wikipedia strata already there. The home corpus keeps 74.5% of the character mass. Dutch is in the fit because it is the sharpest available test of the vocabulary-wall measurement: its orthography is close to English and its whole-word hit rate is not, so a marker family selected partly on Dutch and fitted on a corpus without Dutch would be untestable at home. Finnish, Hungarian, Indonesian, Malay, Uzbek, Cebuano, Welsh and Esperanto are deliberately absent and stay in the held-out evaluation corpus as a transfer test. So does the Gutenberg register: European literary prose is in no fit, and it reads 8.27 under the profile against 21.40 under the default. No text was re-tokenized to build this corpus; every token count is one the large evaluation corpus had already recorded.
Validation corpus. 1,287 chunks, 483 source groups, the same four length bands and the same two
chunkers. 863 chunks of Wikipedia in 19 languages, resolved through interlanguage links from a seed
list disjoint from the calibration corpus's, every chunk carrying its page id and revision id. 32
chunks of literary Chinese from zh-classical, drawn from Special:LongPages instead of the matched
topics, so that stratum is not content-matched. 224 chunks from 39 Project Gutenberg books in seven
languages, boilerplate stripped by Gutenberg's own markers, public domain in the US. 168 chunks from
four public repositories at pinned tags: ripgrep 14.1.1, mdBook v0.4.40, flask 3.0.3, got
v14.4.2. The count of source groups it shares with the calibration corpus is asserted to be zero on
every run.
The two are not the same shape, and that matters when reading any combined figure. The calibration corpus is 56% file content; the validation corpus is 80% prose.
Large evaluation corpus. 598,603 chunks and 1,263,262,482 characters of Wikipedia in 40
languages, from magibu/wikipedia-40-langs at snapshot
3b89135298addfb0292732067be584cd5aba19f1, CC BY-SA 4.0. English and Turkish are downweighted from
140,000 articles to 14,000 by a deterministic hash so all 40 enter at parity, at 4,500 chunks per
language per length band; the group is the article and 15% of articles are held out. Plus 71,654
chunks and 129,225,365 characters of published open-source code from 230,345 files across 2,100
packages in local node_modules, cargo registry and uv wheel caches: TypeScript, JavaScript,
Python, Rust, C, JSON and technical markdown. Bundled and minified JavaScript is identified by a
mean physical line length above 200 characters and kept as its own stratum. 1,392,487,847 characters
were tokenized five times over, once per generation, offline. Two things this corpus lacks that the
calibration corpus has, both of which cost a refit dearly: no Thai and no Hindi. It also contains
Armenian, which no bucket covers, so Armenian was measured and then excluded from every fit.
Non-negative least squares, by projected coordinate descent, with no intercept. Both choices are
measured. Plain least squares hands you negative bucket prices that cross-validate well and collapse
on the next refit; an early run produced word2 = -0.77 and word68 = -1.63 before the switch. An
intercept is worse: adding one costs 2.13 MAPE points on o200k and 2.21 on cl100k, because the
corpus spans 40 to 16,000 characters, squared-error loss fits the constant to the large rows, and
every small row then inherits it. The same defect exists in the wild: one published estimator ships
an intercept of -28.83 and floors every prediction at its absolute value, so its minimum possible
output is 29 tokens for any string.
Validation is five-fold cross-validation over five seeds, grouped by source, 495 groups over 808 rows, with the coefficients refitted from scratch inside every fold. Nothing is fitted on test. The multilingual table is fitted the same way over the 1,096-row corpus, with six floats pinned.
Fixed before any candidate ran: a candidate earns a place if it gains at least 0.30 MAPE points on at least one generation and loses more than 0.30 on none. Gains are quoted per generation and never averaged, and the full five-generation delta row is reported whether the candidate passes or fails. The bar is a decision threshold and not a significance test. Several verdicts on this page turn on differences between 0.15 and 0.42 and would be better with a fold-level spread, which was not computed.
Re-seeding the cross-validation split moves any figure on this page by up to 0.10 points, and two aggregation conventions in circulation differ by up to 0.12 on the same rows. Treat differences under about 0.1 as noise. The 0.30 bar is set at roughly three times that, deliberately.
A parity gate runs before any new number. The Python counters and the JavaScript counters must agree cell for cell on all 2,095 chunks across every bucket and every profile, and the code-point partition is asserted on every row. The current gate reads 50,280 counter cells and 31,425 estimate comparisons with zero mismatches, and the nine punctuation-run examples above are reproduced from the prose definition alone.
Reject a refit if any of these fail:
- any coefficient is negative, which means you did not use NNLS;
cjkfalls outside 0.55 to 1.4,otheroutside 0.3 to 1.1, oremojioutside 2 to 12;kanafalls outside 0.4 to 1.2,hangoutside 0.6 to 1.5,arboutside 0.25 to 1.0, orheboutside 0.35 to 1.25;punctRunfalls outside 0.4 to 1.2. The five shipped columns span 0.564 to 0.978, and the mechanism says it should sit near one token, since a punctuation run mostly merges into one piece;- more than five of the coefficients in a column are exactly zero;
- held-out MAPE is worse than the accuracy section for text of the same kind.
latacc is exempt from the zero rule and from any lower bound. 0.00000 is a correct answer for a
tokenizer with real accented-Latin merges.
A warning about the emoji gate, measured and then left in place anyway. That gate was written
from an assumption about what an emoji costs, and the bucket's actual contents do not match it. Delete every emoji-bucket code point from the corpus and re-count: the marginal cost is 1.48 /
1.48 / 1.00 / 1.16 / 0.73 tokens per code point on the five generations, against fitted floats of
5.67 / 4.88 / 3.91 / 2.47 / 3.59. The reason is composition. Of the 1,481 emoji-bucket code points
in the whole 2,095-chunk corpus, 1,456 are ordinary symbols, which is arrows, box-drawing and maths
from technical markdown priced at 1.45 on Claude 4.7+; only 21 are true astral emoji, priced at
3.38. The gate is kept because it catches the real failure it was built for, which is the bucket
collapsing when a refit dilutes it, and it has now caught that twice. Treat a value near 1.5 as a
measurement and a value near 0 as the failure.
The cjk gate used to read 0.7 to 1.4. It was written when four generations were on the page, and
Gemini broke it on the first try at 0.651. That was not a bad fit. It was a tokenizer that spends
1.44 characters per token on Chinese where Claude spends 0.94. If a sixth generation lands outside
the widened range, widen it again and check the per-stratum error instead of rejecting the fit.
One warning the gates cannot enforce. A refit on a corpus with no CJK cannot produce a cjk
coefficient, however confidently the solver reports one, because that is a singular system. The
gates catch the near-singular version, which returns plausible-looking numbers. The same applies to
kana, hang, arb, heb and latacc: each rests on a single language stratum in the
calibration corpus, and a corpus without that language cannot price it.
The search protocol was fixed before it ran. At each step try every applicable tweak, adopt the best, record the gain, and stop when two consecutive adopted tweaks each gain less than 0.30 points. It was run independently on each generation, so "which buckets earn their place" is a per-generation answer instead of a claim borrowed from one column.
The Claude 4.7+ curve, starting from a 7-bucket ancestor at 20.51 MAPE: script split +8.95, word-length bins +2.14, punctuation split +1.07, camelCase +0.41, newlines +0.18, then a cliff where every remaining candidate is worth 0.06 or less. The other four generations produce the same ordering, script coverage first by a factor of four to six over anything else. Gemini, fitted last and by a different vendor with a different algorithm, ran the same protocol from 16.688 and also selected the script split first, at +3.70. That replication on a fifth tokenizer is the strongest evidence the shape is real and not fitted to one vendor's quirks.
The word-length bins look like over-engineering and are not. The measured cost of an English word is
not proportional to its length and is not even monotone. A common four or five letter word costs
less than a two-letter one, 1.25 tokens against 1.55, because it is a single dedicated token. No
per-character coefficient and no monotone rule can bend that way, and it is why chars/4 fails
hardest on short words.
Three buckets that the search could not reach were found afterwards, each because the atom set could
not express the counter. Kana was swallowed by the Han ranges: the feature code's CJK list covered
U+3040 to U+30FF, so no candidate the greedy could write would separate kana from Han, and splitting
it takes Japanese from 14.4% to 3.9% on Claude and from 13.7% to 4.6% on Gemini. A direct frame
probe measured kana at 0.815 tokens per character and the fitted coefficient landed at 0.813 without
being told. Arabic and Hebrew were pulling opposite ways inside one bucket, Arabic at +25.3% bias
against Hebrew's -12.9% on cl100k; separating them is worth +0.10 / +0.11 / +0.42 / +1.33 / +1.00
and takes Arabic on cl100k from 25.3% to 2.9%. And latacc turned out to be a contamination guard
instead of a price: 99.5% of the calibration corpus's accented-Latin mass sits in one 32-chunk
Vietnamese stratum, deleting that stratum takes the counter's value to -0.01 on every generation,
and the gain lands on English, Chinese prose, JSON and code, strata with no accented Latin in them
at all. Without the bucket, Vietnamese's 22,496 accented characters are priced through the shared
Latin word-length bins, and those bins become a compromise between two languages. A bucket carried
for one language's benefit is not dead weight, because it stops the other coefficients from being
compromises.
A candidate's measured value is a property of the host shape and not of the candidate.
punctRun was measured twice by the same harness against two different host shapes. The first
reading was +0.05 / +0.08 / +0.49 / +0.01 and it was filed as an o200k-only curiosity. The second
reading, inside the current shape, is +0.41 / +0.72 / +0.47 / +0.38 / +1.16. Both readings reproduce
exactly and nothing about the counter changed. Two things about its surroundings did: the Gemini
column, where it was always worth +1.17, did not exist yet, and latacc had not been added, so the
Latin word coefficients were still absorbing Vietnamese along with whatever run structure punctRun
would later explain. Every disposition on this page is a disposition as measured inside the shipped
shape. One second-order effect from the same change is worth recording: word2 was exactly 0.00000
on both Claude columns before punctRun went in and is 0.42890 and 0.38281 after, because
punctRun took over run structure that ws had been absorbing, which freed the word bins to price
words again.
The twenty-third bucket came out of a different search, because the greedy protocol above cannot express a letter pair.
What it is for. The default table under-counts continental European prose by about a fifth, and a refit on 733 times the corpus closes that gap while costing English 10.56 to 23.77 and source code 7.77 to 14.55. Ten complete refits at mixtures from 90% Wikipedia down to none of it trace a smooth curve, and no point on it reaches the corner where both halves are good. Fitting each population on its own half and scoring on the other shows why: the shipped coefficients are already within 0.5 points of the best a single vector could do on English and on code, and the refit is within 0.3 of the best on Wikipedia. The residual is capacity, and the capacity is one bucket. Splitting the four Latin word buckets by a language label the counters cannot see takes English Wikipedia from 23.77 back to 8.74 and source code from 14.55 back to 8.08 while leaving European prose alone. A label is not a counter and cannot ship.
What a letter pair recovers that a label does. Ten letter pairs pooled into a single counter, added to the same twenty-two buckets on the same 592,284-row fit, take held-out English from 23.77 to 8.33, which is past the label's 8.74, and held-out source code from 14.55 to 11.78. The frontier width goes from 13.18 to 8.10 against the label's 8.74. Three sets of ten random letter pairs drawn from the same 25,230-candidate space and mass-matched pair by pair to the winning set return the baseline to two decimal places, 23.77 / 23.77 / 23.79, with their coefficients pinned at zero. The six strongest English-marking pairs return exactly 23.77 in both additive forms, which isolates the non-negativity limit: a counter that fires on the cheap side can only add cost, so the fit has no use for it. The effect is these letters and not the extra float.
The selection adversary, which is where the shipped set comes from. A marker that helps Europe
can also fire at home, and non-negative least squares makes the fired cost strictly additive with
nothing to pay it back. The first selection ranked candidates on a key that scored the median over
25 expensive Latin languages minus the maximum of English and code, and Vietnamese sat on the reward
side of that key. Under it, i$ ranked 2nd of 25,230. Move Vietnamese to the penalty side, change
nothing else, and the same marker falls to 25,195th. The consequence is measurable to the token:
pin the whole default table, fit one marker float at Europe scale, and the 32-chunk Vietnamese
stratum gains 255 tokens on a 1,895-token chunk and goes from 5.66 to 18.82 MAPE.
The fix was to the adversary and not to the marker. Candidates were ranked on discovery counts from
a train split alone. Sets were fitted on 88,957 European training rows and scored on 30,042 held-out
European selection rows. A set was admitted only if it held 25 named home strata inside
0.30 + 2 SE of their own out-of-fold noise on all five generations at once.
The 25 are the 22 calibration strata plus the 40-to-250 and 250-to-1,000 character bands plus the
corpus as a whole, so nothing in the calibration corpus is unwatched. Under that cap, between 1,435
and 2,073 admissible sets exist at every size from two to eight patterns. The shipped six are one
point in that region, chosen for held-out European recovery.
The cap is not an independent test of a set selected to pass it, so cuts that no fit and no cap touched carry the check instead. Under a full freeze at Europe scale, held-out Vietnamese Wikipedia goes from 8.57 under the default table to 11.26 with an ungated four-pattern set and to 7.18 with the gated six. The validation corpus's 64 Vietnamese chunks go from 4.38 to 15.82 and to 5.37. On the same freeze, held-out European goes to 9.03 with the ungated set and to 8.60 with the gated one, so the guards cost nothing on the population they were meant to serve.
Six floats are deliberately not refitted and carry the default column's value exactly: ws,
nl, punct, punctWide, punctRun and emoji. Those six are identified by the calibration
corpus's file registers, and the European extension contains no telegraph, no emoji, no JSON and
almost no newlines, so refitting them on it moves them without evidence. Measured: letting them move
costs the calibration corpus 0.28 points out of fold, takes four home strata past their measured
noise instead of one, and drives emoji to 1.24 / 0.00 / 1.45 on o200k / cl100k / Gemini, outside
the gate this page publishes.
"Capacity" names a phenomenon without explaining it, so the phenomenon was decomposed. English is cheap because the vocabulary was built on it. Priced word by word on the four exact tokenizers, frequency-weighted over running text and matched to 4-to-8-letter words, the share of words the tokenizer knows as one whole piece:
| English | code identifiers | median of 11 other Latin languages | |
|---|---|---|---|
| Claude 4.7+ | 38.3% | 42.3% | 4.6% |
| Claude pre-4.7 | 81.8% | 81.4% | 38.0% |
o200k_base |
88.1% | 88.9% | 67.9% |
cl100k_base |
86.9% | 89.3% | 34.2% |
Everything else follows from that row. Decomposing the chars-per-token gap into named channels of word length, sub-word splitting, diacritics and capitalisation puts 97% to 120% of it in splitting on every generation, with diacritics between -7% and +6% and length pushing the other way. Source identifiers are English, measured instead of by analogy: 42.3% against 38.3%, and 2.657 characters per token against 2.641 on the same word-level instrument.
The tell you would reach for first points the wrong way. An accented spelling is usually the one in
the vocabulary, so stripping the accents makes a word more expensive: on o200k_base the Polish
word for enterprise costs 2 pieces spelled correctly and 5 with its diacritics removed, and the
Vietnamese word for person costs 1 and 2.
The strongest version of the test restricts both sides to words that carry no observable mark at all: lowercase, pure ASCII, four to eight letters, no consonant run of four or more, vowel ratio at least 0.35, so that a counter looking at code points cannot tell them apart. The gap does not shrink.
| Claude 4.7+, chars/token vs English | German | Dutch | Polish | Finnish |
|---|---|---|---|---|
| all 4-8 letter words | -33.4% | -29.1% | -31.9% | -37.2% |
| orthographically indistinguishable subset | -38.8% | -38.4% | -44.2% | -46.9% |
Whole-word hit rate on that subset: English 47.9%, Dutch 6.2%, German 0.7%, Polish 0.7%, Finnish 0.2%. Dutch carries 2.5 accented words per thousand against English's 1.4, statistically the same page, and runs 2.136 characters per token against English's 2.750 on those same rows.
One sentence of interpretation, marked as interpretation because it is not measured. A vocabulary is built by frequency statistics over scraped text, and nothing in that pipeline is told what a language is; the dead tokens that survive in released vocabularies, reachable by no input, are the documented trace of a process with no such notion. So a counter of character statistics is closer to the thing that actually happened than a language label is.
The corpus is 808 chunks, which is small. That is defensible only if the coefficients had stopped moving, so it was tested. The corpus was expanded to 2,095 chunks with 1,287 new chunks sharing no source with the fit. Frozen against 511 held-out chunks in the same eleven languages and the same register, the shipped coefficients score 4.6 / 4.4 / 5.8 / 4.2 / 5.7 against their published 5.26 / 5.29 / 6.31 / 5.29 / 6.04, better on text they have never seen. Refitting all twenty-two buckets on the doubled corpus makes those rows worse by 1.7 to 3.5 points.
Whether the numbers printed in the table have converged is a different and stricter question. It was measured with a stratified cluster bootstrap: 2,000 resamples of the calibration corpus drawn as whole source groups, so that correlated chunks from one file or one article move together, with all twenty-two buckets refitted from scratch on each draw.
Not one published float is pinned at the precision it is printed to. A coefficient is pinned to
d decimal places when its standard error is below half of the last digit, SE < 0.5e-d. Of the
110 floats in the default table, zero reach five places; 41 reach one place, 62 reach none, and 7 do
not pin even the units digit. The 41 are cjk, kana, hang, cyr, arb, heb and other on
all five generations, plus longChars on o200k, cl100k and Gemini, and ws, punctWide and mark
on o200k. The 7 are emoji on all five generations and camel on both Claude columns.
The smallest standard error on the page is other on o200k at 0.37851, SE 0.0101, a 2.7%
coefficient of variation; on Claude 4.7+ the same bucket is smallest at 0.72418 with SE 0.0140. The
largest is emoji on Claude 4.7+, SE 3.78 against a coefficient of 5.67236, whose 95% interval runs
from 2.07 to 14.05. The split is structural: the seven buckets counting characters in a single
script are the tight ones at 1.3% to 5.5% on every generation, and they carry 48% of the predicted
tokens between them. The word and punctuation buckets are the loose ones, mostly 20% to 55%.
More text buys half a decimal digit per factor of ten, and that law was tested. Fitting the
standard error against corpus size over seven rungs from 200 to 1,154 chunks with the composition
held constant gives a median log-log slope of -0.52 against the -0.50 that 1/sqrt(N) predicts. A
corpus ten times the size therefore divides every standard error by 3.16, which moves 51 of the 110
floats up one decimal place and none of them up two. To earn the fifth printed decimal the median
coefficient needs a corpus about 200 million times larger, roughly 5e14 characters.
That extrapolation was then checked against a corpus 733 times larger and it was right. The same
estimator ran over nine rungs from 792 to 534,892 chunks at 300 resamples per rung, three decades of
corpus size instead of half of one, and the median slope came out at -0.491. Across that 675-fold increase the law predicts
a 26.0x fall in every standard error, and the measured per-bucket falls run from 24.7x for
punctRun to 33.3x for cyr. What that bought: 16 of the 110 floats reach two decimal places, 55
reach one, 32 reach none, and 3 still do not pin the units digit. The median float gained exactly one
place. Two buckets sit far off the law and both are named: emoji at -0.147 and camel at -0.369,
because Wikipedia carries almost none of either and their draws sit on the non-negativity boundary.
What the uncertainty costs the answer is much smaller than what it costs the floats. Scoring all
2,000 bootstrap coefficient vectors as estimators, 95% of them land inside these bands. The
shipped column is the in-fold figure, because that is what a vector fitted on all 808 rows and
scored on all 808 produces:
| generation | shipped, in fold | 95% of refits on a fresh 808 chunks | width | split-noise floor |
|---|---|---|---|---|
| Claude 4.7+ | 5.04 | 5.05 to 5.66 | 0.61 | 0.12 |
| Claude pre-4.7 | 5.02 | 4.98 to 5.98 | 1.00 | 0.12 |
| o200k | 6.02 | 5.84 to 7.12 | 1.28 | 0.12 |
| cl100k | 5.21 | 4.83 to 6.20 | 1.37 | 0.12 |
| Gemini | 5.73 | 5.70 to 6.72 | 1.02 | 0.12 |
The reason the output is so much better determined than its parameters is that the loose floats
trade against each other. In the bootstrap draw distribution word35 and latacc sit at r = -0.875
on Claude 4.7+, and their sum is 2.1 times better determined than the two floats separately;
punct and punctRun sit at -0.769 for 1.46x, and cjk and kana at -0.845 on o200k for 2.51x.
The word35 and latacc trade is the contamination-guard mechanism arriving from a third
direction: the fit cannot separate an accent price from a European word price, so it prices
whichever the objective prefers.
Sampling more text of the same kind does not move these coefficients. Changing the distribution does. Both halves are measured, and neither is an inference from the other.
| what changed | median shift, in units of the coefficient's own standard error | buckets past 2 SE, out of 22 |
|---|---|---|
| a fresh, fully independent 346-chunk sample at the same composition, sharing no source with the fit | 0.55 to 0.92 | 1 to 2, which is the rate chance alone produces |
| a refit at 1,154 chunks, 43% past the published corpus, same twelve languages, composition held | 0.52 to 0.90 | 1 to 4 |
| a refit at 2,095 chunks, adding 512 rows in eight new languages | 1.13 to 1.92 | 6 to 9 |
The first row is the convergence claim as a number. An independently drawn sample from the same
distribution, with zero shared sources, reproduces the published coefficients inside their own error
bars on 20 or 21 of 22 buckets. The exception repeats on every generation and is named: cjk moves
2.5 to 4.0 standard errors, from 1.141 to 1.251 on Claude 4.7+, because the fresh CJK rows include
classical Chinese and Japanese Wikipedia that price denser than the calibration corpus's CJK. So the
convergence claim ships conditioned on the calibration distribution.
The accuracy tables are per chunk. This is per corpus, and it is the receipt for "the longer the text, the better the estimate" and for the ceiling on that claim.
Draw chunks length-proportionally from the 2,095-chunk corpus until the text reaches length L,
take a random window from the final piece, truncate to exactly L, then run the real tokenizer on
the concatenation and estimate on the same string. 13,040 samples, thirteen lengths from 200
characters to 1 MB, four content mixes, seed 20260830, no API calls. The rows below are the honest
out-of-sample mix: the calibration corpus's own file-to-prose split and language balance, rebuilt
from text no coefficient has seen. Median and p90 of the absolute signed error, with k the mean
number of source documents per sample:
| text length | k | Claude 4.7+ | Claude pre-4.7 | o200k | cl100k | Gemini |
|---|---|---|---|---|---|---|
| 200 chars | 1.0 | 5.6 / 14.5 | 4.9 / 12.7 | 5.7 / 16.3 | 4.8 / 13.5 | 4.7 / 11.9 |
| 1,000 | 1.1 | 3.9 / 11.4 | 2.9 / 8.0 | 5.3 / 13.2 | 3.9 / 12.0 | 3.5 / 9.9 |
| 4,000 | 1.3 | 2.9 / 9.0 | 2.8 / 7.9 | 4.0 / 13.8 | 3.3 / 11.2 | 3.0 / 8.1 |
| 16,000 | 3.3 | 2.2 / 5.5 | 1.9 / 5.1 | 3.0 / 7.1 | 2.1 / 5.8 | 2.3 / 6.5 |
| 64,000 | 10.7 | 1.2 / 3.4 | 1.0 / 3.4 | 1.9 / 4.7 | 1.2 / 3.9 | 1.4 / 3.2 |
| 256,000 | 41.1 | 0.7 / 1.7 | 0.7 / 1.8 | 1.1 / 2.7 | 0.8 / 2.0 | 0.9 / 2.1 |
| 1,000,000 | 158.4 | 0.4 / 0.8 | 0.5 / 1.0 | 0.7 / 1.7 | 0.5 / 1.0 | 0.7 / 1.2 |
Length at which the mean absolute error, the same convention as the headline 5.26, first reaches each level on this mix:
| Claude 4.7+ | Claude pre-4.7 | o200k | cl100k | Gemini | |
|---|---|---|---|---|---|
| 5% | 2.1 kB | 0.5 kB | 5.6 kB | 2.3 kB | 0.8 kB |
| 3% | 11.9 kB | 8.1 kB | 28.8 kB | 11.2 kB | 16.1 kB |
| 1% | 167 kB | 154 kB | 557 kB | 268 kB | 275 kB |
| 0.1% | never | never | never | never | never |
So the headline figure is the error of a paragraph. On this out-of-sample mix the mean absolute error equals each generation's published out-of-fold figure between 200 and 1,200 characters, and scored on native corpus chunks instead of cut windows the crossing sits near 800. That offset was measured at 0.7 to 0.8 points, so every crossing above is a mild under-estimate. Read them to one significant figure: the rungs roughly double, so a crossing is located to within a factor of about 1.4 by construction.
The floor, and why 0.1% is out of reach. The error is a random part that shrinks with length plus a bias that does not, and the bias belongs to the content mix. Mean signed error at 1 MB, which is that mix's aggregate error:
| mix | Claude 4.7+ | Claude pre-4.7 | o200k | cl100k | Gemini |
|---|---|---|---|---|---|
| calibration corpus, in sample | -0.03 | -0.03 | +0.07 | +0.05 | -0.03 |
| held out, same mix, new text | -0.27 | -0.40 | +0.62 | +0.03 | -0.67 |
| 70% English and code | -3.55 | -3.98 | -2.01 | -1.80 | -3.81 |
| 50% German, French, Spanish, Italian, Portuguese, Czech, Polish, Turkish | -8.86 | -10.01 | -7.11 | -6.56 | -9.55 |
Out of sample the fitted floor is 0.32% to 0.68%, above 0.1, so no length reaches a tenth of a
percent. In sample the floor fits at 0.07% to 0.23% and 0.1% would need on the order of 10^10
characters. The aggregate error quoted elsewhere is not the error at infinite length. It is the
floor, and a half-European corpus sets that floor thirty times higher.
Length buys precision, not accuracy. On the half-European mix the median error rises from 6.0% at 8 kB to 9.5% at 128 kB, and on the English-heavy mix it bottoms at 2.6% near 32 kB and returns to 3.5% at 1 MB. Averaging turns a wide distribution straddling zero into a narrow one centred on the mix bias, so once the bias is worse than the median single-document error, more text makes the median worse. The p90 falls monotonically on every mix; the median does not. The claim is about the tail, and about a mix whose bias is small.
sqrt(a^2/L + b^2) describes the curve only above about 16 kB, where its worst relative residual is
8% to 34%. Across the whole range it is off by 81% to 129%, because below 16 kB a sample is one
document and there is nothing to average; there the error decays as about L^-1/3.
One caveat for anyone summing per-file estimates. estimate is a sum of counters, so it adds across
files. A tokenizer does not, because a merge can cross the join. Measured on 3,000 document pairs,
tokenising the concatenation costs 0.739 fewer tokens per join on both Claude generations, 0.124 on
o200k, 0.119 on cl100k and 0.038 on Gemini. The OpenAI encodings pre-split with a regular expression
so 86% to 87% of joins change nothing; Claude has no such barrier and 70% of its joins change.
Summing per-file estimates therefore over-counts by about 0.74 * files tokens on Claude, which is
under 0.02% at typical file sizes and worth attention only across thousands of small files.
Every figure in this subsection is the default profile, out of fold on the calibration corpus, under the grouped cross-validation above. The aggregation convention is one out-of-fold prediction per row at seed 1, with the metrics computed on the complete 808-row vector.
MAPE / median / p90 / share within +/-10% / aggregate error:
| n | Claude 4.7+ | Claude pre-4.7 | o200k | cl100k | Gemini | |
|---|---|---|---|---|---|---|
| everything | 808 | 5.26 / 3.8 / 11.5 / 87% / -0.1 | 5.29 / 3.7 / 11.6 / 87% / -0.1 | 6.31 / 4.9 / 13.9 / 82% / -0.3 | 5.29 / 3.7 / 12.2 / 86% / -0.4 | 6.04 / 4.4 / 13.7 / 82% / -0.1 |
A second aggregation convention is in circulation, the mean over 25 fold-and-seed pairs of each fold's MAPE, and it reads 5.25 / 5.30 / 6.37 / 5.41 / 6.05 on the same rows. The two are never mixed inside one cell. A third figure appears wherever novocab has to be scored the way an unfitted comparand is scored, fitted on all 808 rows and evaluated on all 808: that reads 5.04 / 5.02 / 6.02 / 5.21 / 5.73 and is labelled in fold at every site where it is used.
Signed error is (estimate - true) / true * 100, so a negative percentile is an under-count, and
the percentiles are nearest-rank on the signed values instead of on their absolute value.
| generation | p10 | p50 | p90 | p95 | worst under | worst over | within 5% | within 10% | within 20% |
|---|---|---|---|---|---|---|---|---|---|
| Claude 4.7+ | -8.7 | +0.0 | +7.3 | +9.5 | -55.4 | +33.3 | 61% | 87% | 98% |
| Claude pre-4.7 | -9.1 | +0.2 | +7.3 | +10.0 | -49.1 | +37.5 | 62% | 87% | 97% |
| o200k | -9.7 | +0.1 | +9.4 | +12.7 | -32.8 | +57.1 | 51% | 82% | 97% |
| cl100k | -8.9 | +0.0 | +8.0 | +10.7 | -38.3 | +57.1 | 62% | 86% | 98% |
| Gemini | -8.3 | +0.9 | +10.0 | +13.6 | -30.9 | +44.4 | 57% | 82% | 96% |
The median chunk is right to within a percent and the distribution is close to symmetric, which is why the aggregate stays under half a percent: the tails cancel when you sum. That makes the formula useful for sizing a directory and unsuitable for gating one short string against a budget. One chunk in twenty is off by more than 10%, and the worst chunk of 808 is off by half, so budget against the p90. The worst 5% are concentrated instead of spread evenly: on Claude 4.7+ 58% of them are in the 40 to 250 character band, and on o200k and Gemini the tail is dominated by classical Chinese, 15 of 40 chunks each.
By text kind, MAPE / signed aggregate bias:
| text | n | Claude 4.7+ | Claude pre-4.7 | o200k | cl100k | Gemini |
|---|---|---|---|---|---|---|
| source code (TS) | 56 | 4.8 / +3.1 | 4.0 / +1.9 | 4.4 / +0.7 | 3.6 / +0.6 | 4.7 / +1.7 |
| source code (Rust) | 56 | 5.4 / -1.1 | 4.6 / -0.6 | 5.6 / +3.1 | 5.7 / +2.3 | 3.0 / -0.3 |
| source code (JS) | 56 | 4.8 / -1.9 | 4.8 / -1.8 | 5.2 / -1.7 | 6.2 / -0.2 | 4.6 / -1.7 |
| English technical markdown | 56 | 5.9 / +3.7 | 7.2 / +6.1 | 6.5 / +2.1 | 6.4 / +0.1 | 6.2 / +4.3 |
| dense JSON | 56 | 5.9 / -1.2 | 6.7 / -1.8 | 7.2 / -1.3 | 6.7 / -2.1 | 5.6 / -0.9 |
| mixed CN/EN notes | 56 | 5.5 / -3.2 | 6.0 / -3.6 | 5.4 / -3.5 | 6.0 / -3.0 | 4.9 / -3.1 |
| Chinese prose | 56 | 6.6 / +2.7 | 6.4 / +2.2 | 6.8 / +5.2 | 7.4 / +7.2 | 6.6 / +4.0 |
| English | 32 | 6.2 / -1.8 | 6.8 / -2.9 | 5.0 / +3.0 | 5.5 / +2.7 | 3.4 / +0.3 |
| Chinese | 32 | 4.1 / +0.9 | 4.0 / +1.0 | 6.6 / -3.4 | 7.4 / -4.4 | 5.2 / -0.3 |
| Japanese | 32 | 3.9 / +0.5 | 4.1 / +0.7 | 3.3 / -0.2 | 3.9 / +0.2 | 4.6 / -0.5 |
| Korean | 32 | 3.0 / +0.1 | 3.2 / +0.0 | 6.2 / -0.7 | 4.4 / -0.3 | 5.6 / -0.6 |
| Russian | 32 | 9.5 / -0.0 | 9.3 / +0.1 | 8.6 / +0.0 | 5.5 / -0.3 | 8.6 / +0.3 |
| Arabic | 32 | 3.3 / -0.2 | 3.3 / -0.2 | 6.6 / -0.7 | 2.8 / -0.3 | 6.1 / -0.2 |
| Greek | 32 | 3.3 / +0.2 | 3.4 / +0.4 | 6.1 / +0.7 | 1.2 / +0.1 | 5.2 / +0.8 |
| Hebrew | 32 | 3.5 / +0.2 | 3.4 / +0.1 | 4.2 / -0.2 | 2.4 / -0.2 | 3.6 / -0.2 |
| Hindi | 32 | 4.1 / +0.3 | 4.1 / +0.2 | 6.6 / +1.9 | 2.6 / -0.1 | 6.9 / +0.6 |
| Thai | 32 | 4.0 / -0.5 | 4.1 / -0.6 | 7.3 / -0.8 | 3.6 / +0.2 | 11.1 / +0.3 |
| Vietnamese | 32 | 7.1 / -0.6 | 6.9 / -0.7 | 6.7 / -1.2 | 4.7 / -0.6 | 4.6 / -1.4 |
| Japanese source files | 14 | 3.3 / +1.3 | 2.5 / +0.3 | 6.1 / +2.3 | 4.4 / +0.3 | 10.7 / +8.1 |
| classical Chinese | 17 | 13.7 / -12.5 | 13.0 / -12.0 | 20.0 / -19.3 | 14.0 / -13.6 | 23.7 / -23.6 |
The thinnest rows above were re-checked at two to three times the sample, same coefficients, frozen, on validation-corpus chunks from sources the fit never saw. Claude 4.7+, old rows / new rows / combined: Japanese 3.8 / 5.3 / 4.7 at n=110, Korean 2.3 / 3.2 / 2.8 at n=111, Vietnamese 5.7 / 4.4 / 4.8 at n=96, Arabic 3.4 / 2.7 / 3.0 and Hebrew 3.5 / 2.2 / 2.9 at n=64 each, classical Chinese 13.1 / 11.8 / 12.3 at n=49. Every one holds. Russian is the exception and keeps its name: at n=128 it reads 11.0 against the 9.5 above, because fresh Russian Wikipedia scores 5.2 while 32 chunks of Russian literary prose score 21.8 and pull the combined figure up. Literary Russian costs 1.97 characters per token against encyclopedic Russian's 2.64, and Cyrillic is priced per letter with no run term of its own, so it has the least structure of any script to absorb a change of register.
Accuracy of a single chunk depends strongly on that chunk's length, monotonically:
| chunk length | Claude 4.7+ | Claude pre-4.7 | o200k | cl100k | Gemini |
|---|---|---|---|---|---|
| 4,000 to 16,000 chars | 3.8% | 3.8% | 4.6% | 3.5% | 4.2% |
| 1,000 to 4,000 | 4.3% | 4.2% | 5.2% | 4.2% | 4.9% |
| 250 to 1,000 | 5.2% | 5.1% | 6.4% | 5.3% | 6.1% |
| 40 to 250 | 7.6% | 7.9% | 8.9% | 8.0% | 8.8% |
| 25 to 40 (below the fitted floor) | 15.7% * | 17.1% * | 12.5% | 14.2% | 8.4% |
| 15 to 25 (below the fitted floor) | 21.3% * | 19.8% * | 15.8% | 12.4% | 14.6% |
| 5 to 15 (below the fitted floor) | 29.0% * | 28.9% * | 18.0% | 17.5% | 18.9% |
The two starred columns are inherited 16-bucket figures and are not re-measured. The 150 sub-40-character slices reconstruct deterministically, but their Claude ground truth was computed in-process by an earlier round and never written to disk. On the three generations whose ground truth is local and exact, the current shape improves the below-floor bands by 2.4 to 10.2 points against those same 16-bucket figures, so the starred numbers are almost certainly pessimistic.
The default coefficients were fitted on five private repositories plus Wikipedia. If your text is
unlike that, refit; chat transcripts, OCR output, minified bundles and base64 are all outside the
calibrated register. Collect chunks of your real text, get the true count for each with the
instruments above, isolate each string as count(marker + text) - count(marker) with a fixed short
marker, and solve for the coefficients by non-negative least squares.
Fit on N chunks, score the rest, median of 40 draws:
| N | Claude 4.7+ | p90 | Gemini | p90 |
|---|---|---|---|---|
| 25 | 22.8% | 476% | 27.4% | 597% |
| 40 | 12.8% | 22.5% | 15.6% | 89.8% |
| 60 | 9.2% | 14.1% | 10.4% | 23.8% |
| 100 | 6.7% | 7.9% | 7.6% | 10.7% |
| 200 | 6.0% | 6.3% | 6.9% | 7.7% |
| 400 | 5.5% | 5.9% | 6.3% | 6.7% |
Use at least 100 chunks. 200 gets you within half a point of the floor. Never fewer than 60. This guidance has survived three shape changes and gets more necessary each time: the N=25 tail now runs between 389% and 597%, so one refit in ten at that sample size is catastrophic on some generation. Twenty-five samples against twenty-two buckets is three degrees of freedom. If you are refitting on European text specifically, take the capitalisation counter first and the accent split second, and expect the accent split to cost you on source code.
The procedure above is the whole procedure, and a comparable corpus reaches comparable coefficients. That is measured twice. A fully independent 346-chunk sample at the same composition, sharing no source group with the fit, reproduces the published floats inside their own bootstrap error bars on 20 or 21 of 22 buckets, which is the rate chance alone produces. And at 200 chunks a fresh fit lands within half a point of the floor on both the tightest and the loosest generation. The calibration corpus itself is private, so its composition and scale are given in full above instead of the text; nothing in the fitting recipe depends on the particular rows.
Everything here was implemented and scored under the protocol above. The gain column is the best any single generation saw.
| candidate | number that killed it |
|---|---|
| a single vendor chars-per-token constant | chars/4 reads 52.25 on Claude 4.7+ and 28.35 on o200k on the same rows. Both documented rules are stated for English and neither lands within 10% of either Claude generation on English: on content-matched English Wikipedia chars/4 reads 23.51 with a -23.05% bias and chars/3.5 reads 13.02 |
| one constant that serves two vendors | Claude 4.7+ spends 3.078 characters per token on English against o200k's 4.910, and 4.910 / 3.078 = 1.60 falls either side of 4, so each vendor's rule is wrong for the other in the opposite direction and by almost the same magnitude |
| a language label instead of counters | it is not a counter, and at zero Wikipedia in the mixture it scores 80.54 on English because those coefficients then have nothing to be fitted on |
| a per-generation feature set instead of one shared set | -0.74 on Claude pre-4.7. Where swapping wins it wins by 0.196 and 0.277, both under the bar, and it costs four counting implementations that all have to stay correct forever |
| a per-message constant (intercept) | -2.13 on o200k, -2.21 on cl100k |
| candidate | number that killed it |
|---|---|
| refit on 592,284 chunks and 1.24 billion characters | held-out English Wikipedia 10.56 to 23.77, source code 7.77 to 14.55, the calibration corpus 5.04 to 10.42. It does close the language gap, taking held-out European from 20.70 to 6.99 and wiki-40 from 19.50 to 9.12 |
| a better mixture weight for that refit | ten complete refits from 90% Wikipedia down to none of it trace a smooth curve and no point on it reaches the corner where both halves are good. Neither end is inefficient: the shipped coefficients are within 0.5 points of the best a single vector could do on English and on code, and the refit is within 0.3 of the best on Wikipedia |
| a larger corpus without a coverage audit | Hindi goes to 29.93 and Thai to 27.98 under that refit, because wiki-40 contains neither language and the other bucket was priced on Greek alone |
| refit on the doubled 2,095-chunk corpus | the 511 held-out rows in the calibrated languages get worse by 1.7 to 3.5 points, and the refit breaches the sanity gates on three of five generations with emoji between 0.000 and 1.203, cjk at 1.413 and punctRun at 0.228 |
| candidate | number that killed it |
|---|---|
| 25 honest orthographic counters: word buckets split by accent, consonant run and vowel ratio, plus capitalisation, run and length counters | recovers 30% of the English error a language label recovers and 24% of the code error, while regressing 0.35 to 0.53 points on the calibration corpus. It closes the frontier by 0.07 points of 4.44 |
| word buckets crossed with accent presence, 4 extra floats | English 23.77 to 20.54, and source code goes the wrong way at -8.9% recovery |
| the best honest 47-bucket shape under NNLS | 19.23 on held-out English. Unconstrained least squares on the identical design reaches 16.33 with 13 of its 47 floats negative, so the constraint costs 19 points of achievable recovery here. On the shipped 22 the same experiment moves nothing |
| a 23rd bucket for letters in scripts no class covers | +1.25 on cl100k, and it clears the bar only on a corpus containing an uncovered script |
latacc split at U+1E00, Vietnamese apart from European accents |
+0.76 at 2,095 rows, and at 592,284 rows carrying twenty accent-bearing languages it is worth +0.10 at most |
the emoji pool split into its six atoms |
+0.00. Per code point every emoji class sits at 2.5 to 4.5 |
| Arabic and Hebrew pooled as one abjad bucket | +0.15. The value is in separating them, not in grouping them |
| Greek split out as well, or all five scripts separate | +0.11. Once Arabic and Hebrew are apart the remaining three price alike |
| a CJK run count, one unit per unbroken CJK run | +0.17 on Claude 4.7+ and -0.26 on o200k, -0.50 on cl100k. It takes classical Chinese from 16.2% to 7.6% on Claude 4.7+, so if you only ever score Claude it is the single highest-value change you can make to this formula |
| whitespace split into free seams and paid whitespace | +0.09 on Gemini, under the bar even there |
| a capitalised-word marker | +0.09 here, +0.50 on the 592,284-row corpus where German is present. The mechanism is German noun capitalisation, which both costs a token and evicts the word from the cased vocabulary, and it carries 19% of the English-to-German gap on Claude 4.7+ |
| accented Latin priced per word span instead of per letter | +0.00. Marginally better on cl100k, worse on both Claude columns |
| a "first accent per 15 ASCII letters" non-linear clamp | -0.04. NNLS prefers the corpus-weighted average to the conditional |
| a per-digit-run constant on top of the 3-digit groups | +0.01 |
| a sentence-boundary count | +0.00. Its mass is already inside the punctuation bucket |
| total UTF-8 byte length | +0.32 on Claude 4.7+ only, and it duplicates the script buckets |
| other-script letters priced by UTF-8 bytes | -2.66. The five scripts in that bucket split 2-byte against 3-byte, so the rewrite re-weights the coefficient the wrong way |
| short ALL-CAPS acronyms priced apart; a per-character term beyond 5 letters | never selected |
| candidate | number that killed it |
|---|---|
| ten random letter pairs, mass-matched to the winning set | English 23.77 / 23.77 / 23.79 against the winner's 8.33, coefficients pinned at zero. Under the admission rule the three controls score +0.000, +0.000 and -0.001 against the selected set's +1.681 |
the six strongest English-marking pairs, s$ e$ ^t th d$ ^c |
exactly 23.77 in both additive forms. A counter that fires on the cheap side can only add cost under non-negativity |
| a float per marker instead of one pooled float | English 9.48 against the pooled float's 8.33, and it fails the admission rule at -0.202 on the calibration corpus. This is the learned n-gram table in its smallest form, and it is worse than the single float |
| an unguarded pooled marker on a full freeze of the default table | the Vietnamese stratum goes 5.66 to 18.82 against a 2.10 bar, Japanese source files +2.66 against 0.47, Chinese file prose +2.71 against 1.96, the 40-to-250 band 7.22 to 8.34 and held-out source code 7.77 to 10.66 |
| the gated marker on a full freeze, refitting nothing else | it holds the home block, 0 / 0 / 0 / 1 / 0 of the 25 watched strata over their bars. It loses every held-out cut to the frozen-six refit that ships: European 8.60 against 8.10, source code 8.66 against 7.75, 39-language Wikipedia 10.76 against 10.48, the validation set 7.19 against 6.72 |
| an unguarded four-pattern set as the shipped profile | at the profile's own rung it puts 1 / 1 / 8 / 8 / 11 of the 25 watched home strata over their noise bars across the five generations, against 1 / 0 / 2 / 0 / 4 for the gated six. Under a full freeze of the whole default table it is 6 / 8 / 7 / 11 / 10 against 0 / 0 / 0 / 1 / 0 |
| re-running the selection with 39-language Wikipedia as the objective instead of the eight European targets | wiki-40 closes to 9.95 and held-out European opens to 9.38. No set is weakly better on both published numbers, so the trade is a property of the language roster and not of the guards |
Russian gets consistently worse with punctRun, by 0.83 to 1.32 points on every generation: 8.6 to
9.5 on Claude 4.7+, 7.2 to 8.6 on o200k, 4.7 to 5.5 on cl100k. Cyrillic is priced per letter, has no
run term of its own, and encyclopedic Russian is the corpus's least punctuation-dense register, so
it absorbs a little more of the shared fit. Vietnamese, Chinese and classical Chinese each moved
against by under 0.75 in the same change. No generation-level regression exists there: all five
improved overall by 0.38 to 1.15 points, which is the unit the admission rule is written in.
Three strata on the Claude 4.7+ column are worse under the shipped 22-bucket shape than under a 16-bucket one: Chinese at 4.1 against 3.4, Russian at 9.5 against 8.9, and Vietnamese at 7.1 against 7.0, while seventeen improved, several by more than five points. Widening the shared feature set from four tokenizer columns to five costs exactly two cells: Chinese on Claude 4.7+ from 3.39 to 3.72, and TypeScript on o200k from 5.93 to 6.35.
Two counters were found sitting unused, and the finding is about process. shift counts a
capitalised not-all-caps Latin word. The feature code had computed it from the start and nothing
ever scored it. On the 592,284-row corpus it is worth +0.500 and +0.305 on the two Claude columns,
and it is near zero on the 808-row corpus, which is why it does not ship. The loader discarded 54 of
the 87 atoms the measurement had already produced, and nothing in the harness printed what was being
thrown away.
Vocabulary knowledge is invisible to buckets, and it is the largest single fact about this formula. Within English, at five letters a common word costs 1.20 tokens and a rare one 2.55, a factor of 2.1 that depends on nothing except whether the tokenizer has seen the word. That difference is exactly the content of the 29 MB table this formula is refusing to carry, and a length-binned coefficient can only choose where to sit between the two. Across Latin-script languages the same wall is the whole story. The blind-subset evidence is in the vocabulary wall. Restrict both populations to lowercase pure-ASCII words with no consonant run of four or more and a vowel ratio above 0.35, so that no code-point counter can tell them apart. The German, Dutch, Polish and Finnish gaps to English then grow from a -29% to -37% band up to -38% to -47%. The wall appears symbol by symbol as well: U+2192 RIGHTWARDS ARROW costs 1 Claude token and U+2191 UPWARDS ARROW costs 4; U+00B7 MIDDLE DOT costs 1 and U+25E6 WHITE BULLET costs 4. Some symbols got a dedicated merge and their neighbours did not, and no code-point range separates them.
The default profile's language coverage is narrow, and the multilingual profile's is a different narrow. The default reads 10.4% to 42.0% across the 24 Latin-script languages of a held-out 39-language Wikipedia set that its calibration corpus does not contain, with a one-sided bias near -20%. The multilingual profile takes 29 of those 39 languages below the default. It does not serve every Latin-script language. On that held-out set it is worse than the default on Hungarian by 5.4 points, 10.41 to 15.81.
Which languages a marker set serves is itself a choice inside the admissible region. Measured against an unguarded set that was scored and not shipped, the six gated patterns improve German (10.28 to 8.87), Italian (9.19 to 8.01), English (8.72 to 7.66), Vietnamese (8.64 to 7.68) and Spanish (7.60 to 6.71). They cost Dutch (8.58 to 12.72), Malay (9.12 to 14.27), Indonesian (8.43 to 12.02), Norwegian (7.71 to 10.61), Danish (7.84 to 10.13), French (7.87 to 10.05) and Welsh (29.30 to 33.98). Neither set serves all of them.
That chart is what the coverage gap does to the benchmark. Every rung was re-run over the combined 2,095 chunks under the same protocol with zero API calls, and the packages were executed this time instead of quoted. novocab's rows are the frozen shipped default coefficients evaluated in fold, so the 808-row column reads 5.04 instead of the out-of-fold 5.26. The chart carries every rung; the table below excerpts the four that carry the argument.
| rung | Claude 4.7+, 808 | Claude 4.7+, 2,095 | o200k, 808 | o200k, 2,095 |
|---|---|---|---|---|
chars/4 |
52.25 | 49.86 | 28.35 | 25.21 |
| OpenClaw | 38.94 | 39.47 | 16.76 | 16.22 |
| bpe-lite 0.5.1 | 29.08 | 26.93 | 0.00 | 0.00 |
| novocab, default, in fold | 5.04 | 9.21 | 6.02 | 9.36 |
The direction of that change is the opposite of what it looks like. Adding 512 European chunks makes
chars/4 better, from 52.03 on the non-European rows to 43.17 on the European ones, because
European languages run 2.06 to 2.47 characters per token, far closer to 4 than the calibrated
corpus's centre of gravity: classical Chinese runs 0.78, Chinese 0.92, Korean 1.05, Japanese 1.09,
Thai 1.16, Hebrew 1.46, Hindi 1.46, Greek 1.53, Arabic 1.55. novocab is the only rung in the table
that degrades on European text, from 5.69 to 20.09, and every one of the 45 rung-by-target gaps
narrowed because novocab moved. On the 1,583 non-European chunks the gap to chars/4 is intact at
52.03 against 5.69.
The marker set is conditional on a distribution. It was selected on Wikipedia in 39 languages
under a cap measured against a 12-language calibration corpus, and both of those rosters are
choices. Between 1,435 and 2,073 admissible sets exist at every size from two to eight patterns, so
the shipped six are one point in a large region and not a unique answer. Re-running the selection
with the 39-language mixture as the objective instead of the eight European targets gets held-out
wiki-40 to 9.95 and opens held-out European to 9.38, and no set is weakly better on both. The gates
are also normalisation-dependent in one place: #r reads an accented code point inside a word run,
and a word run ends at a combining mark, so an NFD corpus would decompose accented letters into an
ASCII base plus a standalone mark and #r would stop firing. Every number here is on an NFC corpus.
A script no bucket covers is priced as a Latin word, silently. The script classifier enumerates Han, kana, Hangul, Cyrillic, Greek, Arabic, Hebrew, Thai and Devanagari. Every other alphabetic code point falls through to the Latin path, so Armenian, Georgian, Bengali, Tamil and the rest are counted as English words of the same length. Measured on 2,351 Armenian Wikipedia chunks, the error is 50.9 / 68.3 / 44.4 / 86.0 / 62.6 MAPE across the five generations, three to eight times the error on any covered script. Nothing warns you: the code-point partition audit still passes, because a misclassification is not a leak.
Strings under 40 characters degrade. Roughly 8% at 40 to 250 characters, 8% to 17% at 25 to 40, 12% to 21% at 15 to 25, and 17% to 29% at 5 to 15, with the bias turning negative. Below 15 characters this is not fit for per-string decisions. A worked failure: a 25-character string made of three emoji and three short words costs 18 Claude tokens and this formula says 23, because a string dominated by emoji is entirely at the mercy of an averaged emoji coefficient.
The emoji coefficient is the least trustworthy float shipped. The bucket pools six atoms, four
of them zero on all 808 corpus chunks, so the fitted number is carried entirely by ordinary symbols
(116 chunks) and astral emoji (11 chunks) and describes ordinary symbols. Its bootstrap 95% interval
runs from 2.07 to 14.05 against a published 5.67236. Per-class costs for ZWJ sequences, skin tones
and regional indicators are known from direct frame probes and are consistent with the pooled figure
per code point, but nothing in the fit measures them, so any claim about a ZWJ family sequence from
this coefficient is extrapolation.
Classical Chinese is the worst stratum on all five generations, at 13.7% on Claude and 23.7% on Gemini, biased 12% to 24% low; on 49 rows it reads 12.3. Outside it the weakest cells are Japanese source files on Gemini at 10.7% and Thai on Gemini at 11.1%, and on the Claude columns Russian at 9.5%, or 11.0 across two registers at n=128.
Several buckets rest on thin evidence. In the calibration corpus latacc rests on 32 Vietnamese
chunks and no other accented-Latin language, so the shipped float is a Vietnamese float. The
validation corpus answers that with 110,002 accented-Latin characters across nine languages, and the
answer is mixed: the bucket still earns its place with Vietnamese deleted, but a direct probe
measures the mechanical cost of an accent on Claude 4.7+ at close to zero against a shipped
coefficient of 1.09752, and the bootstrap agrees from a third direction. The Arabic and Hebrew split
rests on 64 rows of one register here; it was re-checked on 64 fresh rows and held, and confirmed
again at 592,284 rows. Vocalised Arabic and pointed Hebrew were never re-probed after the split.
The published floats are worth about one decimal place each. None is pinned at the five places it
is printed to, seven are not pinned even at the units digit, and the 1/sqrt(N) law says no
achievable corpus changes that. Copy them exactly for reproducibility; do not treat a fourth decimal
as information, do not compare two coefficients that differ in the third, and do not carry a
difference of 0.001 into an argument.
- A Claude model id appears that is not in the ids listed under "which column". Do not assume a
new id joins the newer family. The break between
sonnet-4-5andopus-4-7moved English by 45%, and nothing in the version number said so. - A Gemini model returns a count the
gemma3SentencePiece file does not predict. The check is exact and takes one call:countTokens(text) == len(gemma3.encode(text)) + 1. That held on 418 of 418 strings, and a separate 123-probe battery returned byte-identical counts on all 21 generative model ids. Google's owngoogle-genaiSDK routes Gemini 3.1 and 3.5 to a different tokenizer loaded from HuggingFace, and measured against the live API that routing is wrong today on all 38 corpus chunks where the two artefacts disagree. The mechanism is a lossy conversion and not a new vocabulary: both artefacts hold 262,144 pieces with identical ids, and the HTML-tag pieces that differ are declaredUSER_DEFINEDin the SentencePiece file, a flag the conversion did not carry over. Trust the probe, not the SDK's table. - OpenAI ships an encoding that is not
o200k_base. Checktiktoken's model map instead of a release announcement.o200k_harmonylooked like a new encoding and changes no merge. - Your text drifts out of the calibrated register. The corpora are code, technical markdown, JSON and encyclopedic prose. Dialogue, OCR noise, minified code and base64 are unmeasured, and register alone is worth 10 to 20 points.
- Your text is in a Latin-script language neither profile was fitted on, or in a script the classifier does not enumerate. See the limits above.
- The number has to be right instead of close. This is an estimator. If a billing decision or a hard context limit hangs on it, count properly.
- The coefficients go stale when vendors change tokenizers. The default was fitted 2026-08-29, the multilingual table 2026-08-30.
Twenty-two counters get within about 5% of four tokenizers and 6% of the fifth, on the register they were fitted on. A twenty-third counter buys nine more languages, at the cost stated in the profile table. Every number above names the corpus it came from and the instrument that produced it. The measurements that went the wrong way sit in the same tables as the ones that went the right way.
Exact counters, when you need a count and not an estimate:
- sanderland/ctok. Reconstructs Claude token counts offline by least-pieces tiling over a measured vocabulary. It reproduced both Claude ground-truth columns on all 808 chunks with zero tokens of error out of 2.37 M, so on this corpus it is not an estimator; it is the tokenizer. It also served as the oracle for every later round here, which removes the API from the loop entirely. Two conditions: pin the family by probe instead of by version string, and match the isolation convention.
- openai/tiktoken and gpt-tokenizer for the OpenAI encodings.
- google/sentencepiece over the
gemma3model file published in google/gemma_pytorch, for Gemini and Gemma.
Other estimators, all read as primary sources and measured above:
- tokenx. 2 kB, segment-and-rule-table, calibrated against
o200k_base. - bpe-lite and ai-tokenizer. Real BPE with shipped vocabularies per provider, 494 kB to 30 MB.
- infinigence/tokenestimate. Ten-bucket linear regression over character classes, and the source of the one non-linear idea in the field.
- petasbytes/token-approx. Four features plus a fitting methodology. Small, and its shape is not bad.
- grohan/ctoc. Counts Claude tokens across a directory the way
cloccounts lines, from a greedy longest-match vocabulary.
Background:
- Schmidt et al., Tokenization Is More Than Compression, arXiv:2402.18376. Introduces PathPiece, which segments text into the minimum number of tokens for a given vocabulary, and reports that fewer tokens do not lead to better downstream performance.
- MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment, arXiv:2606.27019. Minimum-token-path segmentation with a unigram score as tiebreak.
Least-pieces tiling of that kind is what ctok implements, which is why an exact Claude
reconstruction is possible at all without the merge table.
MIT. See LICENSE.