Number tokenization

[hammer]

Make sure our tokenizer handles numbers well, e.g. right-to-left parsing, place-aligned chunking, digit splitting, etc.

• From Digits to Decisions: How Tokenization Impacts Arithmetic in LLMs
• Arcee Trinity Large Technical Report Section 2.1.1 notes a standard regex has pathological backtracking behavior and details their fix.

[pc0618]

🤖 Initial implementation and 5821 training plan update:

I added a 4915-style numeric-aware tokenizer variant to the tokenizer-sensitivity sweep. The implementation is a serializable HF tokenizers pretokenizer sequence:

1. isolate numeric runs from surrounding text;
2. cap numeric runs into chunks of at most 510 numeric characters;
3. apply right-aligned place grouping within each bounded chunk, so 1234567 -> 1|234|567;
4. continue through the original upstream tokenizer pretokenizer.

This keeps the desired Singh/Strouse-style fixed-place grouping while avoiding the unbounded zero-width lookahead over arbitrarily long digit strings. For digit runs up to 510 characters, the splitter matches the original lookahead boundary behavior exactly; longer runs are bounded deliberately to avoid the pathological runtime described in the Arcee/Liu-style note.

Implemented experiment arms:

• gpt-oss-place-digits, initialized from openai/gpt-oss-20b
• llama-place-digits, initialized from meta-llama/Meta-Llama-3.1-8B

These are explicit sibling arms to the existing learned controls (gpt-oss, llama), not silent replacements. They train the same 262,144-token base vocab, then derive 131,072 / 32,768 / 8,192 by BPE-rank truncation. Step hashes and metadata include the place-aligned digit policy so these artifacts remain distinct.

Submitted Iris prep run for the two numeric-aware HF arms on the same 50B-token-equivalent deterministic random Datakit subset used for #5821:

/pc0618/tokenizer-sweep-5821-4915-place-digits-hf-50b-random

Submission settings:

• dataset: data/datakit/sample/2026-05-26
• tokenizer train tokens: 50,000,000,000
• sample mode: random-shards
• seed: 5821
• families: gpt-oss-place-digits,llama-place-digits
• sizes: 262144,131072,32768,8192
• region: europe-west4

Verification added locally:

• equivalence to the old lookahead splitter for digit-run lengths 1..510;
• isolation of contiguous digit runs from surrounding text;
• tokenizer pretokenizer serialization smoke test;
• 100k-digit uninterrupted string runtime guard.

Training/eval plan for crossing this off under #5821:

• finish the numeric-aware tokenizer prep run;
• retokenize the same 50B train cache and validation/holdout material for gpt-oss-place-digits and llama-place-digits;
• train the same fixed MoE architecture/budget as the current 5821 learned-tokenizer comparison;
• compare each numeric-aware arm against its same-family non-numeric control using BPB / bpb_fixed, per-domain and per-language metrics, plus numeric/arithmetic-heavy slices where available;
• report whether the 4915 policy materially changes MoE training/eval behavior without changing corpus, vocab-size ladder, training budget, or eval denominator.

[pc0618]

Numeric-aware tokenizer variant update

The *-place-digits variants currently differ from their corresponding normal gpt-oss / llama variants only in numeric pretokenization before BPE training.

The three numeric-specific changes are:

1. Digit-run isolation: contiguous Unicode numeric runs are separated from surrounding text before the normal tokenizer pretokenizer runs. Example: abc1234567def is handled as abc | 1234567 | def before numeric chunking.
2. Right-to-left, place-aligned 3-digit grouping: each digit run is split from the right into groups of three, with a leading group of one or two digits if needed. Example: 1234567 -> 1 | 234 | 567. This keeps each 3-digit token aligned to a fixed place-value block.
3. Bounded, backtracking-safe implementation: instead of one zero-width lookahead over arbitrary-length digit strings, the implementation caps numeric-run chunks at 510 characters, peels a leading 1-2 digit group when needed, then splits exact groups of 3. This preserves the intended place-aligned split for normal numeric runs and avoids pathological regex backtracking on very long digit strings.

Implementation reference in this branch:

• experiments/datakit_testbed/tokenizer_sweep_20260526.py
• tests/datakit_testbed/test_tokenizer_sweep_20260526.py

Paper/source rationale:

• Singh & Strouse (2024), "Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs": motivates right-to-left numeric grouping; they find right-to-left number tokenization substantially improves arithmetic behavior. https://arxiv.org/abs/2402.14903
• Schmidt, Reddy, Tanner & Pinter (2025), "Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier": uses right-to-left 3-digit numeric pretokens and gives the same 1234567 -> [1, 234, 567] behavior. https://arxiv.org/html/2504.00178v2

Latest preliminary tokenizer-MoE validation results show the numeric-aware variants are useful for gpt-oss, especially at d768:

comparison	vocab	dim	baseline all BPB	digits all BPB	delta
gpt-oss -> gpt-oss-place-digits	8k	d512	1.4006	1.3525	-0.0481
gpt-oss -> gpt-oss-place-digits	8k	d768	1.1432	1.1134	-0.0298
gpt-oss -> gpt-oss-place-digits	32k	d512	1.4418	1.3652	-0.0766
gpt-oss -> gpt-oss-place-digits	32k	d768	1.1705	1.1051	-0.0654
llama -> llama-place-digits	8k	d512	1.4729	1.4852	+0.0123
llama -> llama-place-digits	8k	d768	1.2196	1.2203	+0.0007
llama -> llama-place-digits	32k	d512	1.4656	1.4277	-0.0379
llama -> llama-place-digits	32k	d768	1.2076	1.1751	-0.0325

Interpretation: this is a promising signal for explicit numeric pretokenization, especially for GPT-OSS-family tokenizers and 32k vocabularies. The 8k Llama-family result is neutral/slightly worse in this preliminary train50B tokenizer-MoE table, so we should keep the numeric-aware arm as an explicit sibling variant rather than silently replacing the default tokenizer.