perf: speed up batch tokenization and decoding hot paths - #561

Open

eonr wants to merge 10 commits into

openai:mainfrom

eonr:codex/native-batch-tokenization

Commits on May 27, 2026

Add native batch encoding fast path
eonr
committed
c310455
Add native batch decoding fast path
eonr
committed
95336f0
Add native offset decoding fast path
eonr
committed
6673e38
Speed up encoding construction
eonr
committed
91c6c96
Speed up default special-token encoding
eonr
committed
71312ca
Speed up public encoding construction
eonr
committed
7d62b08
Lazily materialize public mergeable ranks
eonr
committed
68611c3
Fix arbitrary-text roundtrip property test
eonr
committed
9b1c3a6
Speed up unstable encoding prefixes
eonr
committed
d34f256
Defer special regex construction
eonr
committed
6e4a6d7