[Tokenizer] Add tiktoken format loader for OpenAI BPE vocabularies.#23663

Merged

benvanik merged 6 commits intomainfrom

users/benvanik/tiktoken

Mar 5, 2026

Commits on Mar 5, 2026

[Tokenizer] Add tiktoken format loader for OpenAI BPE vocabularies.

benvanik
and
claude
committed
[Base] Harden base64 encode/decode with IREE-native types and checked math.

benvanik
and
claude
committed
[Tokenizer] Add tiktoken integration tests, smoketest, and benchmark support.

benvanik
and
claude
committed
[Tokenizer] Fix CodeQL regex backtracking warning in JSON compaction.

benvanik
and
claude
committed
[Tokenizer] Add all 7 tiktoken encoding configs and fix p50k_base special token ID.

benvanik
and
claude
committed
[Tokenizer] Cache FALSE results in pair validation cache for 6x p50k_base speedup.

benvanik
and
claude
committed