Summary
gpt_oss.tokenizer.get_tokenizer() copies the o200k_base special tokens and then generates every reserved_* token from 200013 through 201087.
o200k_base already assigns ID 200018 to <|endofprompt|>, so the generated <|reserved_200018|> creates a second special-token name for the same ID.
Impact
The tokenizer's special-token mapping is no longer one-to-one. Encoding <|endofprompt|> yields ID 200018, while inverse special-token lookup can resolve that ID to the later reserved alias instead, breaking special-token round-trip expectations and making the local tokenizer inconsistent with a collision-free vocabulary.
The same collision was identified and fixed independently in openai/tiktoken; the local tokenizer implementation in this repository still reproduces the duplicate.
Proposed resolution
When generating Harmony reserved_* entries, skip IDs already occupied by explicit/base special tokens. Add a regression that asserts all special-token IDs are unique and that <|reserved_200018|> is not created while <|endofprompt|> remains mapped to 200018.
Summary
gpt_oss.tokenizer.get_tokenizer()copies theo200k_basespecial tokens and then generates everyreserved_*token from 200013 through 201087.o200k_basealready assigns ID200018to<|endofprompt|>, so the generated<|reserved_200018|>creates a second special-token name for the same ID.Impact
The tokenizer's special-token mapping is no longer one-to-one. Encoding
<|endofprompt|>yields ID 200018, while inverse special-token lookup can resolve that ID to the later reserved alias instead, breaking special-token round-trip expectations and making the local tokenizer inconsistent with a collision-free vocabulary.The same collision was identified and fixed independently in
openai/tiktoken; the local tokenizer implementation in this repository still reproduces the duplicate.Proposed resolution
When generating Harmony
reserved_*entries, skip IDs already occupied by explicit/base special tokens. Add a regression that asserts all special-token IDs are unique and that<|reserved_200018|>is not created while<|endofprompt|>remains mapped to 200018.