Skip to content

Validate base64 tokens when loading BPE files - #592

Open
weivwang wants to merge 1 commit into
openai:mainfrom
weivwang:fix/validate-bpe-base64
Open

Validate base64 tokens when loading BPE files#592
weivwang wants to merge 1 commit into
openai:mainfrom
weivwang:fix/validate-bpe-base64

Conversation

@weivwang

Copy link
Copy Markdown

Summary

Reject malformed Base64 token fields when loading .tiktoken BPE files instead of silently discarding invalid characters.

Python's permissive base64.b64decode turns input such as !!! into b"", so a corrupted vocabulary line could create an empty-byte token and continue loading. Enabling the standard library's strict validation keeps the existing per-line error context while preventing silent vocabulary corruption.

Testing

  • .venv/bin/python -m pytest tests/test_load.py -q (2 passed)
  • .venv/bin/python -m pytest tests -q --deselect tests/test_encoding.py::test_simple --deselect tests/test_encoding.py::test_basic_encode (31 passed, 4 deselected by existing test selection)
  • .venv/bin/python -m compileall -q tiktoken tests/test_load.py
  • git diff --check

The two deselected tests fetch tokenizer assets from openaipublic.blob.core.windows.net; they failed before assertions because the local proxy terminated TLS with UNEXPECTED_EOF_WHILE_READING. The new loader tests use local files and cover both valid and invalid Base64 deterministically.

@weivwang
weivwang marked this pull request as ready for review August 13, 2026 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant