Skip to content

README.md

Latest commit

 

History

History
41 lines (28 loc) · 832 Bytes

README.md

File metadata and controls

41 lines (28 loc) · 832 Bytes
Raw

rusty-tokey

                 _         _        _                
 _ __ _   _ ___ | |_ _   _| |_ ___ | | _____ _   _ 
| '__| | | / __|| __| | | | __/ _ \| |/ / _ \ | | |
| |  | |_| \__ \| |_| |_| | || (_) |   <  __/ |_| |
|_|   \__,_|___/ \__|\__, |\__\___/|_|\_\___|\__, |
                     |___/                   |___/

Byte Pair Encoding tokenizer. Rust core with Python bindings.

Build

maturin develop

Run

python test.py

What it does

  • Pre-tokenizes text using regex patterns
  • Counts byte pair frequencies
  • Merges most frequent pairs iteratively
  • Outputs vocabulary and merge operations

Core algorithm in Rust for speed. Python handles I/O and orchestration.

Future Work

  • fix linked-array implementation.

Acknowledgements

Made while following CS336.