Go Module v0.1.0

Go Module Release

This release of the Go tokenizers module is compatible with Rust library version built-locally.

Installation

go get github.com/amikos-tech/pure-tokenizers@v0.1.0

Features

• CGo-free implementation using purego
• Automatic library download and caching
• Support for multiple platforms (Linux, macOS, Windows)
• Compatible with HuggingFace tokenizers

Requirements

• Go 1.24 or later
• Compatible Rust tokenizers library (downloaded automatically)

Environment Variables

• TOKENIZERS_LIB_PATH: Override library path
• TOKENIZERS_GITHUB_REPO: Custom GitHub repository
• TOKENIZERS_VERSION: Specific Rust library version to use

Documentation

See the README for detailed usage instructions.

What's Changed

• feat: Test GH download by @tazarov in #17
• feat: Fixes, more tests and examples by @tazarov in #18
• chore: Increasing test coverage by @tazarov in #19
• feat: ABI compatibility checks by @tazarov in #23
• [ENH] Split Rust and Go release cycles (#24) by @tazarov in #25
• feat: implement ABI-compatible library download system by @tazarov in #27
• [ENH] Complete README.md overhaul for better developer experience by @tazarov in #29
• [BUG] Fix release circular dependency issue (#20) by @tazarov in #30
• Add Claude Code GitHub Workflow by @tazarov in #37
• feat: HuggingFace tokenizer loading functionality by @tazarov in #36
• [PERF] Reuse HTTP client for HuggingFace downloads by @tazarov in #51
• [PERF] Add comprehensive benchmark suite by @tazarov in #53
• [ENH] Honor Retry-After header in retry logic by @tazarov in #56
• [DOC] Document HuggingFace cache structure by @tazarov in #58
• [DOC] Documentation and examples for HuggingFace support by @tazarov in #59
• [PERF] Only run benchmark CI on code change by @tazarov in #61
• [BUG] Fix decode function crash with HuggingFace tokenizers (#38) by @tazarov in #62
• [ENH] Make HTTP client pooling configuration configurable by @tazarov in #63

Full Changelog: v0.0.1...v0.1.0

Rust Library rust-v0.1.0

Rust Tokenizers Library Release

This release contains pre-built tokenizers libraries for multiple platforms.

Supported Platforms

• Linux x86_64 (GNU)
• Linux aarch64 (GNU)
• Linux x86_64 (MUSL)
• Linux aarch64 (MUSL)
• macOS x86_64
• macOS aarch64 (Apple Silicon)
• Windows x86_64

Installation

Download the appropriate archive for your platform and extract the library file.

Usage with Go bindings

The Go bindings will automatically download these libraries when needed.
You can also set TOKENIZERS_LIB_PATH environment variable to use a specific library.

What's Changed

• feat: Test GH download by @tazarov in #17
• feat: Fixes, more tests and examples by @tazarov in #18
• chore: Increasing test coverage by @tazarov in #19
• feat: ABI compatibility checks by @tazarov in #23
• [ENH] Split Rust and Go release cycles (#24) by @tazarov in #25
• feat: implement ABI-compatible library download system by @tazarov in #27
• [ENH] Complete README.md overhaul for better developer experience by @tazarov in #29
• [BUG] Fix release circular dependency issue (#20) by @tazarov in #30
• Add Claude Code GitHub Workflow by @tazarov in #37
• feat: HuggingFace tokenizer loading functionality by @tazarov in #36
• [PERF] Reuse HTTP client for HuggingFace downloads by @tazarov in #51
• [PERF] Add comprehensive benchmark suite by @tazarov in #53
• [ENH] Honor Retry-After header in retry logic by @tazarov in #56
• [DOC] Document HuggingFace cache structure by @tazarov in #58
• [DOC] Documentation and examples for HuggingFace support by @tazarov in #59
• [PERF] Only run benchmark CI on code change by @tazarov in #61
• [BUG] Fix decode function crash with HuggingFace tokenizers (#38) by @tazarov in #62
• [ENH] Make HTTP client pooling configuration configurable by @tazarov in #63

Full Changelog: v0.0.1...rust-v0.1.0

v0.0.1

What's Changed

• feat: Windows support by @tazarov in #2
• feat: License and codeowners by @tazarov in #7
• fix: Fix CI windows tests by @tazarov in #10
• feat: Improved Rust error handling by @tazarov in #12
• feat: Adding LICENSE.md to distributable packages by @tazarov in #15

New Contributors

• @tazarov made their first contribution in #2

Full Changelog: https://github.com/amikos-tech/pure-tokenizers/commits/v0.0.1