Go Module v0.1.5

Go Module Release

This release of the Go tokenizers module is compatible with Rust library version built-locally.

Installation

go get github.com/amikos-tech/pure-tokenizers@v0.1.5

Features

• CGo-free implementation using purego
• Automatic library download and caching
• Support for multiple platforms (Linux, macOS, Windows)
• Compatible with HuggingFace tokenizers

Requirements

• Go 1.24 or later
• Compatible Rust tokenizers library (downloaded automatically)

Environment Variables

• TOKENIZERS_LIB_PATH: Override library path
• TOKENIZERS_VERSION: Specific Rust library version to use
• GITHUB_TOKEN / GH_TOKEN: Optional GitHub API auth for fallback requests

Documentation

See the README for detailed usage instructions.

What's Changed

• chore(rust): bump tokenizers to 0.22.2 by @tazarov in #103

Full Changelog: v0.1.4...v0.1.5