Skip to content

Go Module v0.1.3

Choose a tag to compare

View all tags
@github-actions github-actions released this 28 Feb 12:34
· 21 commits to main since this release
v0.1.3
0968418
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Go Module Release

This release of the Go tokenizers module is compatible with Rust library version built-locally.

Installation

go get github.com/amikos-tech/pure-tokenizers@v0.1.3

Features

  • CGo-free implementation using purego
  • Automatic library download and caching
  • Support for multiple platforms (Linux, macOS, Windows)
  • Compatible with HuggingFace tokenizers

Requirements

  • Go 1.24 or later
  • Compatible Rust tokenizers library (downloaded automatically)

Environment Variables

  • TOKENIZERS_LIB_PATH: Override library path
  • TOKENIZERS_VERSION: Specific Rust library version to use
  • GITHUB_TOKEN / GH_TOKEN: Optional GitHub API auth for fallback requests

Documentation

See the README for detailed usage instructions.

What's Changed

  • ci: publish signed Rust releases to R2 by @tazarov in #98
  • Harden release downloader and finalize releases-first model by @tazarov in #99

Full Changelog: v0.1.2...v0.1.3

Contributors

tazarov
Assets 2
Loading