Skip to content

Go Module v0.1.4

Choose a tag to compare

View all tags
@github-actions github-actions released this 28 Feb 17:38
· 6 commits to main since this release
v0.1.4
6db29c3
This commit was signed with the committer’s verified signature.
tazarov Trayan Azarov
GPG key ID: 8DC2D17009F8F1CB
Verified
Learn about vigilant mode.

Go Module Release

This release of the Go tokenizers module is compatible with Rust library version built-locally.

Installation

go get github.com/amikos-tech/pure-tokenizers@v0.1.4

Features

  • CGo-free implementation using purego
  • Automatic library download and caching
  • Support for multiple platforms (Linux, macOS, Windows)
  • Compatible with HuggingFace tokenizers

Requirements

  • Go 1.24 or later
  • Compatible Rust tokenizers library (downloaded automatically)

Environment Variables

  • TOKENIZERS_LIB_PATH: Override library path
  • TOKENIZERS_VERSION: Specific Rust library version to use
  • GITHUB_TOKEN / GH_TOKEN: Optional GitHub API auth for fallback requests

Documentation

See the README for detailed usage instructions.

What's Changed

  • [TST] Eliminate HFHubBaseURL global mutations in tests by @tazarov in #100
  • [REL] Publish signed releases index to R2 by @tazarov in #101
  • release: externalize and locally test releases index generation by @tazarov in #102

Full Changelog: v0.1.3...v0.1.4

Contributors

tazarov
Assets 2
Loading