Merge pull request #99 from amikos-tech/codex/releases-fallback-finalize

Harden release downloader and finalize releases-first model

Author: tazarov

.github/workflows/benchmark.yml

@@ -44,7 +44,7 @@ jobs:
       uses: ./.github/actions/setup-rust
 
     - name: Build Rust library
-      run: cargo build --release
+      run: cargo build --release --locked
 
     - name: Run benchmarks
       run: |
@@ -95,7 +95,7 @@ jobs:
       run: go install golang.org/x/perf/cmd/benchstat@latest
 
     - name: Build Rust library
-      run: cargo build --release
+      run: cargo build --release --locked
 
     - name: Run benchmarks on PR branch
       run: |

.github/workflows/go-release.yml

@@ -108,8 +108,8 @@ jobs:
 
           ### Environment Variables
           - `TOKENIZERS_LIB_PATH`: Override library path
-          - `TOKENIZERS_GITHUB_REPO`: Custom GitHub repository
           - `TOKENIZERS_VERSION`: Specific Rust library version to use
+          - `GITHUB_TOKEN` / `GH_TOKEN`: Optional GitHub API auth for fallback requests
 
           ### Documentation
           See the [README](https://github.com/${{ github.repository }}) for detailed usage instructions.

.github/workflows/test-download.yml

@@ -36,16 +36,10 @@ jobs:
     - name: Test automatic download
       env:
         TOKENIZERS_VERSION: ${{ github.event.inputs.test_version || 'latest' }}
+        TOKENIZERS_REQUIRE_ONLINE_TESTS: "1"
       run: |
-        go test -v -run "TestDownloadFunctionality" -timeout=10m
+        go test -v -run "TestDownloadLibraryFromGitHub|TestDownloadFunctionality/GetAvailableVersions" -timeout=10m
 
     - name: Test library info
       run: |
         go test -v -run "TestGetLibraryInfo"
-
-    - name: Test with environment variable override
-      env:
-        TOKENIZERS_GITHUB_REPO: ${{ github.repository }}
-        TOKENIZERS_VERSION: ${{ github.event.inputs.test_version || 'latest' }}
-      run: |
-        go test -v -run "TestDownloadFunctionality/GetAvailableVersions" -timeout=5m

.gitignore

@@ -20,3 +20,5 @@ go.work.sum
 .env
 /unit.xml
 target/
+.agents/skills/releasing-to-r2/
+skills-lock.json

CLAUDE.md

@@ -109,7 +109,7 @@ The system follows a priority order for loading the tokenizer library:
 1. User-provided path via `WithLibraryPath()` option
 2. `TOKENIZERS_LIB_PATH` environment variable
 3. Cached library in platform-specific directory
-4. Automatic download from GitHub releases to cache
+4. Automatic download from `releases.amikos.tech` (with GitHub Releases fallback) to cache
 
 ### Version Management
 The project uses a single version from `Cargo.toml` for both the library and ABI compatibility:
@@ -139,7 +139,8 @@ The project uses a single version from `Cargo.toml` for both the library and ABI
 
 **Download System (download.go)**
 - Automatic platform detection and asset selection
-- GitHub releases integration with checksum verification
+- Primary `releases.amikos.tech` endpoint with GitHub Releases fallback
+- Checksum verification for all downloaded assets
 - Intelligent caching in OS-appropriate directories
 
 **Rust Layer (src/lib.rs)**
@@ -171,9 +172,8 @@ For detailed cache structure and management, see `docs/CACHE_MANAGEMENT.md`.
 ## Environment Variables
 
 - `TOKENIZERS_LIB_PATH`: Override library path
-- `TOKENIZERS_GITHUB_REPO`: Custom GitHub repository (default: `amikos-tech/pure-tokenizers`)
 - `TOKENIZERS_VERSION`: Specific version to download (default: `latest`)
-- `GITHUB_TOKEN` or `GH_TOKEN`: GitHub authentication for API requests
+- `GITHUB_TOKEN` or `GH_TOKEN`: Optional GitHub authentication for fallback API requests
 - `HF_TOKEN`: HuggingFace authentication token for private/gated models
 - `HF_HUB_CACHE`: Override HuggingFace cache directory
 - `HF_MAX_TOKENIZER_SIZE`: Maximum tokenizer file size in bytes (default: 524288000 / 500MB)
@@ -211,4 +211,4 @@ make clean
 - **Memory Safety**: Proper cleanup of FFI resources with defer statements
 - **Buffer Management**: Zero-copy where possible, explicit memory management for C strings
 - **Cross-platform**: Uses runtime detection for platform-specific library names and paths
-- Always lint both golang and rust before commiting or pushing code
+- Always lint both golang and rust before commiting or pushing code

Cargo.lock

@@ -259,9 +259,8 @@ tokenizer, err := tokenizers.FromFile("tokenizer.json",
 | Variable | Description | Default |
 |----------|-------------|---------|
 | `TOKENIZERS_LIB_PATH` | Custom library path | Auto-detect |
-| `TOKENIZERS_GITHUB_REPO` | GitHub repo for downloads | `amikos-tech/pure-tokenizers` |
 | `TOKENIZERS_VERSION` | Library version to download | `latest` |
-| `GITHUB_TOKEN` | GitHub API token (for rate limits) | None |
+| `GITHUB_TOKEN` / `GH_TOKEN` | Optional token for GitHub API/authenticated fallback requests | unset |
 
 ### Library Loading Options
 
@@ -274,7 +273,7 @@ tokenizer, err := tokenizers.FromFile("tokenizer.json",
 // 1. User-provided path via WithLibraryPath()
 // 2. TOKENIZERS_LIB_PATH environment variable
 // 3. Cached library in platform directory
-// 4. Automatic download from GitHub releases
+// 4. Automatic download from releases.amikos.tech (with GitHub Releases fallback)
 ```
 
 ### Cache Management
@@ -291,6 +290,11 @@ err := tokenizers.ClearLibraryCache()
 
 // Download and cache a specific version
 err := tokenizers.DownloadAndCacheLibraryWithVersion("v0.1.0")
+
+// Discover release versions
+versions, err := tokenizers.GetAvailableVersions()
+// Note: current release metadata exposes only latest.json,
+// so this returns at most one latest version.
 ```
 
 #### HuggingFace Cache
@@ -406,4 +410,4 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 
 ## Acknowledgments
 
-Built on top of the excellent [Hugging Face Tokenizers](https://github.com/huggingface/tokenizers) library.
+Built on top of the excellent [Hugging Face Tokenizers](https://github.com/huggingface/tokenizers) library.

README.md

@@ -25,37 +25,36 @@ The project uses GitHub Actions for CI/CD with multiple workflows to ensure code
 - Go linting with golangci-lint
 - Caching for faster builds
 
-### 2. Build and Release Workflow (`.github/workflows/build-and-release.yml`)
+### 2. Rust Release Workflow (`.github/workflows/rust-release.yml`)
 
-**Triggers:** Git tags starting with 'v', Pull Requests, Manual dispatch
+**Triggers:** Rust release tags (`rust-v*`)
 
-**Purpose:** Build libraries for all supported platforms and create releases
+**Purpose:** Build native library artifacts for all supported platforms and publish them to the releases endpoint
 
 **Supported Platforms:**
 - Linux: `x86_64-unknown-linux-gnu`, `aarch64-unknown-linux-gnu`
 - macOS: `x86_64-apple-darwin`, `aarch64-apple-darwin`
 - Windows: `x86_64-pc-windows-msvc`
 
 **Jobs:**
-- **build**: Cross-compiles for all target platforms
-- **test**: Tests Go bindings with built libraries
-- **release**: Creates GitHub releases with all platform assets
+- **build**: Cross-compiles for all target platforms and uploads platform-specific `libtokenizers-*.tar.gz` artifacts
+- **release**: Generates `SHA256SUMS` and per-asset checksums, signs/verifies artifacts, then publishes to the releases endpoint
 
 **Artifacts:**
 - Platform-specific tar.gz archives containing the shared libraries
 - SHA256 checksum files for each archive
 - Automatic GitHub release creation for tagged versions
 
-### 3. Cross Compilation Test (`.github/workflows/cross-compile.yml`)
+### 3. Go Release Workflow (`.github/workflows/go-release.yml`)
 
-**Triggers:** Changes to Rust source code or build configuration
+**Triggers:** Go release tags (`v*`)
 
-**Purpose:** Verify cross-compilation works for all targets
+**Purpose:** Validate Go module release flow against published Rust artifacts and publish Go module release metadata
 
 **Features:**
-- Tests compilation for all supported targets
-- Includes additional targets like musl variants
-- Verifies library files are created correctly
+- Tests downloads against released native artifacts
+- Verifies Go bindings against released libraries
+- Publishes Go release output
 
 ### 4. Download Functionality Test (`.github/workflows/test-download.yml`)
 
@@ -121,7 +120,7 @@ This installs:
 
 # Or step by step
 make build
-make test-v2
+make test
 
 # Test specific functionality
 make test-download
@@ -157,7 +156,6 @@ Examples:
 
 ### User Environment Variables
 
-- `TOKENIZERS_GITHUB_REPO`: Override GitHub repository for downloads
 - `TOKENIZERS_VERSION`: Specify version to download
 - `TOKENIZERS_LIB_PATH`: Override library path
 
@@ -217,4 +215,4 @@ When contributing:
 - **CI Status**: Check GitHub Actions tab
 - **Release Status**: Monitor releases page
 - **Download Stats**: Available in GitHub insights
-- **Test Coverage**: Generated in CI artifacts
+- **Test Coverage**: Generated in CI artifacts

docs/CI-CD.md

@@ -18,7 +18,7 @@ This guide covers the deployment and release process for the CGo-free Tokenizers
 
 3. **Monitor the build:**
    - Go to GitHub Actions tab
-   - Watch the "Build and Release" workflow
+   - Watch the `rust-release.yml` and `go-release.yml` workflows
    - Release will be created automatically when complete
 
 ## Supported Platforms
@@ -31,13 +31,14 @@ The CI system builds for the following platforms:
 | Linux | ARM64 | `aarch64-unknown-linux-gnu` | `libtokenizers.so` |
 | macOS | Intel | `x86_64-apple-darwin` | `libtokenizers.dylib` |
 | macOS | Apple Silicon | `aarch64-apple-darwin` | `libtokenizers.dylib` |
-| Windows | x86_64 | `x86_64-pc-windows-msvc` | `libtokenizers.dll` |
+| Windows | x86_64 | `x86_64-pc-windows-msvc` | `tokenizers.dll` |
 
 ## Release Assets
 
 Each release includes:
 
 - **Platform-specific archives**: `libtokenizers-{arch}-{platform}.tar.gz`
+- **Manifest checksum file**: `SHA256SUMS` (primary verification source)
 - **Checksum files**: `libtokenizers-{arch}-{platform}.tar.gz.sha256`
 - **Automatic release notes**: Generated from commits and PRs
 
@@ -47,20 +48,20 @@ The Go library automatically downloads the appropriate platform library:
 
 ```go
 // Automatic download
-tokenizer, err := tokenizers.FromFile("config.json", 
-    tokenizers.WithDownloadLibrary())
+tokenizer, err := tokenizers.FromFile("config.json")
 
 // Manual path
 tokenizer, err := tokenizers.FromFile("config.json", 
     tokenizers.WithLibraryPath("/path/to/lib"))
 ```
 
+Downloads are attempted from `releases.amikos.tech` first and fall back to GitHub Releases if needed.
+
 ## Environment Variables
 
 ### For Users
 
 - `TOKENIZERS_LIB_PATH`: Override library path
-- `TOKENIZERS_GITHUB_REPO`: Custom repository for downloads
 - `TOKENIZERS_VERSION`: Specific version to download
 
 ### For CI/CD
@@ -97,7 +98,7 @@ make create-release-assets
 
 ```bash
 # Run all tests
-make test-v2
+make test
 
 # Test download functionality
 make test-download
@@ -113,15 +114,15 @@ make test-rust
 - **Purpose**: Basic testing and validation
 - **Platforms**: Linux, macOS, Windows
 
-### 2. Build and Release (`build-and-release.yml`)
-- **Trigger**: Git tags (`v*`)
-- **Purpose**: Create releases with all platform assets
-- **Features**: Cross-compilation, checksum generation, automatic releases
+### 2. Rust Release (`rust-release.yml`)
+- **Trigger**: Rust tags (`rust-v*`)
+- **Purpose**: Build and publish platform assets to releases endpoint
+- **Features**: Cross-compilation, `SHA256SUMS` + per-asset checksums, artifact publishing
 
-### 3. Cross Compilation Test (`cross-compile.yml`)
-- **Trigger**: Changes to Rust code
-- **Purpose**: Verify cross-compilation works
-- **Targets**: All supported platforms + additional variants
+### 3. Go Release (`go-release.yml`)
+- **Trigger**: Go tags (`v*`)
+- **Purpose**: Validate and publish Go module releases
+- **Features**: Tests against released Rust artifacts and publishes module release
 
 ### 4. Download Test (`test-download.yml`)
 - **Trigger**: Weekly schedule, manual dispatch
@@ -208,4 +209,4 @@ For deployment issues:
 1. Check the [CI/CD documentation](CI-CD.md)
 2. Review GitHub Actions logs
 3. Test locally with provided scripts
-4. Open an issue with detailed error information
+4. Open an issue with detailed error information