Skip to content

Commit 324c500

Browse files
tazarovclaude
tazarov
and
claude
committed
feat: Split Rust and Go release cycles (#24)
Implement separate release workflows for Rust and Go libraries with different versioning schemes to allow independent release cycles. - Rust releases use `rust-vX.Y.Z` tags (e.g., `rust-v0.1.0`) - Go releases use standard `vX.Y.Z` tags (e.g., `v0.1.0`) Changes: - Add rust-ci.yml for Rust-specific linting and tests - Add go-ci.yml for Go-specific linting and tests - Rename build-and-release.yml to rust-release.yml - Add go-release.yml for Go module releases - Update CI workflow to focus on integration testing - Document new release process in CLAUDE.md This allows the Rust library and Go bindings to be released independently while maintaining compatibility through versioning. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent df78e40 commit 324c500

8 files changed

Lines changed: 677 additions & 46 deletions

File tree

‎.claude/commands/feature.md‎

Whitespace-only changes.

‎.claude/commands/ifeed.md‎

Whitespace-only changes.

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
1-
name: CI
1+
name: Integration CI
22

3+
# Main integration CI that runs when both Rust and Go code changes
4+
# Individual language CI runs are in rust-ci.yml and go-ci.yml
35
on:
46
push:
57
branches: [ main ]

‎.github/workflows/go-ci.yml‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
name: Go CI
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
paths:
7+
- '**.go'
8+
- 'go.mod'
9+
- 'go.sum'
10+
- '.github/workflows/go-ci.yml'
11+
pull_request:
12+
branches: [ main ]
13+
paths:
14+
- '**.go'
15+
- 'go.mod'
16+
- 'go.sum'
17+
- '.github/workflows/go-ci.yml'
18+
19+
permissions:
20+
contents: read
21+
22+
jobs:
23+
go-test:
24+
name: Go Tests
25+
runs-on: ${{ matrix.os }}
26+
strategy:
27+
matrix:
28+
os: [ubuntu-latest, macos-latest, windows-latest]
29+
go-version: ['1.24']
30+
31+
steps:
32+
- name: Checkout code
33+
uses: actions/checkout@v4
34+
35+
- name: Set up Go
36+
uses: actions/setup-go@v5
37+
with:
38+
go-version: ${{ matrix.go-version }}
39+
40+
- name: Run Go linting
41+
uses: golangci/golangci-lint-action@v6
42+
with:
43+
version: latest
44+
args: --timeout=5m
45+
46+
# Download pre-built Rust library for testing
47+
# This assumes a recent Rust release is available
48+
- name: Download latest Rust library (Linux)
49+
if: matrix.os == 'ubuntu-latest'
50+
run: |
51+
LATEST_RELEASE=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
52+
if [ -z "$LATEST_RELEASE" ]; then
53+
echo "No Rust release found, building locally"
54+
sudo apt-get update && sudo apt-get install -y cargo
55+
cargo build --release
56+
echo "TOKENIZERS_LIB_PATH=$(pwd)/target/release/libtokenizers.so" >> $GITHUB_ENV
57+
else
58+
echo "Using Rust release: $LATEST_RELEASE"
59+
gh release download $LATEST_RELEASE --pattern "libtokenizers-x86_64-unknown-linux-gnu.tar.gz"
60+
mkdir -p test-lib
61+
tar -xzf libtokenizers-x86_64-unknown-linux-gnu.tar.gz -C test-lib
62+
echo "TOKENIZERS_LIB_PATH=$(pwd)/test-lib/libtokenizers.so" >> $GITHUB_ENV
63+
fi
64+
env:
65+
GITHUB_TOKEN: ${{ github.token }}
66+
67+
- name: Download latest Rust library (macOS)
68+
if: matrix.os == 'macos-latest'
69+
run: |
70+
ARCH=$(uname -m)
71+
if [ "$ARCH" = "arm64" ]; then
72+
ARCH="aarch64"
73+
elif [ "$ARCH" = "x86_64" ]; then
74+
ARCH="x86_64"
75+
fi
76+
LATEST_RELEASE=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
77+
if [ -z "$LATEST_RELEASE" ]; then
78+
echo "No Rust release found, building locally"
79+
cargo build --release
80+
echo "TOKENIZERS_LIB_PATH=$(pwd)/target/release/libtokenizers.dylib" >> $GITHUB_ENV
81+
else
82+
echo "Using Rust release: $LATEST_RELEASE"
83+
gh release download $LATEST_RELEASE --pattern "libtokenizers-${ARCH}-apple-darwin.tar.gz"
84+
mkdir -p test-lib
85+
tar -xzf libtokenizers-${ARCH}-apple-darwin.tar.gz -C test-lib
86+
echo "TOKENIZERS_LIB_PATH=$(pwd)/test-lib/libtokenizers.dylib" >> $GITHUB_ENV
87+
fi
88+
env:
89+
GITHUB_TOKEN: ${{ github.token }}
90+
91+
- name: Download latest Rust library (Windows)
92+
if: matrix.os == 'windows-latest'
93+
shell: pwsh
94+
run: |
95+
$latestRelease = gh release list --repo ${{ github.repository }} --limit 10 | Select-String -Pattern "^rust-v" | Select-Object -First 1
96+
if ($latestRelease) {
97+
$releaseTag = ($latestRelease -split "`t")[0]
98+
Write-Host "Using Rust release: $releaseTag"
99+
gh release download $releaseTag --pattern "libtokenizers-x86_64-pc-windows-msvc.tar.gz"
100+
New-Item -ItemType Directory -Force -Path test-lib
101+
tar -xzf libtokenizers-x86_64-pc-windows-msvc.tar.gz -C test-lib
102+
"TOKENIZERS_LIB_PATH=$(Get-Location)\test-lib\tokenizers.dll" | Out-File -FilePath $env:GITHUB_ENV -Append
103+
} else {
104+
Write-Host "No Rust release found, building locally"
105+
cargo build --release
106+
"TOKENIZERS_LIB_PATH=$(Get-Location)\target\release\tokenizers.dll" | Out-File -FilePath $env:GITHUB_ENV -Append
107+
}
108+
env:
109+
GITHUB_TOKEN: ${{ github.token }}
110+
111+
- name: Run Go tests
112+
run: |
113+
go mod tidy
114+
go test -v -coverprofile=coverage.out -timeout=30m ./...
115+
env:
116+
GITHUB_TOKEN: ${{ github.token }}
117+
118+
integration-test:
119+
name: Integration Tests
120+
runs-on: ubuntu-latest
121+
needs: go-test
122+
123+
steps:
124+
- name: Checkout code
125+
uses: actions/checkout@v4
126+
127+
- name: Set up Go
128+
uses: actions/setup-go@v5
129+
with:
130+
go-version: '1.24'
131+
132+
- name: Download latest Rust library
133+
run: |
134+
LATEST_RELEASE=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
135+
if [ -z "$LATEST_RELEASE" ]; then
136+
echo "No Rust release found, building locally"
137+
cargo build --release
138+
echo "TOKENIZERS_LIB_PATH=$(pwd)/target/release/libtokenizers.so" >> $GITHUB_ENV
139+
else
140+
echo "Using Rust release: $LATEST_RELEASE"
141+
gh release download $LATEST_RELEASE --pattern "libtokenizers-x86_64-unknown-linux-gnu.tar.gz"
142+
mkdir -p test-lib
143+
tar -xzf libtokenizers-x86_64-unknown-linux-gnu.tar.gz -C test-lib
144+
echo "TOKENIZERS_LIB_PATH=$(pwd)/test-lib/libtokenizers.so" >> $GITHUB_ENV
145+
fi
146+
env:
147+
GITHUB_TOKEN: ${{ github.token }}
148+
149+
- name: Test download functionality
150+
run: |
151+
go test -v -run "TestDownloadFunctionality|TestGetLibraryInfo"
152+
153+
- name: Test library loading
154+
run: |
155+
go test -v -run "TestGetCachedLibraryPath|TestCacheDirectory"

‎.github/workflows/go-release.yml‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
name: Go Release
2+
3+
# Triggered by v* tags for Go module releases
4+
on:
5+
push:
6+
tags:
7+
- 'v*'
8+
- '!v*-*' # Exclude pre-releases like v1.0.0-beta
9+
10+
permissions:
11+
contents: write
12+
13+
jobs:
14+
test:
15+
name: Test Go Module
16+
runs-on: ${{ matrix.os }}
17+
strategy:
18+
matrix:
19+
os: [ubuntu-latest, macos-latest, windows-latest]
20+
go-version: ['1.24']
21+
22+
steps:
23+
- name: Checkout code
24+
uses: actions/checkout@v4
25+
26+
- name: Set up Go
27+
uses: actions/setup-go@v5
28+
with:
29+
go-version: ${{ matrix.go-version }}
30+
31+
- name: Run Go linting
32+
uses: golangci/golangci-lint-action@v6
33+
with:
34+
version: latest
35+
args: --timeout=5m
36+
37+
# Download the latest Rust library release for testing
38+
- name: Get latest Rust release (Linux)
39+
if: matrix.os == 'ubuntu-latest'
40+
run: |
41+
LATEST_RUST=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
42+
if [ -z "$LATEST_RUST" ]; then
43+
echo "ERROR: No Rust release found. Please create a Rust release first."
44+
exit 1
45+
fi
46+
echo "Using Rust release: $LATEST_RUST"
47+
gh release download $LATEST_RUST --pattern "libtokenizers-x86_64-unknown-linux-gnu.tar.gz"
48+
mkdir -p test-lib
49+
tar -xzf libtokenizers-x86_64-unknown-linux-gnu.tar.gz -C test-lib
50+
echo "TOKENIZERS_LIB_PATH=$(pwd)/test-lib/libtokenizers.so" >> $GITHUB_ENV
51+
env:
52+
GITHUB_TOKEN: ${{ github.token }}
53+
54+
- name: Get latest Rust release (macOS)
55+
if: matrix.os == 'macos-latest'
56+
run: |
57+
ARCH=$(uname -m)
58+
if [ "$ARCH" = "arm64" ]; then
59+
ARCH="aarch64"
60+
elif [ "$ARCH" = "x86_64" ]; then
61+
ARCH="x86_64"
62+
fi
63+
LATEST_RUST=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
64+
if [ -z "$LATEST_RUST" ]; then
65+
echo "ERROR: No Rust release found. Please create a Rust release first."
66+
exit 1
67+
fi
68+
echo "Using Rust release: $LATEST_RUST"
69+
gh release download $LATEST_RUST --pattern "libtokenizers-${ARCH}-apple-darwin.tar.gz"
70+
mkdir -p test-lib
71+
tar -xzf libtokenizers-${ARCH}-apple-darwin.tar.gz -C test-lib
72+
echo "TOKENIZERS_LIB_PATH=$(pwd)/test-lib/libtokenizers.dylib" >> $GITHUB_ENV
73+
env:
74+
GITHUB_TOKEN: ${{ github.token }}
75+
76+
- name: Get latest Rust release (Windows)
77+
if: matrix.os == 'windows-latest'
78+
shell: pwsh
79+
run: |
80+
$latestRust = gh release list --repo ${{ github.repository }} --limit 10 | Select-String -Pattern "^rust-v" | Select-Object -First 1
81+
if (-not $latestRust) {
82+
Write-Error "No Rust release found. Please create a Rust release first."
83+
exit 1
84+
}
85+
$releaseTag = ($latestRust -split "`t")[0]
86+
Write-Host "Using Rust release: $releaseTag"
87+
gh release download $releaseTag --pattern "libtokenizers-x86_64-pc-windows-msvc.tar.gz"
88+
New-Item -ItemType Directory -Force -Path test-lib
89+
tar -xzf libtokenizers-x86_64-pc-windows-msvc.tar.gz -C test-lib
90+
"TOKENIZERS_LIB_PATH=$(Get-Location)\test-lib\tokenizers.dll" | Out-File -FilePath $env:GITHUB_ENV -Append
91+
env:
92+
GITHUB_TOKEN: ${{ github.token }}
93+
94+
- name: Run Go tests
95+
run: |
96+
go mod tidy
97+
go test -v -race -coverprofile=coverage.out -timeout=30m ./...
98+
env:
99+
GITHUB_TOKEN: ${{ github.token }}
100+
101+
release:
102+
name: Create Go Module Release
103+
runs-on: ubuntu-latest
104+
needs: test
105+
106+
steps:
107+
- name: Checkout code
108+
uses: actions/checkout@v4
109+
110+
- name: Get Rust library version info
111+
id: rust-version
112+
run: |
113+
LATEST_RUST=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
114+
echo "rust_version=$LATEST_RUST" >> $GITHUB_OUTPUT
115+
env:
116+
GITHUB_TOKEN: ${{ github.token }}
117+
118+
- name: Create Release
119+
uses: softprops/action-gh-release@v2
120+
with:
121+
name: Go Module ${{ github.ref_name }}
122+
body: |
123+
## Go Module Release
124+
125+
This release of the Go tokenizers module is compatible with Rust library version `${{ steps.rust-version.outputs.rust_version }}`.
126+
127+
### Installation
128+
129+
```bash
130+
go get github.com/amikos-tech/pure-tokenizers@${{ github.ref_name }}
131+
```
132+
133+
### Features
134+
- CGo-free implementation using purego
135+
- Automatic library download and caching
136+
- Support for multiple platforms (Linux, macOS, Windows)
137+
- Compatible with HuggingFace tokenizers
138+
139+
### Requirements
140+
- Go 1.24 or later
141+
- Compatible Rust tokenizers library (downloaded automatically)
142+
143+
### Environment Variables
144+
- `TOKENIZERS_LIB_PATH`: Override library path
145+
- `TOKENIZERS_GITHUB_REPO`: Custom GitHub repository
146+
- `TOKENIZERS_VERSION`: Specific Rust library version to use
147+
148+
### Documentation
149+
See the [README](https://github.com/${{ github.repository }}) for detailed usage instructions.
150+
151+
generate_release_notes: true
152+
env:
153+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

0 commit comments

Comments
 (0)