Skip to content

Commit 4314338

Browse files
tazarovtazarov
authored
Merge pull request #25 from amikos-tech/feature/split-releases
[ENH] Split Rust and Go release cycles (#24)
2 parents df78e40 + 871b863 commit 4314338

8 files changed

Lines changed: 730 additions & 46 deletions

File tree

‎.claude/commands/feature.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
Please analyze and implement the GitHub feature: $ARGUMENTS.
2+
3+
Follow these steps:
4+
5+
1. Use `gh issue view` to get the feature issue details
6+
2. Understand the feature described in the issue
7+
3. Create a new branch for the change, ensure the working three is clean
8+
4. Search the codebase for relevant files
9+
5. Implement the necessary changes to address the feature requirements
10+
6. Write and run tests to verify the feature is working as intended
11+
7. Ensure code passes linting and type checking
12+
8. Create a descriptive commit message
13+
9. Push and create a PR
14+
15+
Remember to use the GitHub CLI (`gh`) for all GitHub-related tasks.

‎.claude/commands/ifeed.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
Use the following feedback to do the following:
2+
3+
- Create new issues if the suggestions cause a scope creep for the current PR
4+
- Implement critical issues and those that make sense in the context of the PR so that it is in a complete and usable state
5+
6+
FEEDBACK:
7+
8+
$ARGUMENTS
9+
10+
**IMPORTANT:** Be concise and make sure to lint and test all code changes.

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
1-
name: CI
1+
name: Integration CI
22

3+
# Main integration CI that runs when both Rust and Go code changes
4+
# Individual language CI runs are in rust-ci.yml and go-ci.yml
35
on:
46
push:
57
branches: [ main ]

‎.github/workflows/go-ci.yml‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
name: Go CI
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
paths:
7+
- '**.go'
8+
- 'go.mod'
9+
- 'go.sum'
10+
- '.github/workflows/go-ci.yml'
11+
pull_request:
12+
branches: [ main ]
13+
paths:
14+
- '**.go'
15+
- 'go.mod'
16+
- 'go.sum'
17+
- '.github/workflows/go-ci.yml'
18+
19+
permissions:
20+
contents: read
21+
22+
jobs:
23+
go-test:
24+
name: Go Tests
25+
runs-on: ${{ matrix.os }}
26+
strategy:
27+
matrix:
28+
os: [ubuntu-latest, macos-latest, windows-latest]
29+
go-version: ['1.24']
30+
31+
steps:
32+
- name: Checkout code
33+
uses: actions/checkout@v4
34+
35+
- name: Set up Go
36+
uses: actions/setup-go@v5
37+
with:
38+
go-version: ${{ matrix.go-version }}
39+
40+
- name: Install Rust toolchain (for fallback builds)
41+
uses: dtolnay/rust-toolchain@stable
42+
43+
- name: Run Go linting
44+
uses: golangci/golangci-lint-action@v6
45+
with:
46+
version: latest
47+
args: --timeout=5m
48+
49+
# Download pre-built Rust library for testing
50+
# Falls back to building locally if no rust-v* releases exist yet
51+
- name: Download latest Rust library (Linux)
52+
if: matrix.os == 'ubuntu-latest'
53+
run: |
54+
LATEST_RELEASE=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
55+
if [ -z "$LATEST_RELEASE" ]; then
56+
echo "No Rust release found, building locally"
57+
cargo build --release
58+
echo "TOKENIZERS_LIB_PATH=$(pwd)/target/release/libtokenizers.so" >> $GITHUB_ENV
59+
else
60+
echo "Using Rust release: $LATEST_RELEASE"
61+
gh release download $LATEST_RELEASE --pattern "libtokenizers-x86_64-unknown-linux-gnu.tar.gz"
62+
mkdir -p test-lib
63+
tar -xzf libtokenizers-x86_64-unknown-linux-gnu.tar.gz -C test-lib
64+
echo "TOKENIZERS_LIB_PATH=$(pwd)/test-lib/libtokenizers.so" >> $GITHUB_ENV
65+
fi
66+
env:
67+
GITHUB_TOKEN: ${{ github.token }}
68+
69+
- name: Download latest Rust library (macOS)
70+
if: matrix.os == 'macos-latest'
71+
run: |
72+
ARCH=$(uname -m)
73+
if [ "$ARCH" = "arm64" ]; then
74+
ARCH="aarch64"
75+
elif [ "$ARCH" = "x86_64" ]; then
76+
ARCH="x86_64"
77+
fi
78+
LATEST_RELEASE=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
79+
if [ -z "$LATEST_RELEASE" ]; then
80+
echo "No Rust release found, building locally"
81+
cargo build --release
82+
echo "TOKENIZERS_LIB_PATH=$(pwd)/target/release/libtokenizers.dylib" >> $GITHUB_ENV
83+
else
84+
echo "Using Rust release: $LATEST_RELEASE"
85+
gh release download $LATEST_RELEASE --pattern "libtokenizers-${ARCH}-apple-darwin.tar.gz"
86+
mkdir -p test-lib
87+
tar -xzf libtokenizers-${ARCH}-apple-darwin.tar.gz -C test-lib
88+
echo "TOKENIZERS_LIB_PATH=$(pwd)/test-lib/libtokenizers.dylib" >> $GITHUB_ENV
89+
fi
90+
env:
91+
GITHUB_TOKEN: ${{ github.token }}
92+
93+
- name: Download latest Rust library (Windows)
94+
if: matrix.os == 'windows-latest'
95+
shell: pwsh
96+
run: |
97+
$latestRelease = gh release list --repo ${{ github.repository }} --limit 10 | Select-String -Pattern "^rust-v" | Select-Object -First 1
98+
if ($latestRelease) {
99+
$releaseTag = ($latestRelease -split "`t")[0]
100+
Write-Host "Using Rust release: $releaseTag"
101+
gh release download $releaseTag --pattern "libtokenizers-x86_64-pc-windows-msvc.tar.gz"
102+
New-Item -ItemType Directory -Force -Path test-lib
103+
tar -xzf libtokenizers-x86_64-pc-windows-msvc.tar.gz -C test-lib
104+
"TOKENIZERS_LIB_PATH=$(Get-Location)\test-lib\tokenizers.dll" | Out-File -FilePath $env:GITHUB_ENV -Append
105+
} else {
106+
Write-Host "No Rust release found, building locally"
107+
cargo build --release
108+
"TOKENIZERS_LIB_PATH=$(Get-Location)\target\release\tokenizers.dll" | Out-File -FilePath $env:GITHUB_ENV -Append
109+
}
110+
env:
111+
GITHUB_TOKEN: ${{ github.token }}
112+
113+
- name: Run Go tests (Unix)
114+
if: matrix.os != 'windows-latest'
115+
run: |
116+
go mod tidy
117+
go test -v -coverprofile=coverage.out -timeout=30m ./...
118+
env:
119+
GITHUB_TOKEN: ${{ github.token }}
120+
121+
- name: Run Go tests (Windows)
122+
if: matrix.os == 'windows-latest'
123+
shell: pwsh
124+
run: |
125+
go mod tidy
126+
go test -v -timeout=30m ./...
127+
env:
128+
GITHUB_TOKEN: ${{ github.token }}
129+
130+
integration-test:
131+
name: Integration Tests
132+
runs-on: ubuntu-latest
133+
needs: go-test
134+
135+
steps:
136+
- name: Checkout code
137+
uses: actions/checkout@v4
138+
139+
- name: Set up Go
140+
uses: actions/setup-go@v5
141+
with:
142+
go-version: '1.24'
143+
144+
- name: Install Rust toolchain (for fallback builds)
145+
uses: dtolnay/rust-toolchain@stable
146+
147+
- name: Download latest Rust library
148+
run: |
149+
LATEST_RELEASE=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
150+
if [ -z "$LATEST_RELEASE" ]; then
151+
echo "No Rust release found, building locally"
152+
cargo build --release
153+
echo "TOKENIZERS_LIB_PATH=$(pwd)/target/release/libtokenizers.so" >> $GITHUB_ENV
154+
else
155+
echo "Using Rust release: $LATEST_RELEASE"
156+
gh release download $LATEST_RELEASE --pattern "libtokenizers-x86_64-unknown-linux-gnu.tar.gz"
157+
mkdir -p test-lib
158+
tar -xzf libtokenizers-x86_64-unknown-linux-gnu.tar.gz -C test-lib
159+
echo "TOKENIZERS_LIB_PATH=$(pwd)/test-lib/libtokenizers.so" >> $GITHUB_ENV
160+
fi
161+
env:
162+
GITHUB_TOKEN: ${{ github.token }}
163+
164+
- name: Test download functionality
165+
run: |
166+
go test -v -run "TestDownloadFunctionality|TestGetLibraryInfo"
167+
168+
- name: Test library loading
169+
run: |
170+
go test -v -run "TestGetCachedLibraryPath|TestCacheDirectory"

‎.github/workflows/go-release.yml‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
name: Go Release
2+
3+
# Triggered by v* tags for Go module releases
4+
on:
5+
push:
6+
tags:
7+
- 'v*'
8+
- '!v*-*' # Exclude pre-releases like v1.0.0-beta
9+
10+
permissions:
11+
contents: write
12+
13+
jobs:
14+
test:
15+
name: Test Go Module
16+
runs-on: ${{ matrix.os }}
17+
strategy:
18+
matrix:
19+
os: [ubuntu-latest, macos-latest, windows-latest]
20+
go-version: ['1.24']
21+
22+
steps:
23+
- name: Checkout code
24+
uses: actions/checkout@v4
25+
26+
- name: Set up Go
27+
uses: actions/setup-go@v5
28+
with:
29+
go-version: ${{ matrix.go-version }}
30+
31+
- name: Install Rust toolchain (for fallback builds)
32+
uses: dtolnay/rust-toolchain@stable
33+
34+
- name: Run Go linting
35+
uses: golangci/golangci-lint-action@v6
36+
with:
37+
version: latest
38+
args: --timeout=5m
39+
40+
# Download the latest Rust library release for testing
41+
- name: Get latest Rust release (Linux)
42+
if: matrix.os == 'ubuntu-latest'
43+
run: |
44+
LATEST_RUST=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
45+
if [ -z "$LATEST_RUST" ]; then
46+
echo "ERROR: No Rust release found. Please create a Rust release first."
47+
exit 1
48+
fi
49+
echo "Using Rust release: $LATEST_RUST"
50+
gh release download $LATEST_RUST --pattern "libtokenizers-x86_64-unknown-linux-gnu.tar.gz"
51+
mkdir -p test-lib
52+
tar -xzf libtokenizers-x86_64-unknown-linux-gnu.tar.gz -C test-lib
53+
echo "TOKENIZERS_LIB_PATH=$(pwd)/test-lib/libtokenizers.so" >> $GITHUB_ENV
54+
env:
55+
GITHUB_TOKEN: ${{ github.token }}
56+
57+
- name: Get latest Rust release (macOS)
58+
if: matrix.os == 'macos-latest'
59+
run: |
60+
ARCH=$(uname -m)
61+
if [ "$ARCH" = "arm64" ]; then
62+
ARCH="aarch64"
63+
elif [ "$ARCH" = "x86_64" ]; then
64+
ARCH="x86_64"
65+
fi
66+
LATEST_RUST=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
67+
if [ -z "$LATEST_RUST" ]; then
68+
echo "ERROR: No Rust release found. Please create a Rust release first."
69+
exit 1
70+
fi
71+
echo "Using Rust release: $LATEST_RUST"
72+
gh release download $LATEST_RUST --pattern "libtokenizers-${ARCH}-apple-darwin.tar.gz"
73+
mkdir -p test-lib
74+
tar -xzf libtokenizers-${ARCH}-apple-darwin.tar.gz -C test-lib
75+
echo "TOKENIZERS_LIB_PATH=$(pwd)/test-lib/libtokenizers.dylib" >> $GITHUB_ENV
76+
env:
77+
GITHUB_TOKEN: ${{ github.token }}
78+
79+
- name: Get latest Rust release (Windows)
80+
if: matrix.os == 'windows-latest'
81+
shell: pwsh
82+
run: |
83+
$latestRust = gh release list --repo ${{ github.repository }} --limit 10 | Select-String -Pattern "^rust-v" | Select-Object -First 1
84+
if (-not $latestRust) {
85+
Write-Error "No Rust release found. Please create a Rust release first."
86+
exit 1
87+
}
88+
$releaseTag = ($latestRust -split "`t")[0]
89+
Write-Host "Using Rust release: $releaseTag"
90+
gh release download $releaseTag --pattern "libtokenizers-x86_64-pc-windows-msvc.tar.gz"
91+
New-Item -ItemType Directory -Force -Path test-lib
92+
tar -xzf libtokenizers-x86_64-pc-windows-msvc.tar.gz -C test-lib
93+
"TOKENIZERS_LIB_PATH=$(Get-Location)\test-lib\tokenizers.dll" | Out-File -FilePath $env:GITHUB_ENV -Append
94+
env:
95+
GITHUB_TOKEN: ${{ github.token }}
96+
97+
- name: Run Go tests (Unix)
98+
if: matrix.os != 'windows-latest'
99+
run: |
100+
go mod tidy
101+
go test -v -race -coverprofile=coverage.out -timeout=30m ./...
102+
env:
103+
GITHUB_TOKEN: ${{ github.token }}
104+
105+
- name: Run Go tests (Windows)
106+
if: matrix.os == 'windows-latest'
107+
shell: pwsh
108+
run: |
109+
go mod tidy
110+
go test -v -race -timeout=30m ./...
111+
env:
112+
GITHUB_TOKEN: ${{ github.token }}
113+
114+
release:
115+
name: Create Go Module Release
116+
runs-on: ubuntu-latest
117+
needs: test
118+
119+
steps:
120+
- name: Checkout code
121+
uses: actions/checkout@v4
122+
123+
- name: Get Rust library version info
124+
id: rust-version
125+
run: |
126+
LATEST_RUST=$(gh release list --repo ${{ github.repository }} --limit 10 | grep -E "^rust-v" | head -1 | cut -f1)
127+
echo "rust_version=$LATEST_RUST" >> $GITHUB_OUTPUT
128+
env:
129+
GITHUB_TOKEN: ${{ github.token }}
130+
131+
- name: Create Release
132+
uses: softprops/action-gh-release@v2
133+
with:
134+
name: Go Module ${{ github.ref_name }}
135+
body: |
136+
## Go Module Release
137+
138+
This release of the Go tokenizers module is compatible with Rust library version `${{ steps.rust-version.outputs.rust_version }}`.
139+
140+
### Installation
141+
142+
```bash
143+
go get github.com/amikos-tech/pure-tokenizers@${{ github.ref_name }}
144+
```
145+
146+
### Features
147+
- CGo-free implementation using purego
148+
- Automatic library download and caching
149+
- Support for multiple platforms (Linux, macOS, Windows)
150+
- Compatible with HuggingFace tokenizers
151+
152+
### Requirements
153+
- Go 1.24 or later
154+
- Compatible Rust tokenizers library (downloaded automatically)
155+
156+
### Environment Variables
157+
- `TOKENIZERS_LIB_PATH`: Override library path
158+
- `TOKENIZERS_GITHUB_REPO`: Custom GitHub repository
159+
- `TOKENIZERS_VERSION`: Specific Rust library version to use
160+
161+
### Documentation
162+
See the [README](https://github.com/${{ github.repository }}) for detailed usage instructions.
163+
164+
generate_release_notes: true
165+
env:
166+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

0 commit comments

Comments
 (0)