-
Notifications
You must be signed in to change notification settings - Fork 1
Use GitHub Actions for package releases #67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
katemartian
wants to merge
89
commits into
main
Choose a base branch
from
useGitHubActions
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 70 commits
Commits
Show all changes
89 commits
Select commit
Hold shift + click to select a range
40fae75
Edit gitignore
katemartian 7b25e94
Add workflow for deeplabcut
katemartian a2b34c6
Add yml draft for caiman releases
katemartian 0cea2ba
Edit caiman yml
katemartian 015a709
Try to use miniconda to install caiman
katemartian e9efcc3
Remove miniconda installation -- It was installed on the service
katemartian c00514c
Update caiman installation
katemartian 957de35
Change hard-coded path to the caiman_data
katemartian 25ed141
Add missing imports in caiman spec
katemartian 4b7c4e8
Add caiman_data to the environment
katemartian 88f919f
Edit spec files: add missing imports, specify build and dist paths
katemartian 316793b
Draft release yml for MiniAn
katemartian 8ba3a22
Add draft of suite2p release yml
katemartian 71896d7
Remove torch installation from suite2p release
katemartian d6d3d0d
Edit minian release: try to use venv
katemartian 8792f41
Revert to use conda for minian installation
katemartian fd3f931
Set python version to 3.9 for minian release
katemartian 4ceec5c
Change python to 3.8 for minian installation
katemartian f2cfc60
Edit caiman release file
katemartian b8916e0
Fix caiman data dir
katemartian b9dde9a
Change place for caiman_data
katemartian bd2d973
Resolve caiman_data location
katemartian 590e39c
Try to resolve caiman_data location again
katemartian 0f4d854
Merge branch 'useGitHubActions' of https://github.com/doriclenses/pyD…
katemartian f624697
Change caiman data location in spec file
katemartian f2cfc49
Add caiman_data to collect
katemartian 36506b4
Update location of the zip file for minian
katemartian e092582
Fix caiman zip in the workflow
katemartian 6e6f86a
Define missing caiman data dir in spec file
katemartian b29b8c2
Merge branch 'main' into useGitHubActions
katemartian f9b7504
Add constraints for deeplabcut installation. Fix git ignore
katemartian 9c92e1f
Fix error llvmpy for suite2p
katemartian 341c360
Add collect all for h5py in minian
katemartian 42840e1
Make sure to collect all dlls for h5py library in minian release
katemartian 3d99e02
Print dlls that were taken when building minian
katemartian 03ad35e
Include all dlls for h5py in minian release
katemartian 853b4b2
Collect all data from pyarrow
katemartian ecc67c9
Collect data from skipped libraries
katemartian d89eb2c
Add torch, torchvision and pyarrow
katemartian ee3ad66
Remove unnecessary import in suite2p pack
katemartian b19f1a3
Add hook for torch
katemartian b3f0910
Add early fail if torch is installed incorrectly
katemartian 1cabbe0
Use 2.8 torch version
katemartian 8a660f9
Rearrange package collection for suite2p
katemartian e66091d
Edit package collection in minian
katemartian a9d9423
Edit package collection for caiman
katemartian 2382a9f
Minor edits in spec files
katemartian 426f362
Remove heavy libraries from the hook that are already collected by th…
katemartian 70ac715
Space edits in spec files
katemartian 8fe1456
Rename spec files
katemartian b0d5319
add sss for test
d350fc7
remove test commit
a846de2
Remove extra hooks for h5py for minian
katemartian 95ef4d9
Try to collect more binaries for minian
katemartian ee0dc4b
Capture the environment during github release for minian
katemartian d0dce71
Try to reproduce the same environment as local for minian release
katemartian 7b8ff4a
Try to use evironment yml from minian github
katemartian ab3d5c5
Add missing minian installation
katemartian 1a00dcf
Collect all hdf5 dlls for minian release
katemartian 67513de
Install SimpleITK with pip to avoid errors
katemartian 61870d1
Update conda tooling and clean cache
katemartian 3dc09a9
Pin versions of minian, h5py and pyinstaller
katemartian 9d6b812
Place minian, h5py, and pyinstaller as the last libraries to install
katemartian 32f7185
Fix parameters that are passed to pyinstaller Analysis in minian
katemartian 32d41bc
Hide binary collection that wasn't here before. Change pyinstaller ve…
katemartian ed5ebdd
Missed file
katemartian d669d4a
Merge branch 'main' into useGitHubActions
katemartian caed858
Return back the collection of the hdf dlls
katemartian a341a40
Freeze release of h5py and pyinstaller to the ones before last minian…
katemartian deefa87
Uncomment dynamic lib collection
katemartian dec0d42
Set upx to false in minian
katemartian ba90745
Set release of h5py and pyinstaller to the ones used locally for minian
katemartian ea1d7ea
Pin versions of h5py and pyinstaller suggested by Codex for minian re…
katemartian 23ee317
Set all versions to the local ones for minian release
katemartian dbce723
Fix typo in the yml file
katemartian ea251d6
Add missing dlls for minian release
katemartian 20c20d5
Fix version of torch for cuda 12.8 in deeplabcut release
katemartian 073ce45
Freeze version of suite2p
katemartian 01f457e
Minor edits in the caiman and suite2p spec files
katemartian ad6f002
Test deeplabcut release including all the optional packages
katemartian d614de8
Mock matplotlib.patches.Ellipse
katemartian 4b2f406
missing coma
5f69baf
remove commented code in spec deeplabcut
60b1edc
remove commented import
146ff88
Remove unnecessary comment
katemartian eb3a362
Update pins for suite2p
katemartian 02e9d07
Remove redundnt spaces
katemartian f5bb2a5
Copy release zips to the shared directory on the workstation
katemartian 593f590
Collect all the tbb dlls for suite2p release
katemartian File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,147 @@ | ||
| name: Release CaImAn | ||
|
|
||
| on: | ||
| push: | ||
| tags: ['caiman-v*'] | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: write | ||
|
|
||
| concurrency: | ||
| group: cmn-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: [self-hosted, Windows, X64, GPU] | ||
| timeout-minutes: 180 | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Extract version from tag | ||
| shell: cmd | ||
| run: | | ||
| set VERSION=%GITHUB_REF_NAME:caiman-v=% | ||
| echo VERSION=%VERSION%>> %GITHUB_ENV% | ||
|
|
||
| - name: Verify Miniconda | ||
| shell: cmd | ||
| run: C:\Miniconda3\Scripts\conda.exe --version | ||
|
|
||
| - name: Create conda env for caiman | ||
| shell: cmd | ||
| run: | | ||
| C:\Miniconda3\Scripts\conda.exe env remove -n caiman_release -y || echo "no old env" | ||
| C:\Miniconda3\Scripts\conda.exe create -n caiman_release python=3.10 -y | ||
|
|
||
| - name: Install caiman deps with conda-forge | ||
| shell: cmd | ||
| run: | | ||
| C:\Miniconda3\Scripts\conda.exe install -n caiman_release -c conda-forge pyinstaller caiman -y | ||
| C:\Miniconda3\Scripts\conda.exe run -n caiman_release python -c "import caiman; print('Caiman version:', getattr(caiman,'__version__','?'))" | ||
|
|
||
| - name: caimanmanager install (detect path and export) | ||
| shell: cmd | ||
| run: | | ||
| C:\Miniconda3\Scripts\conda.exe run -n caiman_release caimanmanager install --force | ||
| for /f "delims=" %%H in ('C:\Miniconda3\Scripts\conda.exe run -n caiman_release python -c "import pathlib; print(pathlib.Path.home())"') do set "CM_HOME=%%H" | ||
| if exist "%CM_HOME%\caiman_data" ( | ||
| echo CAIMAN_DATA_DIR=%CM_HOME%\caiman_data>> %GITHUB_ENV% | ||
| ) else if exist "%CM_HOME%\.caimanmanager\caiman_data" ( | ||
| echo CAIMAN_DATA_DIR=%CM_HOME%\.caimanmanager\caiman_data>> %GITHUB_ENV% | ||
| ) else ( | ||
| echo WARNING: caiman_data not found after install | ||
| ) | ||
|
|
||
| - name: Preflight paths | ||
| shell: cmd | ||
| run: | | ||
| cd | ||
| dir | ||
| dir Deploy | ||
| dir Deploy\pack_caiman.spec | ||
|
|
||
| - name: Clean old dist/build | ||
| shell: cmd | ||
| working-directory: Deploy | ||
| run: | | ||
| if exist dist rmdir /s /q dist | ||
| if exist build rmdir /s /q build | ||
|
|
||
| - name: Build caiman exe | ||
| shell: cmd | ||
| working-directory: Deploy | ||
| run: | | ||
| C:\Miniconda3\Scripts\conda.exe run -n caiman_release pyinstaller --clean --noconfirm pack_caiman_run.spec | ||
| echo === DIST TREE === | ||
| dir /s /b dist | ||
|
|
||
| - name: Zip onedir output with build-info | ||
| shell: cmd | ||
| working-directory: Deploy | ||
| env: | ||
| GIT_SHA: ${{ github.sha }} | ||
| run: | | ||
| rem Locate the onedir app folder under dist | ||
| set "APPDIR=" | ||
| for /d %%D in (dist\*) do ( set "APPDIR=%%D" & goto :found ) | ||
| :found | ||
| if not defined APPDIR ( echo ERROR: no onedir folder found in Deploy\dist & exit /b 1 ) | ||
|
|
||
| rem Write build-info into the app folder (included in the zip) | ||
| echo Version: %VERSION%> "%APPDIR%\build-info.txt" | ||
| echo Commit: %GIT_SHA%>> "%APPDIR%\build-info.txt" | ||
| echo Date: %DATE% %TIME%>> "%APPDIR%\build-info.txt" | ||
|
|
||
| set "ZIPNAME=caiman-%VERSION%.zip" | ||
|
|
||
| rem Use conda env's python (single-line, no backslashes) | ||
| C:\Miniconda3\Scripts\conda.exe run -n caiman_release python -c "import os,zipfile; a=r'%APPDIR%'; zf=r'%ZIPNAME%'; assert os.path.isdir(a), 'no APPDIR: '+a; z=zipfile.ZipFile(zf,'w',zipfile.ZIP_DEFLATED); [z.write(os.path.join(r,f), os.path.relpath(os.path.join(r,f),a)) for r,_,fs in os.walk(a) for f in fs]; z.close()" | ||
| echo Created zip: %ZIPNAME% | ||
|
|
||
| - name: Show zip contents | ||
| shell: cmd | ||
| working-directory: Deploy | ||
| run: | | ||
| set "ZN=caiman-%VERSION%.zip" | ||
| if not exist "%ZN%" ( echo ERROR: zip not found & exit /b 1 ) | ||
| C:\Miniconda3\Scripts\conda.exe run -n caiman_release python -c "import os,zipfile; zn=r'%ZN%'; z=zipfile.ZipFile(zn,'r'); n=z.namelist(); print('Entries in',zn,':',len(n)); [print(' ',x) for x in n[:50]]; z.close()" | ||
|
|
||
| - name: SHA256 of release zip | ||
| shell: cmd | ||
| working-directory: Deploy | ||
| run: | | ||
| setlocal EnableDelayedExpansion | ||
| set ZIP=caiman-%VERSION%.zip | ||
| if not exist "%ZIP%" ( echo ERROR: zip not found & exit /b 1 ) | ||
| for /f "usebackq skip=1 tokens=1" %%H in (`certutil -hashfile "%ZIP%" SHA256 ^| findstr /v /i "hash certutil"`) do ( | ||
| set HASH=%%H | ||
| goto :got | ||
| ) | ||
| :got | ||
| echo !HASH!> caiman-%VERSION%.zip.sha256 | ||
| type caiman-%VERSION%.zip.sha256 | ||
|
|
||
| - name: Create GitHub Release (attach zip + checksum) | ||
| uses: softprops/action-gh-release@v2 | ||
| with: | ||
| name: caiman ${{ env.VERSION }} | ||
| body: | | ||
| Automated caiman release for tag ${{ github.ref_name }}. | ||
| - Version: ${{ env.VERSION }} | ||
| - Commit: ${{ github.sha }} | ||
| files: | | ||
| Deploy/caiman-${{ env.VERSION }}.zip | ||
| Deploy/caiman-${{ env.VERSION }}.zip.sha256 | ||
|
|
||
| - name: Upload dist artifact (only on failure) | ||
| if: ${{ failure() }} | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: caiman-dist-${{ env.VERSION }} | ||
| path: Deploy/dist/** | ||
| retention-days: 3 | ||
| compression-level: 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,129 @@ | ||
| name: Release DeepLabCut | ||
|
|
||
| on: | ||
| push: | ||
| tags: ['deeplabcut-v*'] | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: write | ||
|
|
||
| concurrency: | ||
| group: dlc-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: [self-hosted, Windows, X64, GPU] | ||
| timeout-minutes: 180 | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Extract version from tag (cmd) | ||
| shell: cmd | ||
| run: | | ||
| set VERSION=%GITHUB_REF_NAME:deeplabcut-v=% | ||
| echo VERSION=%VERSION%>> %GITHUB_ENV% | ||
|
|
||
| - name: Create venv | ||
| shell: cmd | ||
| run: | | ||
| C:\Python310\python.exe -V | ||
| C:\Python310\python.exe -m venv venv | ||
|
|
||
| - name: Install deps | ||
| shell: cmd | ||
| run: | | ||
| venv\Scripts\python -m pip install --upgrade pip | ||
| venv\Scripts\pip install --constraint Deploy\constraints\pins.txt h5py pyinstaller deeplabcut==3.0.0rc8 | ||
| venv\Scripts\pip install --constraint Deploy\constraints\pins.txt --upgrade --index-url https://download.pytorch.org/whl/cu126 torch==2.8.0+cu126 torchvision==0.23.0+cu126 | ||
| venv\Scripts\python -c "import torch;print('Torch:',torch.__version__);print('CUDA build:',getattr(torch.version,'cuda',None));print('CUDA available:',torch.cuda.is_available())" | ||
|
|
||
| - name: Clean old dist/build | ||
| shell: cmd | ||
| working-directory: Deploy | ||
| run: | | ||
| if exist dist rmdir /s /q dist | ||
| if exist build rmdir /s /q build | ||
|
|
||
| - name: Build deeplabcut | ||
| shell: cmd | ||
| working-directory: Deploy | ||
| run: | | ||
| ..\venv\Scripts\pyinstaller --clean --noconfirm pack_deeplabcut.spec | ||
| echo === DIST TREE === | ||
| dir /s /b dist | ||
|
|
||
| - name: Smoke test torch/deeplabcut import | ||
| shell: cmd | ||
| run: | | ||
| venv\Scripts\python -c "import torch; import deeplabcut; print('torch', torch.__version__)" | ||
|
|
||
| - name: Zip onedir output with build-info | ||
| shell: cmd | ||
| working-directory: Deploy | ||
| env: | ||
| GIT_SHA: ${{ github.sha }} | ||
| run: | | ||
| rem Locate the onedir app folder under dist | ||
| set "APPDIR=" | ||
| for /d %%D in (dist\*) do ( set "APPDIR=%%D" & goto :found ) | ||
| :found | ||
| if not defined APPDIR ( echo ERROR: no onedir folder found in Deploy\dist & exit /b 1 ) | ||
|
|
||
| rem Write build-info into the app folder (included in the zip) | ||
| echo Version: %VERSION%> "%APPDIR%\build-info.txt" | ||
| echo Commit: %GIT_SHA%>> "%APPDIR%\build-info.txt" | ||
| echo Date: %DATE% %TIME%>> "%APPDIR%\build-info.txt" | ||
|
|
||
| set "ZIPNAME=deeplabcut-%VERSION%.zip" | ||
|
|
||
| rem One-line Python (no backslashes/newlines) | ||
| ..\venv\Scripts\python -c "import os,zipfile; a=r'%APPDIR%'; zf=r'%ZIPNAME%'; assert os.path.isdir(a), 'no APPDIR: '+a; z=zipfile.ZipFile(zf,'w',zipfile.ZIP_DEFLATED); [z.write(os.path.join(r,f), os.path.relpath(os.path.join(r,f),a)) for r,_,fs in os.walk(a) for f in fs]; z.close()" | ||
| echo Created zip: %ZIPNAME% | ||
|
|
||
| - name: Show zip contents | ||
| shell: cmd | ||
| working-directory: Deploy | ||
| run: | | ||
| set "ZN=deeplabcut-%VERSION%.zip" | ||
| if not exist "%ZN%" ( echo ERROR: zip not found & exit /b 1 ) | ||
| ..\venv\Scripts\python -c "import os,zipfile; zn=r'%ZN%'; z=zipfile.ZipFile(zn,'r'); n=z.namelist(); print('Entries in',zn,':',len(n)); [print(' ',x) for x in n[:50]]; z.close()" | ||
|
|
||
|
|
||
| - name: SHA256 of release zip | ||
| shell: cmd | ||
| working-directory: Deploy | ||
| run: | | ||
| setlocal EnableDelayedExpansion | ||
| set ZIP=deeplabcut-%VERSION%.zip | ||
| for /f "usebackq skip=1 tokens=1" %%H in (`certutil -hashfile "%ZIP%" SHA256 ^| findstr /v /i "hash certutil"`) do ( | ||
| set HASH=%%H | ||
| goto :got | ||
| ) | ||
| :got | ||
| echo !HASH!> deeplabcut-%VERSION%.zip.sha256 | ||
| type deeplabcut-%VERSION%.zip.sha256 | ||
|
|
||
| - name: Create GitHub Release (attach zip + checksum) | ||
| uses: softprops/action-gh-release@v2 | ||
| with: | ||
| name: deeplabcut ${{ env.VERSION }} | ||
| body: | | ||
| Automated deeplabcut release for tag ${{ github.ref_name }}. | ||
| - Version: ${{ env.VERSION }} | ||
| - Commit: ${{ github.sha }} | ||
| files: | | ||
| Deploy/deeplabcut-${{ env.VERSION }}.zip | ||
| Deploy/deeplabcut-${{ env.VERSION }}.zip.sha256 | ||
|
|
||
| - name: Upload dist artifact (only on failure) | ||
| if: ${{ failure() }} | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: deeplabcut-dist-${{ env.VERSION }} | ||
| path: Deploy/dist/** | ||
| retention-days: 3 | ||
| compression-level: 1 | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This need to be changed to the 12.8 cuda version. Because of new graphic card.
venv\Scripts\pip install --constraint Deploy\constraints\pins.txt --upgrade --index-url https://download.pytorch.org/whl/cu128 torch==2.8.0+cu128 torchvision==0.23.0+cu128