ci: split CUDA tests into separate jobs - #5896
Merged
Merged
Conversation
Run the Python and C++ CUDA suites in independent jobs, with each runner installing the complete CUDA and Python dependency stack. Coding-Agent: Codex Codex-Version: codex-cli 0.144.6 Model: gpt-5.6-sol Reasoning-Effort: xhigh
njzjz
approved these changes
Jul 22, 2026
njzjz
enabled auto-merge
July 22, 2026 16:48
Contributor
📝 WalkthroughWalkthroughCUDA CI testing is split from one combined job into independent Python and C++/CUDA GPU jobs. The aggregation job now depends on both jobs and permits either job to be skipped. ChangesCUDA CI workflow
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Checkov (3.3.8).github/workflows/test_cuda.ymlTraceback (most recent call last): Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
.github/workflows/test_cuda.yml (1)
88-88: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winPin
lukka/get-cmaketo an immutable revision.The new C++ job tracks
@latest, allowing unreviewed action changes to alter the build environment. Pin this action to a reviewed commit SHA or immutable release.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/test_cuda.yml at line 88, Update the lukka/get-cmake action reference in the CUDA workflow to use a reviewed immutable commit SHA or immutable release instead of `@latest`. Keep the action’s existing behavior and configuration unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/test_cuda.yml:
- Around line 17-20: Declare explicit least-privilege workflow permissions at
the top level of the CUDA workflow, using contents: read and no write
permissions. Keep the existing test_python job configuration unchanged.
- Line 109: Remove the --trusted-host options from the paddlepaddle-gpu
installation command in the CUDA workflow, while preserving the HTTPS find-links
URL, PyPI index, and pinned package version.
---
Nitpick comments:
In @.github/workflows/test_cuda.yml:
- Line 88: Update the lukka/get-cmake action reference in the CUDA workflow to
use a reviewed immutable commit SHA or immutable release instead of `@latest`.
Keep the action’s existing behavior and configuration unchanged.
🪄 Autofix (Beta)
❌ Autofix failed (check again to retry)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 009343c5-58eb-430e-8e41-c1ea81001a14
📒 Files selected for processing (1)
.github/workflows/test_cuda.yml
Contributor
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. The agent generated fixes only for
Lines 13–18 group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
cancel-in-progress: true
name: Test CUDA
+permissions:
+ contents: read
jobs:
test_python:
name: Test Python on CUDALines 53–59 - run: |
export PYTORCH_ROOT=$(python -c 'import torch;print(torch.__path__[0])')
export TENSORFLOW_ROOT=$(python -c 'import importlib.util,pathlib;print(pathlib.Path(importlib.util.find_spec("tensorflow").origin).parent)')
- pip install --find-links "https://www.paddlepaddle.org.cn/packages/nightly/cu126/paddlepaddle-gpu/" --index-url https://pypi.org/simple --trusted-host www.paddlepaddle.org.cn --trusted-host paddlepaddle.org.cn "paddlepaddle-gpu==3.4.0.dev20260310"
+ pip install --find-links "https://www.paddlepaddle.org.cn/packages/nightly/cu126/paddlepaddle-gpu/" --index-url https://pypi.org/simple "paddlepaddle-gpu==3.4.0.dev20260310"
source/install/uv_with_retry.sh pip install --system -v -e .[gpu,test,lmp,cu12,torch,jax] mpi4py --reinstall-package deepmd-kit
# See https://github.com/jax-ml/jax/issues/29042
source/install/uv_with_retry.sh pip install --system -U 'nvidia-cublas-cu12>=12.9.0.13'Lines 106–112 - run: |
export PYTORCH_ROOT=$(python -c 'import torch;print(torch.__path__[0])')
export TENSORFLOW_ROOT=$(python -c 'import importlib.util,pathlib;print(pathlib.Path(importlib.util.find_spec("tensorflow").origin).parent)')
- pip install --find-links "https://www.paddlepaddle.org.cn/packages/nightly/cu126/paddlepaddle-gpu/" --index-url https://pypi.org/simple --trusted-host www.paddlepaddle.org.cn --trusted-host paddlepaddle.org.cn "paddlepaddle-gpu==3.4.0.dev20260310"
+ pip install --find-links "https://www.paddlepaddle.org.cn/packages/nightly/cu126/paddlepaddle-gpu/" --index-url https://pypi.org/simple "paddlepaddle-gpu==3.4.0.dev20260310"
source/install/uv_with_retry.sh pip install --system -v -e .[gpu,test,lmp,cu12,torch,jax] mpi4py --reinstall-package deepmd-kit
# See https://github.com/jax-ml/jax/issues/29042
source/install/uv_with_retry.sh pip install --system -U 'nvidia-cublas-cu12>=12.9.0.13' |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5896 +/- ##
==========================================
- Coverage 78.87% 78.61% -0.27%
==========================================
Files 1054 1054
Lines 121770 121770
Branches 4409 4411 +2
==========================================
- Hits 96046 95729 -317
- Misses 24158 24468 +310
- Partials 1566 1573 +7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merged
via the queue into
deepmodeling:master
with commit Jul 22, 2026
5f7b747
62 of 67 checks passed
wanghan-iapcm
pushed a commit
to wanghan-iapcm/deepmd-kit
that referenced
this pull request
Jul 22, 2026
Master's deepmodeling#5896 split the CUDA workflow into separate test_python and test_cc jobs on separate runners, superseding this PR's lane-1 cc-prep tail. Resolution keeps deepmodeling#5896's two-job structure and re-applies this PR's remaining pieces: the aoti_compile/gpu two-lane pytest overlap inside test_python (hides the CPU-bound .pt2 compile tests behind the GPU lane), the per-lane JUnit reports, and the persistent AOTInductor compile cache -- now restored in BOTH jobs, since the test_cc fixture builds (gen_*.py) drive the same inductor backend and reuse the cached kernels.
pull Bot
pushed a commit
to ishandutta2007/deepmd-kit
that referenced
this pull request
Jul 23, 2026
…g#5882) ## What The CUDA python job runs `source/tests` as one serial pytest process. The tests that freeze a `.pt2` (torch.export + AOTInductor) are CPU-bound: inductor / g++ / ptxas generate code for minutes while the GPU sits idle. On the CI GPU the device measured ~98% idle (peak <250 MiB) for the entire duration of that test group, so those minutes are pure serialization in front of the GPU-bound tests. Since deepmodeling#5896 moved the C++ build + ctest + LAMMPS to a separate `test_cc` job, this PR's scope is the python side plus caching: - **Two pytest lanes in `test_python`, concurrently on the one GPU:** lane 1 = the CPU-bound `.pt2`-freezing tests (`-m aoti_compile`), `nice`d and capped to half the cores so the GPU lane keeps CPU priority; lane 2 = the GPU-bound remainder (`-m "not aoti_compile"`). Wall-clock becomes `max(lane 1, lane 2)` instead of their sum. - **Persistent AOTInductor compile cache** (`actions/cache`) restored in BOTH jobs: the `test_cc` fixture builds (`gen_*.py`) drive the same inductor backend on the same architectures, so they reuse kernels the python lanes (or previous runs) already compiled, and unchanged models never recompile across runs. - **Per-lane JUnit reports** uploaded as artifacts for lane-balance monitoring. ## How the partition stays correct The `aoti_compile` marker is auto-applied in `source/tests/conftest.py` to any test whose module references a `.pt2` freeze entry point (`deserialize_to_file` / `_trace_and_export` / `aoti_compile_and_package`). No hand-maintained file list — new compiling tests are partitioned automatically. The two selections are complementary and exhaustive (verified locally: A + B = total, disjoint). A fail-fast guard aborts if an AOT compile happens in an untagged test. The custom-op `.so` install in `test_cc_local.sh` is **atomic** (`copy` to a temp file + `os.replace`) rather than in place: overwriting the destination inode while another process has it `mmap`'d corrupts its live code pages and SIGSEGVs it. `os.replace` swaps the directory entry to a new inode, so live mappings stay intact. ## Measured on CUDA CI Lane wall-clocks from the green run at sha `86128ef7` (measured under the pre-deepmodeling#5896 single-job layout; the pytest lanes are unchanged by the rebase): - Lane 1 (`aoti_compile`): **4063 s (~68 min)** - Lane 2 (`-m "not aoti_compile"`): **8158 s (~2h16m)** Serial python pytest is by construction the sum, 12221 s (~3h24m); the overlap runs them in `max(4063, 8158) =` **8158 s (~2h16m)** — the `test_python` job's pytest step drops by **~68 min (~33%)**, with the entire compile stream hidden behind the GPU lane. The inductor cache additionally removes recompilation of unchanged models from both jobs on subsequent runs. ## Notes / limitations - Overlap quality scales with the runner's core count (`nice` is a no-op unless the CPU saturates); the numbers above are for the CI runner. - Whole-file granularity: a compiling module's few non-compiling tests also run in the compile lane (harmless). - The compile lane tolerates pytest exit 5 ("no tests collected") so the split is safe on branches without `.pt2`-freezing tests. - The two jobs share one cache key per sha; if both try to save it, the second save is skipped with a warning (caches are immutable), which is harmless. - Generic CI orchestration, independent of any feature branch. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests / CI** * Improved CUDA CI by running AOT-related and non-AOT GPU test groups concurrently, with separate JUnit reports and “no tests collected” treated as success. * Persisted the compilation cache across workflow runs to speed up repeated runs. * Added formal AOT compilation test tagging via a new pytest marker and a fail-fast guard if AOT compilation occurs without the expected tag. * **Build & Test Scripts** * Updated local C++ test runner to support skipping build vs ctest phases, improve runtime library setup, handle leak-sanitizer generator behavior, and generate fixtures more reliably (clean stale outputs and run generation in parallel). <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Han Wang <wang_han@iapcm.ac.cn>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
AI attribution
Coding agent: Codex
Codex version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning effort: xhigh
Summary by CodeRabbit