Commit df5567a
authored
ci(cuda): build only for CI runner GPU arch and raise parallelism (deepmodeling#7753)
* ci(cuda): build only for CI runner GPU arch and raise parallelism
The CUDA CI built every .cu file for 7 GPU architectures
(60/70/75/80/86/89/90) with a hardcoded -j4, so the Configure & Build
step took ~33 min even with a warm ccache.
- Pin CMAKE_CUDA_ARCHITECTURES=70: the CI GPU pool is Tesla V100
(sm_70, per nvidia-smi in the run logs and the '16V100' Slurm
partition in .ci/slurm/config.ini). This cuts nvcc work by ~7x.
- Build with -j $(nproc) instead of -j4; with the arch list reduced,
the higher parallelism is memory-safe.
Expected: Configure & Build ~33 min -> ~10 min on a cache-cold run.
* ci: auto-detect GPU arch via nvidia-smi at CMake configure time
Sister commit to e614a28. The previous commit hardcoded
-DCMAKE_CUDA_ARCHITECTURES=70 assuming the CI pool is Tesla V100.
Reviewers (Stardust0831 + chenmohan) pointed out this couples the
workflow to a specific GPU model and breaks if the runner pool is
heterogeneous or upgraded.
This commit moves the arch selection into CMakeLists.txt:
- When CMAKE_CUDA_ARCHITECTURES is unset and nvidia-smi is available,
query --query-gpu=compute_cap and set the arch from the result.
Map '7.0' -> 70, '8.0' -> 80, '8.9' -> 89, '9.0' -> 90, etc.
- Falls back to the historical multi-arch default if nvidia-smi is not
present (CPU-only build host) or returns an unrecognized value.
- User-provided CMAKE_CUDA_ARCHITECTURES still takes precedence.
Workflow file: removed the -DCMAKE_CUDA_ARCHITECTURES=70 line; the
Configure step now relies on the CMake-side detection. Verified the
plain -DCMAKE_CUDA_ARCHITECTURES=70 still produces a 38 min cold Build
on the existing V100 runner.
* ci(cuda): revert broken auto-detect; use explicit sm_70 pin
The previous commit (167904f) tried to make the workflow adapt to
whatever GPU the runner has by auto-detecting compute capability at
CMake configure time. Reviewers (Stardust0831 + 张笑扬) correctly
flagged two independent defects and a deeper design issue.
Defects in the previous commit:
1. if(COMMAND nvidia-smi) tests for a CMake command, not an
executable on PATH. It was always false, so execute_process never
ran. The correct check is find_program(NVIDIA_SMI_EXECUTABLE nvidia-smi).
2. The detection block and the historical default list were both
inside the same if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES) outer
guard. The historical default uses plain set() which shadows the
cache entry; all 7 archs were appended regardless of detection.
3. AND USE_CUDA inside the detection block was redundant; the
surrounding if(USE_CUDA) at line 419 already guards it.
Design issue:
Auto-detection is the wrong default for a cluster codebase. ABACUS is
configured on whichever host runs cmake (often a login node on an
HPC system) but the resulting binary may run on a different compute
node. Silent auto-detection produces a binary that fails at run time
with 'no kernel image is available for execution on the device' and
no warning at configure time. HPC convention is to build on the
compute node, where the workflow's hardcoded -D matches the hardware.
Furthermore, heterogeneous auto-detection fragments the ccache key per
node, which defeats cache_warmer (deepmodeling#7756). An explicit uniform pin
across CI is exactly what cache_warmer relies on.
This commit:
- Reverts the CMakeLists.txt auto-detect block.
- Restores -DCMAKE_CUDA_ARCHITECTURES=70 in cuda.yml, with a comment
noting that the value assumes a homogeneous V100 pool and should be
updated if the pool changes.
- Keeps -j $(nproc), which is the real win in e614a28 (~2x parallelism).
Note on the measured 33 min -> 18 min result: the reviewer is correct
that the savings probably come almost entirely from -j4 -> -j $(nproc),
not from the arch cut (7 -> 1 arch). With nvcc being fast and C++ TUs
dominating build time, doubling the build parallelism is enough to
halve the wall time; the arch reduction's contribution, if any, is
small. The arch pin still avoids fatbin bloat and uniform ccache keys,
but the headline number in the PR body should not over-claim it.
* Clean up comments in CUDA workflow
Removed comments explaining the CUDA architecture pinning process.1 parent f36f201 commit df5567a
1 file changed
Lines changed: 3 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
50 | | - | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
51 | 52 | | |
52 | 53 | | |
53 | 54 | | |
| |||
0 commit comments