Skip to content

Commit 70b5175

Browse files
author
Han Wang
committed
ci(cuda): overlap C++ build + fixture gen with the Python GPU lane
The C++ portion ran serially after the Python tests: cmake --build plus the gen_*.py .pt2 fixtures are ~1h of CPU-bound work with the GPU idle (measured: 56m gen + 7.5m build + 5m convert-models), followed by ~39m of GPU tests (ctest + LAMMPS). The Python GPU lane meanwhile has a ~1h23m idle-CPU tail (compile lane ends ~1h13m, GPU lane runs to ~2h36m). Run the C++ CPU-prep as a background step (GitHub Actions parallel steps) so it overlaps the Python GPU lane, with a `- wait: cc-prep` barrier before the ctest step consumes the fixtures. The prep is niced -19 so both Python lanes keep CPU priority and it only fills idle cores. test_cc_local.sh gains DP_CC_SKIP_CTEST / DP_CC_SKIP_BUILD phase gates (default runs both, so test_cc.yml is unaffected); convert-models.sh moves into the prep step. Projected: hides ~1h14m of GPU-idle compile, whole CUDA job ~4h57m -> ~3h43m.
1 parent 6f8d76b commit 70b5175

2 files changed

Lines changed: 126 additions & 83 deletions

File tree

‎.github/workflows/test_cuda.yml‎

Lines changed: 29 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,27 @@ jobs:
8181
DP_ENABLE_NATIVE_OPTIMIZATION: 1
8282
DP_ENABLE_PYTORCH: 1
8383
- run: dp --version
84+
- name: Build C++ and generate fixtures (overlaps the Python GPU lane)
85+
# The C++ build + gen_*.py .pt2 fixtures are CPU-bound with the GPU idle
86+
# (~1h of the job) and independent of the Python tests. Run them as a
87+
# background step so they overlap the Python GPU lane's idle-CPU tail;
88+
# the ctest step waits on this (- wait: cc-prep) before consuming the
89+
# fixtures. Niced hard (19) so both Python lanes keep CPU priority --
90+
# this only fills otherwise-idle cores. DP_CC_SKIP_CTEST builds + gens
91+
# only; ctest runs later once this joins.
92+
id: cc-prep
93+
background: true
94+
run: |
95+
export LD_LIBRARY_PATH=$CUDA_PATH/lib64:/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH
96+
source/tests/infer/convert-models.sh
97+
DP_CC_SKIP_CTEST=1 nice -n 19 source/install/test_cc_local.sh
98+
env:
99+
OMP_NUM_THREADS: 1
100+
TF_INTRA_OP_PARALLELISM_THREADS: 1
101+
TF_INTER_OP_PARALLELISM_THREADS: 1
102+
CMAKE_GENERATOR: Ninja
103+
DP_VARIANT: cuda
104+
DP_USE_MPICH2: 1
84105
- name: Run Python tests (overlap AOTI compiles with GPU unit tests)
85106
# The tests that freeze a .pt2 (-m aoti_compile) are CPU-bound: the GPU
86107
# sits ~idle (measured ~98% idle, <250 MiB) while inductor/g++/ptxas
@@ -135,11 +156,15 @@ jobs:
135156
junit-aoti.xml
136157
junit-gpu.xml
137158
if-no-files-found: warn
138-
- name: Convert models
139-
run: source/tests/infer/convert-models.sh
140-
- run: |
159+
# Barrier: the background C++ build + fixture generation must be done
160+
# before ctest consumes the fixtures. By now the Python step has run for
161+
# ~2.5h, so the ~1h of C++ prep has almost always already finished during
162+
# it -- this wait is usually instant.
163+
- wait: cc-prep
164+
- name: C++ tests (ctest; build + fixtures done in the background step)
165+
run: |
141166
export LD_LIBRARY_PATH=$CUDA_PATH/lib64:/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH
142-
source/install/test_cc_local.sh
167+
DP_CC_SKIP_BUILD=1 source/install/test_cc_local.sh
143168
env:
144169
OMP_NUM_THREADS: 1
145170
TF_INTRA_OP_PARALLELISM_THREADS: 1

‎source/install/test_cc_local.sh‎

Lines changed: 97 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,14 @@
11
#!/bin/bash
22
set -ex
33

4+
# Phase gating (default: run everything, so existing callers like test_cc.yml
5+
# are unaffected). The CUDA workflow splits the CPU-bound build+fixture-gen
6+
# from the GPU-bound ctest so the former can overlap the Python GPU tests:
7+
# DP_CC_SKIP_CTEST=1 -> configure + build + install + gen fixtures, no ctest
8+
# DP_CC_SKIP_BUILD=1 -> skip build/gen, run only ctest on the built tree
9+
DP_CC_SKIP_BUILD=${DP_CC_SKIP_BUILD:-0}
10+
DP_CC_SKIP_CTEST=${DP_CC_SKIP_CTEST:-0}
11+
412
if [ "$DP_VARIANT" = "cuda" ]; then
513
CUDA_ARGS="-DUSE_CUDA_TOOLKIT=TRUE"
614
elif [ "$DP_VARIANT" = "rocm" ]; then
@@ -20,27 +28,37 @@ BUILD_TMP_DIR=${SCRIPT_PATH}/../build_tests
2028
PADDLE_INFERENCE_DIR=${BUILD_TMP_DIR}/paddle_inference_install_dir
2129
mkdir -p ${BUILD_TMP_DIR}
2230
cd ${BUILD_TMP_DIR}
23-
cmake \
24-
-D ENABLE_TENSORFLOW=${ENABLE_TENSORFLOW:-TRUE} \
25-
-D ENABLE_PYTORCH=${ENABLE_PYTORCH:-TRUE} \
26-
-D ENABLE_PADDLE=${ENABLE_PADDLE:-TRUE} \
27-
-D INSTALL_TENSORFLOW=FALSE \
28-
-D USE_TF_PYTHON_LIBS=${ENABLE_TENSORFLOW:-TRUE} \
29-
-D USE_PT_PYTHON_LIBS=${ENABLE_PYTORCH:-TRUE} \
30-
-D CMAKE_INSTALL_PREFIX=${INSTALL_PREFIX} \
31-
-D BUILD_TESTING:BOOL=TRUE \
32-
-D LAMMPS_VERSION=stable_22Jul2025_update2 \
33-
${CUDA_ARGS} ..
34-
cmake --build . -j${NPROC}
35-
cmake --install .
36-
# Generate PT/PT2 model files for C++ tests.
37-
# Must run after cmake --build so that libdeepmd_op_pt.so (custom ops) is available.
38-
if [ "${ENABLE_PYTORCH:-TRUE}" == "TRUE" ]; then
39-
# Install the custom op .so to SHARED_LIB_DIR so that `import deepmd.pt`
40-
# loads it via cxx_op.py. The .so depends on libdeepmd.so (compute
41-
# kernels) in the install prefix, so add that to LD_LIBRARY_PATH too.
42-
export LD_LIBRARY_PATH=${INSTALL_PREFIX}/lib:${LD_LIBRARY_PATH}
43-
python -c '
31+
32+
# LD_LIBRARY_PATH additions needed by BOTH the gen scripts (which import
33+
# deepmd.pt and dlopen the custom op .so that depends on libdeepmd.so in the
34+
# install prefix) AND ctest. Set once up front so either phase works alone.
35+
# The install prefix may not exist yet during the build; a missing dir in
36+
# LD_LIBRARY_PATH is harmless.
37+
export LD_LIBRARY_PATH=${INSTALL_PREFIX}/lib:${LD_LIBRARY_PATH}
38+
if [ "${ENABLE_PADDLE:-TRUE}" == "TRUE" ]; then
39+
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${PADDLE_INFERENCE_DIR}/third_party/install/onednn/lib:${PADDLE_INFERENCE_DIR}/third_party/install/mklml/lib
40+
fi
41+
42+
if [ "${DP_CC_SKIP_BUILD}" != "1" ]; then
43+
cmake \
44+
-D ENABLE_TENSORFLOW=${ENABLE_TENSORFLOW:-TRUE} \
45+
-D ENABLE_PYTORCH=${ENABLE_PYTORCH:-TRUE} \
46+
-D ENABLE_PADDLE=${ENABLE_PADDLE:-TRUE} \
47+
-D INSTALL_TENSORFLOW=FALSE \
48+
-D USE_TF_PYTHON_LIBS=${ENABLE_TENSORFLOW:-TRUE} \
49+
-D USE_PT_PYTHON_LIBS=${ENABLE_PYTORCH:-TRUE} \
50+
-D CMAKE_INSTALL_PREFIX=${INSTALL_PREFIX} \
51+
-D BUILD_TESTING:BOOL=TRUE \
52+
-D LAMMPS_VERSION=stable_22Jul2025_update2 \
53+
${CUDA_ARGS} ..
54+
cmake --build . -j${NPROC}
55+
cmake --install .
56+
# Generate PT/PT2 model files for C++ tests.
57+
# Must run after cmake --build so that libdeepmd_op_pt.so (custom ops) is available.
58+
if [ "${ENABLE_PYTORCH:-TRUE}" == "TRUE" ]; then
59+
# Install the custom op .so to SHARED_LIB_DIR so that `import deepmd.pt`
60+
# loads it via cxx_op.py.
61+
python -c '
4462
import shutil, sys
4563
from pathlib import Path
4664
from deepmd.env import SHARED_LIB_DIR
@@ -55,67 +73,67 @@ else:
5573
shutil.copy2(str(so), str(dst))
5674
print(f"Installed {so} -> {dst}")
5775
'
58-
# When the build uses -fsanitize=leak, the custom op .so requires the LSAN
59-
# runtime to be preloaded (otherwise dlopen fails). We disable leak detection
60-
# in the gen scripts to avoid false reports from torch/paddle internals.
61-
INFER_SCRIPT_PATH=${SCRIPT_PATH}/../tests/infer
62-
# Remove stale generated model files so they can't be accidentally reused
63-
# if gen scripts change format or the code version changes.
64-
rm -f ${INFER_SCRIPT_PATH}/*.pt2 ${INFER_SCRIPT_PATH}/*.pte
65-
_GEN_ENV=""
66-
if echo "${CXXFLAGS:-}" | grep -q fsanitize=leak; then
67-
_LSAN_LIB=$(gcc -print-file-name=liblsan.so 2>/dev/null || true)
68-
if [ -n "${_LSAN_LIB}" ] && [ -f "${_LSAN_LIB}" ]; then
69-
# DP_GEN_UNDER_SANITIZER: explicit signal for gen scripts that need
70-
# to skip sanitizer-incompatible sections (e.g. gen_dpa2.py's
71-
# AOTInductor graph .pt2 eval, which can SEGV under the LSAN
72-
# runtime). Sniffing LD_PRELOAD inside the gen script is NOT
73-
# reliable: the sanitizer runtime removes its own entry from the
74-
# process environment during startup.
75-
_GEN_ENV="LD_PRELOAD=${_LSAN_LIB} LSAN_OPTIONS=detect_leaks=0 DP_GEN_UNDER_SANITIZER=lsan"
76+
# When the build uses -fsanitize=leak, the custom op .so requires the LSAN
77+
# runtime to be preloaded (otherwise dlopen fails). We disable leak detection
78+
# in the gen scripts to avoid false reports from torch/paddle internals.
79+
INFER_SCRIPT_PATH=${SCRIPT_PATH}/../tests/infer
80+
# Remove stale generated model files so they can't be accidentally reused
81+
# if gen scripts change format or the code version changes.
82+
rm -f ${INFER_SCRIPT_PATH}/*.pt2 ${INFER_SCRIPT_PATH}/*.pte
83+
_GEN_ENV=""
84+
if echo "${CXXFLAGS:-}" | grep -q fsanitize=leak; then
85+
_LSAN_LIB=$(gcc -print-file-name=liblsan.so 2>/dev/null || true)
86+
if [ -n "${_LSAN_LIB}" ] && [ -f "${_LSAN_LIB}" ]; then
87+
# DP_GEN_UNDER_SANITIZER: explicit signal for gen scripts that need
88+
# to skip sanitizer-incompatible sections (e.g. gen_dpa2.py's
89+
# AOTInductor graph .pt2 eval, which can SEGV under the LSAN
90+
# runtime). Sniffing LD_PRELOAD inside the gen script is NOT
91+
# reliable: the sanitizer runtime removes its own entry from the
92+
# process environment during startup.
93+
_GEN_ENV="LD_PRELOAD=${_LSAN_LIB} LSAN_OPTIONS=detect_leaks=0 DP_GEN_UNDER_SANITIZER=lsan"
94+
fi
7695
fi
77-
fi
78-
# Run gen scripts in parallel for faster model generation.
79-
# Wait on each PID separately so any failure is caught by set -e.
80-
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_sea.py &
81-
PID1=$!
82-
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_dpa1.py &
83-
PID2=$!
84-
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_dpa2.py &
85-
PID3=$!
86-
wait $PID1
87-
wait $PID2
88-
wait $PID3
96+
# Run gen scripts in parallel for faster model generation.
97+
# Wait on each PID separately so any failure is caught by set -e.
98+
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_sea.py &
99+
PID1=$!
100+
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_dpa1.py &
101+
PID2=$!
102+
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_dpa2.py &
103+
PID3=$!
104+
wait $PID1
105+
wait $PID2
106+
wait $PID3
89107

90-
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_dpa3.py &
91-
PID4=$!
92-
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_fparam_aparam.py &
93-
PID5=$!
94-
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_model_devi.py &
95-
PID6=$!
96-
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_chg_spin.py &
97-
PID9=$!
98-
wait $PID4
99-
wait $PID5
100-
wait $PID6
101-
wait $PID9
108+
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_dpa3.py &
109+
PID4=$!
110+
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_fparam_aparam.py &
111+
PID5=$!
112+
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_model_devi.py &
113+
PID6=$!
114+
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_chg_spin.py &
115+
PID9=$!
116+
wait $PID4
117+
wait $PID5
118+
wait $PID6
119+
wait $PID9
102120

103-
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_dpa4.py &
104-
PID9=$!
105-
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_dpa1_pairexcl.py &
106-
PID10=$!
107-
wait $PID9
108-
wait $PID10
121+
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_dpa4.py &
122+
PID9=$!
123+
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_dpa1_pairexcl.py &
124+
PID10=$!
125+
wait $PID9
126+
wait $PID10
109127

110-
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_spin.py &
111-
PID7=$!
112-
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_spin_model_devi.py &
113-
PID8=$!
114-
wait $PID7
115-
wait $PID8
128+
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_spin.py &
129+
PID7=$!
130+
env ${_GEN_ENV} python ${INFER_SCRIPT_PATH}/gen_spin_model_devi.py &
131+
PID8=$!
132+
wait $PID7
133+
wait $PID8
134+
fi
116135
fi
117-
if [ "${ENABLE_PADDLE:-TRUE}" == "TRUE" ]; then
118-
PADDLE_INFERENCE_DIR=${BUILD_TMP_DIR}/paddle_inference_install_dir
119-
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${PADDLE_INFERENCE_DIR}/third_party/install/onednn/lib:${PADDLE_INFERENCE_DIR}/third_party/install/mklml/lib
136+
137+
if [ "${DP_CC_SKIP_CTEST}" != "1" ]; then
138+
ctest --output-on-failure
120139
fi
121-
ctest --output-on-failure

0 commit comments

Comments
 (0)