Skip to content

Commit 9245a7b

Browse files
wanghan-iapcmHan Wang
andauthored
fix(test): tear down LAMMPS before MPI.Finalize() in mpirun test runners (deepmodeling#5455)
## Summary - Adds an explicit `del lammps` before `MPI.Finalize()` in all four mpirun-driven LAMMPS test runners (`run_mpi_pair_deepmd.py`, `run_mpi_pair_deepmd_spin.py`, `run_mpi_pair_deepmd_dpa3_pt2.py`, `run_mpi_pair_deepmd_spin_dpa3_pt2.py`). - Fixes a teardown-order race that intermittently manifests as **subprocess exit code 136 (SIGFPE)** for `test_pair_deepmd_mpi_dpa3_spin_empty_subdomain` on the GitHub Actions CUDA runner image. ## Background Recent CI runs on multiple unrelated PRs (deepmodeling#5446, deepmodeling#5450) hit the identical failure signature: ``` short test summary info ============================ FAILED source/lmp/tests/test_lammps_spin_dpa3_pt2.py::test_pair_deepmd_mpi_dpa3_spin_empty_subdomain - subprocess.CalledProcessError: ... returned non-zero exit status 136. ``` - Reproduces ~1 in 5 runs on the GitHub Actions CUDA image (`nvidia/cuda:12.9.1-cudnn-devel-ubuntu22.04`). - **Does not reproduce on a V100 Bohrium dev box** — 60/60 consecutive passes. So it's a pre-existing flake, not caused by either of the recent PRs. ## Root cause (empirically confirmed) The runner ends with: ```python forces_global = lammps.lmp.gather_atoms(...) ... MPI.Finalize() ``` `lammps` is still alive when `MPI.Finalize()` returns. Python then garbage-collects it during interpreter shutdown, which triggers `LAMMPS::~LAMMPS` → `Finish::end()` → **`MPI_Allreduce`** for timing aggregation. By that time, MPI has already been finalized, which is undefined behavior. I instrumented the runner with timestamped prints to verify the order directly. Without the fix: ``` t=3311.770 R1: BEFORE MPI.Finalize t=3311.778 R0/R1: AFTER MPI.Finalize ← MPI is finalized t=3311.778 R0/R1: PY ATEXIT … process exit, LAMMPS destructor runs HERE ``` With the fix: ``` t=3423.100 R1: AFTER del lammps (LAMMPS destructor done) ← MPI still up t=3423.108 R0/R1: BEFORE MPI.Finalize t=3423.108 R0/R1: AFTER MPI.Finalize ``` So the LAMMPS destructor now runs while MPI is still up, which is what its `MPI_Allreduce`/`MPI_Gather` calls require. The reason this manifests as SIGFPE only on the CUDA CI image (not on V100) is most likely that the CI image (or one of its preloaded libraries) enables FP-exception trapping; on V100 the same MPI-after-Finalize errors return silently. The flake is environment-specific, but the underlying antipattern is unconditional and worth fixing in any environment. ## Test plan - [x] Local CPU: 29/29 LAMMPS tests pass (`test_lammps_dpa3_pt2.py`, `test_lammps_spin_dpa3_pt2.py`) - [x] Remote V100: 50/50 stress runs of the previously-failing test - [x] Empirical confirmation that the fix flips the LAMMPS-destructor-vs-MPI.Finalize ordering (see Background) - [ ] CI: re-run the spin LAMMPS suite multiple times to confirm the SIGFPE no longer appears ## Known limitations - Cannot directly observe the SIGFPE on V100, so the fix has not been observed *preventing* the actual crash — only correcting the antipattern that we have strong reason to believe causes it. - If the failure persists after merge, the next candidate root cause is CUDA stream destruction order, and we should revisit. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved MPI cleanup sequence in multiple test runners to prevent finalization-related crashes when executing tests in distributed MPI environments. <!-- review_stack_entry_start --> [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/deepmodeling/deepmd-kit/pull/5455?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Han Wang <wang_han@iapcm.ac.cn>
1 parent 3384efb commit 9245a7b

4 files changed

Lines changed: 19 additions & 0 deletions

File tree

source/lmp/tests/run_mpi_pair_deepmd.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,4 +62,7 @@
6262
pe = lammps.eval("pe")
6363
arr = [pe]
6464
np.savetxt(output, np.array(arr))
65+
# Tear down LAMMPS before MPI.Finalize() to avoid MPI-after-Finalize
66+
# in the LAMMPS destructor. See run_mpi_pair_deepmd_spin_dpa3_pt2.py.
67+
del lammps
6568
MPI.Finalize()

source/lmp/tests/run_mpi_pair_deepmd_dpa3_pt2.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -225,4 +225,8 @@
225225
row = np.concatenate([fi, vi])
226226
f.write(" ".join(f"{v:.16e}" for v in row) + "\n")
227227

228+
# Tear down LAMMPS before MPI.Finalize() — see the matching comment in
229+
# ``run_mpi_pair_deepmd_spin_dpa3_pt2.py``. Same teardown-order race
230+
# class; spin happens to hit it more often on CUDA CI.
231+
del lammps
228232
MPI.Finalize()

source/lmp/tests/run_mpi_pair_deepmd_spin.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,4 +62,7 @@
6262
pe = lammps.eval("pe")
6363
arr = [pe]
6464
np.savetxt(output, np.array(arr))
65+
# Tear down LAMMPS before MPI.Finalize() to avoid MPI-after-Finalize
66+
# in the LAMMPS destructor. See run_mpi_pair_deepmd_spin_dpa3_pt2.py.
67+
del lammps
6568
MPI.Finalize()

source/lmp/tests/run_mpi_pair_deepmd_spin_dpa3_pt2.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,4 +144,13 @@
144144
row = np.concatenate([fi, fmi, vi])
145145
f.write(" ".join(f"{v:.16e}" for v in row) + "\n")
146146

147+
# Tear down the LAMMPS instance *before* ``MPI.Finalize()`` so its
148+
# destructor's MPI calls (fix/compute cleanup, timing reductions inside
149+
# ``Finish::end``, the deep-spin pair-style destructor chain, etc.) run
150+
# while the communicator is still valid. Without this, Python keeps
151+
# ``lammps`` alive past ``MPI.Finalize()`` and only releases it during
152+
# interpreter shutdown — and the empty-subdomain rank then hits an
153+
# MPI-after-Finalize call which crashes with SIGFPE on some CUDA CI
154+
# runners (intermittent; not reproducible on V100).
155+
del lammps
147156
MPI.Finalize()

0 commit comments

Comments
 (0)