Skip to content

Commit 5f848ad

Browse files
wanghan-iapcmHan Wang
andauthored
test(pt_expt): tolerance for test_default_fallback (CUDA scatter nondeterminism) (deepmodeling#5508)
## Summary `test_default_fallback` (added in deepmodeling#5491) compared `neighbor_list=None` against an explicit `DefaultNeighborList()` with **exact** equality (`assert_array_equal`). The two are the identical builder — `call_common` does `builder = nl if nl is not None else DefaultNeighborList()` — so on CPU the two forward passes are bit-identical. But the test runs **two independent forward passes**, and on **CUDA** the dpa3 GNN message-passing scatter (atomic adds) is not bit-reproducible run-to-run. The virial differed by ~1 ULP (abs `3.46e-18`, rel `5.64e-16`), failing the exact comparison intermittently: ``` FAILED test_default_fallback[dpa3] - AssertionError: Arrays are not equal dpa3 virial — Mismatched elements: 1 / 9 (11.1%) Max absolute difference: 3.47e-18 ; Max relative difference: 5.64e-16 ``` ## Fix Switch the comparison to `assert_allclose(rtol=1e-10, atol=1e-12)` — far above the fp noise floor (~`1e-16` rel) but orders of magnitude tighter than any real dispatch divergence (e.g. accidentally using a different builder) would produce. Verified on CPU that `None` and `DefaultNeighborList()` are bit-identical (max|Δ| = 0), confirming the residual is CUDA atomic nondeterminism, not a dispatch bug. ## Known limitations - Validated on CPU (8/8 `test_default_fallback` pass). The failing **CUDA** path could not be re-validated locally (no GPU available this session); the CI CUDA job is the confirmation. The tolerance is ~6 orders of magnitude above the observed noise, so it is robust. - Relaxes the previously-exact comparison; this is the intended change. The sibling `test_pt_expt_equivalence` tests already use a `1e-9` tolerance. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Tests** * Enhanced test robustness by implementing floating-point tolerance comparisons instead of strict equality checks, ensuring reliable testing across different hardware configurations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Han Wang <wang_han@iapcm.ac.cn>
1 parent e05af21 commit 5f848ad

1 file changed

Lines changed: 14 additions & 2 deletions

File tree

source/tests/pt_expt/utils/test_neighbor_list.py

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -476,7 +476,17 @@ def test_pt_expt_multiframe_equivalence(name: str) -> None:
476476

477477
@pytest.mark.parametrize("name", list(ALL_MODELS)) # descriptor family
478478
def test_default_fallback(name: str) -> None:
479-
"""``neighbor_list=None`` equals an explicit DefaultNeighborList byte-for-byte."""
479+
"""``neighbor_list=None`` dispatches to the same DefaultNeighborList builder.
480+
481+
``None`` and an explicit ``DefaultNeighborList()`` are the identical builder
482+
(``call_common`` does ``builder = nl if nl is not None else DefaultNeighborList()``),
483+
so the two forward passes are the *same computation*; on CPU they are
484+
bit-identical. We compare with a tight tolerance rather than exact equality
485+
because the two passes are independent forward evaluations, and on CUDA the
486+
GNN message-passing scatter (atomic adds) is not bit-reproducible run-to-run,
487+
so the virial can differ by ~1 ULP between the passes (a real dispatch bug
488+
would differ by orders of magnitude more).
489+
"""
480490
coord_np, atype_np, box_np = _system()
481491
md = get_model(copy.deepcopy(ALL_MODELS[name])).to(env.DEVICE)
482492
md.eval()
@@ -492,8 +502,10 @@ def test_default_fallback(name: str) -> None:
492502
).requires_grad_(True)
493503
outs[tag] = md.forward(coord_t, atype_t, box=box_t, do_atomic_virial=True, **kw)
494504
for k in ("energy", "force", "virial"):
495-
np.testing.assert_array_equal(
505+
np.testing.assert_allclose(
496506
outs["none"][k].detach().cpu().numpy(),
497507
outs["explicit"][k].detach().cpu().numpy(),
508+
rtol=1e-10,
509+
atol=1e-12,
498510
err_msg=f"{name} {k}",
499511
)

0 commit comments

Comments
 (0)