[pull] master from deepmodeling:master - #297
Merged
Merged
Conversation
## Summary - preserve the folded sorted-padding forward contract and make only the first sentinel depend on `two_embed` - scale that sentinel gradient by the padding-tail length while leaving later padding gradients zero on CPU and GPU - add finite-difference and nonuniform-cotangent grad-grad regression tests, including a GPU tail longer than the four-warp tile ## Validation - `source/build/lib/tests/runUnitTests_lib --gtest_filter='TestTabulateSeA.*:TestTabulateSeASortedPaddingTwoEmbed.*'` - CUDA 12.4 build of `deepmd_op_cuda` and `runUnitTests_lib` - RTX 5090 Slurm run of all 8 existing/new SE-A CPU and GPU tests - `ruff format .` - `ruff check .` - `clang-format --dry-run --Werror` on changed C++/CUDA files Closes #5891 Note: #5844 modifies the same GPU kernel for the shared-breakpoint fix, so the later-merging branch may need a small conflict resolution; the gradient contract fixed here is independent. Coding agent: Codex Codex version: codex-cli 0.144.6 Model: gpt-5.6-sol Reasoning effort: xhigh <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Corrected sorted-padding two-embedding gradient behavior so only the first sentinel receives the full tail multiplicity; later padding sentinel gradients remain zero. * Updated CPU and GPU gradient/grad-grad accumulation to apply the padding-tail repeat factor consistently. * Ensured the GPU two-embedding gradient buffer is cleared only when applicable to avoid stale values. * **Tests** * Strengthened sorted-padding assertions and expanded GPU coverage to confirm forward output consistency with CPU before numerical gradient checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: njzjz-bot <njzjz-bot@users.noreply.github.com> Co-authored-by: njzjz-bot <njzjz.bot@gmail.com>
## Summary - forward `is_sorted` through the PyTorch SE-Attention forward helper to the CPU and GPU library entry points - forward the saved flag through the first-backward helper so all autograd stages use the same sorting contract - require a sorted lower neighbor list while DPA1 geometric compression is active, matching the tabulation fold invariant - add float32 and float64 regression coverage for unsorted forward, backward, and double-backward behavior, including a nonzero table derivative - add compressed DPA1 `forward_lower` coverage for unsorted rcut+skin lists and an end-to-end plain `se_atten` case with `exclude_types` and `set_davg_zero: true` - document why the non-attention `se_a` path intentionally keeps the sorted fold enabled Fixes #5890 ## Historical origin - The PyTorch custom-op forwarding bug was introduced by #3877, which added `tabulate_fusion_se_atten` while omitting the backend `is_sorted` argument in forward and first backward. - The compressed DPA1 lower-nlist exposure became reachable in #4227, which added PyTorch `se_atten` geometric compression while `need_sorted_nlist_for_lower()` still returned `False`. #3993 introduced that API/default earlier, but compression did not yet use the sorted fold. ## Performance note With non-empty `exclude_types`, the corrected `is_sorted=False` path traverses the full neighbor list instead of applying the sorted-padding fold. This may be measurably slower, but avoids silently folding interleaved excluded rows. ## Testing - CPU-only PyTorch `deepmd_op_pt` C++ target build - `python -m pytest source/tests/pt/test_tabulate_fusion_se_atten.py -v` (12 passed) - `python -m pytest source/tests/pt/model/test_compressed_se_atten_forward_lower.py -v` (1 passed) - `python -m pytest source/tests/pt/test_model_compression_se_atten.py::TestDeepPotATPBCExcludeTypes::test_1frame -v` (1 passed) - `ruff format .` - `ruff check` on all modified Python files - commit hooks, including Ruff, pylint, and clang-format - `dp --version` - `dp --pt -h` - Python imports for `deepmd` and `deepmd.pt` `ruff check .` was also run; it reports five pre-existing findings in `deepmd/jax/jax_md/__init__.py` and `deepmd/tf/entrypoints/__init__.py`, outside this PR. Coding agent: Codex Codex version: codex-cli 0.144.6 Model: gpt-5.6-sol Reasoning effort: xhigh --------- Co-authored-by: njzjz-bot <njzjz-bot@users.noreply.github.com> Co-authored-by: njzjz-bot <njzjz.bot@gmail.com>
## Summary - add registry-backed, cached `supported_backends(...)` labels while keeping explicit backend keys beside each documented argument and variant - document TensorFlow (`tf`), PyTorch (`pt`), JAX (`jax`), PaddlePaddle (`pd`), PyTorch Exportable (`pt_expt`), and TensorFlow 2 (`tf2`) - make future additions a registry entry and allow deprecated backends to be hidden globally with `visible=False` - update the support matrices after checking current backend registrations and training dispatch, including the gaps tracked in #5755, #5756, and #5757 - add focused tests for display order, cache reuse, hidden/unknown backends, and representative support matrices ## Verification - `ruff format .` - `ruff check .` - `venv/bin/python -m pytest source/tests/common/test_argcheck_backend_docs.py source/tests/common/test_doc_train_input.py -q` (10 passed) - `venv/bin/python -m py_compile deepmd/utils/argcheck.py source/tests/common/test_argcheck_backend_docs.py` - `git diff --check` - generated `dp doc-train-input` and confirmed the six-backend label - ran `dp --version` and imported `deepmd`, `deepmd.tf`, and `deepmd.pt` Refs #5755 Refs #5756 Refs #5757 Coding agent: Codex Codex version: codex-cli 0.144.6 Model: gpt-5.6-sol Reasoning effort: xhigh <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated argument/option doc strings to generate “Supported Backend” labels from a centralized registry, with stable ordering and duplicate removal. * Hidden backends are omitted automatically; “stat_file_mode” now reflects a broader backend set. * Unknown backend identifiers are rejected with a clear error message, and embedded documentation spacing is kept consistent. * **Tests** * Added backend-doc label validation covering ordering/deduplication, hidden/all-hidden filtering, unknown-backend errors, and representative doc-string prefix behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: njzjz-bot <njzjz.bot@gmail.com>
…les (#5914) ## Summary - parallelize LMDB batch decoding across PT and PT-expt backends with bounded worker resources - add LMDB worker configuration and strengthen dataset/resource lifecycle handling - resolve backend-independent epoch-to-step schedules, including multi-task probabilities and distributed rank synchronization - cover LMDB decoding, scheduling, multitask, and distributed-training behavior with regression tests ## Checks No test suite was run per request; the changes were tested separately on another machine. Repository pre-commit style hooks passed while creating the cherry-pick commit. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Unified step scheduling for single- and multi-task training, deriving total steps and task sampling probabilities from config. - Faster LMDB training input pipeline with deterministic parallel batch iteration and config-driven decoding, plus a new LMDB batch loader for training. - **Bug Fixes** - Pinned epoch-to-step/run-length decisions for consistent distributed behavior across ranks. - Improved LMDB safety with reliable decoder/loader cleanup on failure and safer dataset merging/iterator lifecycle. - **Documentation** - Documented `DP_LMDB_NUM_WORKERS` override and auto-selection behavior. - **Tests** - Expanded coverage for step scheduling, LMDB decoding/batching correctness, distributed epoch consistency, and shutdown robustness. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary - correct `MapFltNvnmd`'s stale three-input check to its registered four-input contract - validate table, gradient-table, and interval metadata before dimension/index arithmetic - zero-fill values outside every configured interval, including the corresponding input gradient ## Why existing tests missed this `test_nvnmd_op.py` directly covered the other NVNMD arithmetic primitives but not `MapFltNvnmd`. Normal descriptor and mapping-generation callers keep values inside generated table ranges, release builds compile out the stale `DCHECK`, and freshly allocated output pages often happen to contain zeros. Together those conditions hid both the invalid debug contract and skipped output writes. The regression warms a large same-shaped output with nonzero values before evaluating out-of-range inputs, making allocator reuse expose the old behavior reliably instead of depending on fresh memory contents. ## Validation - `cmake --build source/build --target deepmd_op -j2` - fresh build-tree TensorFlow op: `TestOpMapFltNvnmd` (2 passed; inherited TensorFlow session helper skipped) - `ruff format .` - `ruff check .` - clang-format dry run with `--Werror` - `git diff --check` Fixes #5655 Coding agent: Codex Codex version: codex-cli 0.144.4 Model: gpt-5.6-sol Reasoning effort: xhigh <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Added strict runtime input validation (input count, tensor ranks/shapes, table width and interval metadata rules) with clear error messages. * Improved output initialization to avoid uninitialized results. * Updated interval selection to use the first matching interval (fine-to-coarse), with consistent clamping at table edges. * **Tests** * Added/expanded tests covering clamp behavior, “first matching interval wins,” gradient correctness, and a comprehensive set of invalid-input cases. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: njzjz-bot <njzjz.bot@gmail.com>
## Summary - Calibrate DPA4/SeZM output bias statistics through the complete atomic-model forward path, including model-level preprocessing and analytical contributions. - Apply the same frame masks and intensive reductions used by the public model outputs when computing statistics, including `min_pair_dist` filtering and multitask data handling. - Extend the PyTorch `torch.compile` compatibility range to PyTorch 2.13 and apply the Inductor symbolic-divisibility workaround to the affected releases. - Add and update coverage for SeZM bias calibration, spin-model behavior, training statistics, and compile compatibility. ## Root cause Bias calibration previously used an incomplete output path, so model-level contributions and output reduction semantics could be omitted from the fitted bias. Separately, PyTorch 2.13 was rejected before compilation, even though it requires the same Inductor compatibility handling as PyTorch 2.12. ## Validation - `git diff --check` passed. - Tests were not run locally, per request; the cherry-picked commits include the relevant test updates. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added minimum pair-distance frame filtering for training and statistics, with synchronized distributed step skipping when no frames are valid. * Added model support for statistics-driven atomic output prediction to power “change-by-statistic” bias adjustment. * Introduced an atomic-output-only execution mode for SeZM models, including a safe temporary preservation of training/eval state. * **Bug Fixes** * Fixed edge cases where empty samples/frames could break statistics and reduction logic, including safer intensive/masked reductions. * Improved SeZM spin/bridging output handling and migrated analytical bridging potential naming from InterPotential to InnerPotential. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Closes #5667. ## Summary - reject custom ASE neighbor lists in the TF2 SavedModel inference constructor - explain that SavedModel lower calls accept tensors and require a dedicated ASE-to-extended-tensors adapter plus output fold-back - add a focused regression test proving rejection happens before model loading ## Why existing tests missed this - TestDeepPotNeighborList exercises the TensorFlow 1 .pb inference path, not TF2 SavedModel inference. - TF2 consistency tests use the SavedModel native neighbor construction path and never pass the constructor ASE API. - live TF2 model tests use the internal tensor-compatible NeighborList, which is distinct from the inference constructor ASE object. - TF2 test collection is conditional, further reducing coverage of this backend-specific constructor path. ## Validation - pytest source/tests/tf2/test_deep_eval.py::test_custom_neighbor_list_is_rejected_before_model_loading -v - ruff format . - ruff check . - git diff --check Coding agent: Codex Codex version: codex-cli 0.144.4 Model: gpt-5.6-sol Reasoning effort: xhigh <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * TensorFlow 2 SavedModel inference clearly rejects unsupported custom neighbor-list configurations. * The user-facing error message indicates that the model’s built-in neighbor-list builder must be used. * **Tests** * Added a TensorFlow 2 inference adapter test to verify custom neighbor-list rejection occurs before model loading (with environment gating for TF2-only runs). <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: njzjz-bot <njzjz.bot@gmail.com>
Closes #5663. ## Summary - preserve negative placeholder types when creating dense and lower-interface spin partners - use one array-API-safe lookup that maps virtual atom types to zero instead of applying Python negative indexing - zero placeholder spin displacements, virial corrections, magnetic outputs, and magnetic masks - add dense, lower, end-to-end invariance, and Array API strict regressions ## Why existing tests missed this Existing spin consistency tests use only nonnegative real atom types, including their lower-interface ghost atoms. Negative-type tests cover base atomic models and neighbor-list builders, where masking happens before type-dependent work, but not `SpinModel` preprocessing, which runs earlier. Cross-backend spin tests therefore never supplied a nonzero padded spin or asserted that both the real placeholder and its generated spin partner remained negative. ## Validation - `pytest source/tests/common/dpmodel/test_spin_model_virtual_types.py source/tests/common/dpmodel/test_finetune_spin.py source/tests/common/dpmodel/test_spin_model_legacy_routing.py -q` (12 passed) - `ruff format .` (1664 files unchanged on final pass) - `ruff check .` (passed) - `git diff --check` (passed) Coding agent: Codex Codex version: codex-cli 0.144.4 Model: gpt-5.6-sol Reasoning effort: xhigh <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved spin-model handling of padded and virtual atom placeholders. - Prevented placeholder atoms from receiving displacement, virial, scaling, or magnetic-output corrections. - Ensured consistent behavior across standard and lower-level spin processing paths. - Improved compatibility across supported array backends and processing devices. - **Tests** - Added regression coverage for virtual placeholders, masking, output handling, and consistency between model backends. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: njzjz-bot <njzjz.bot@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )