[pull] master from deepmodeling:master - #307
Merged
Merged
Conversation
## Summary - evaluate `dp test` in atom-bounded chunks, with lazy on-demand decoding for LMDB inputs - preserve run-level MAE/RMSE aggregation and detail output across chunks while consolidating backend-independent test logic under `deepmd.infer.model_test` - build the neighbor representation declared by each atomic model during output-bias statistics, using `NeighborGraph` for graph-native models and the existing dense neighbor list for other models ## Why Large LMDB datasets were fully decoded before frame selection, so even a small test run could require loading tens of millions of frames. Separately, graph-native atomic models were routed through a fixed-capacity dense neighbor list while calibrating output bias; because those models do not declare a finite neighbor capacity, that path could allocate tens of gigabytes and fail before fine-tuning began. This change bounds LMDB decoding and model-evaluation memory while preserving the existing evaluation and output-statistics contracts. Ordinary `DeepmdData` inputs still materialize one test system before chunked evaluation. ## Additional correctness fixes - preserve legacy TensorFlow spin-model dispatch and spin virial/stress reporting - use canonical atomic tensor labels consistently across NPY and LMDB inputs - flatten atomic parameters correctly for graph-native output statistics - keep LMDB frame selection deterministic and honor availability subgroups during iteration - keep detail files isolated across systems and emit a single header across chunks - require atomic property labels when atomic metrics are requested ## Checks - `ruff check .` - `ruff format --check .` - targeted common model-test and LMDB regression tests - targeted PyTorch streaming/output-stat regression tests - `git diff --check` The TensorFlow atomic dipole/polar integration cases could not be run locally because this checkout has no built TensorFlow backend; they are left to CI. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Expanded model evaluation for energy, spin, DOS, property, dipole, and polarizability models. * Added chunked testing, frame limits, per-frame details, and atomic-level metrics. * **Improvements** * Large LMDB test datasets now load frames lazily for improved efficiency. * Added clearer weighted aggregation of MAE and RMSE results. * Graph-capable models now use graph-based neighbor representations during inference. * **Bug Fixes** * Improved handling of periodic systems, mixed atom counts, exclusions, optional outputs, missing labels, and atomic label naming. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Closes #5665. ## Summary - centralize remapping of negative virtual atom types to the reserved final row of padded embedding tables - apply the same pair-index convention across dpmodel, pt_expt, legacy PyTorch, and Paddle DPA1/DPA2/SE_T implementations - keep real-type-only exclusion and normalization tables on a separate clamp-to-real convention - use the shared remapping helper in SeZMTypeEmbedding - cover DPA1 virtual neighbors inside the cutoff, DPA2, both DPA3 local-mapping branches, pt_expt pair arithmetic, and legacy PyTorch descriptors ## Sentinel convention Only tables explicitly constructed with a final padding row may use ntypes as the remapped virtual type. Real-type-only tables such as davg, dstd, exclusion inputs, and spin masks must instead receive masked or clamped real-type indices. TypeEmbedNet reconstructs a literal zero padding row; SeZMTypeEmbedding reserves and initializes its stored final row to zero. ## Validation - focused virtual-type and backend regressions: 15 passed - DPA1 graph parity plus the existing DPA2 descriptor tests: 18 passed - pt_expt DPA1 export tests: 8 passed - changed-file ruff check: passed - ruff format .: passed - git diff --check: passed - Paddle-specific code was statically checked but could not be executed locally because Paddle is not installed - full ruff check . still reports five unrelated pre-existing findings in deepmd/jax/jax_md/__init__.py and deepmd/tf/entrypoints/__init__.py Coding agent: Codex Codex version: codex-cli 0.144.6 Model: gpt-5.6-sol Reasoning effort: xhigh <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added consistent padding-aware type-embedding handling across dense, graph, and accelerated descriptor calculations. - Virtual or negative atom types now map safely to the designated padding embedding. - **Bug Fixes** - Prevented incorrect embedding lookups and invalid type indexing across supported execution paths. - Ensured virtual types produce results equivalent to explicit padding types. - **Tests** - Added regression coverage across NumPy, PyTorch, descriptor modes, and embedding pathways. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: njzjz-bot <njzjz.bot@gmail.com> Co-authored-by: njzjz-bot <njzjz-bot@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )