perf: bound memory in model testing and output statistics (deepmodeling#5944)

## Summary

- evaluate `dp test` in atom-bounded chunks, with lazy on-demand
decoding for LMDB inputs
- preserve run-level MAE/RMSE aggregation and detail output across
chunks while consolidating backend-independent test logic under
`deepmd.infer.model_test`
- build the neighbor representation declared by each atomic model during
output-bias statistics, using `NeighborGraph` for graph-native models
and the existing dense neighbor list for other models

## Why

Large LMDB datasets were fully decoded before frame selection, so even a
small test run could require loading tens of millions of frames.
Separately, graph-native atomic models were routed through a
fixed-capacity dense neighbor list while calibrating output bias;
because those models do not declare a finite neighbor capacity, that
path could allocate tens of gigabytes and fail before fine-tuning began.

This change bounds LMDB decoding and model-evaluation memory while
preserving the existing evaluation and output-statistics contracts.
Ordinary `DeepmdData` inputs still materialize one test system before
chunked evaluation.

## Additional correctness fixes

- preserve legacy TensorFlow spin-model dispatch and spin virial/stress
reporting
- use canonical atomic tensor labels consistently across NPY and LMDB
inputs
- flatten atomic parameters correctly for graph-native output statistics
- keep LMDB frame selection deterministic and honor availability
subgroups during iteration
- keep detail files isolated across systems and emit a single header
across chunks
- require atomic property labels when atomic metrics are requested

## Checks

- `ruff check .`
- `ruff format --check .`
- targeted common model-test and LMDB regression tests
- targeted PyTorch streaming/output-stat regression tests
- `git diff --check`

The TensorFlow atomic dipole/polar integration cases could not be run
locally because this checkout has no built TensorFlow backend; they are
left to CI.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Expanded model evaluation for energy, spin, DOS, property, dipole, and
polarizability models.
* Added chunked testing, frame limits, per-frame details, and
atomic-level metrics.
* **Improvements**
* Large LMDB test datasets now load frames lazily for improved
efficiency.
  * Added clearer weighted aggregation of MAE and RMSE results.
* Graph-capable models now use graph-based neighbor representations
during inference.
* **Bug Fixes**
* Improved handling of periodic systems, mixed atom counts, exclusions,
optional outputs, missing labels, and atomic label naming.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Author: OutisLi

deepmd/dpmodel/atomic_model/base_atomic_model.py

@@ -805,9 +805,22 @@ def _store_out_stat(
         self.out_std = out_std_data
 
     def _get_forward_wrapper_func(self) -> Callable[..., dict[str, np.ndarray]]:
-        """Get a forward wrapper of the atomic model for output bias calculation."""
+        """Get a forward wrapper of the atomic model for output bias calculation.
+
+        The wrapper starts from raw coordinates and therefore has to construct
+        the neighbor representation itself. It builds the one this model
+        declares through :meth:`uses_graph_lower`: a carry-all
+        ``NeighborGraph`` for graph-native models, whose neighbor count follows
+        the geometry, or the fixed-capacity neighbor list sized by
+        :meth:`get_sel` otherwise. Sizing a dense list from ``get_sel`` is not
+        merely wasteful for a graph-native model -- such a model reports no
+        finite capacity, so the allocation is unbounded.
+        """
         import array_api_compat
 
+        from deepmd.dpmodel.utils.neighbor_graph import (
+            build_neighbor_graph,
+        )
         from deepmd.dpmodel.utils.nlist import (
             extend_input_and_build_neighbor_list,
         )
@@ -841,31 +854,64 @@ def model_forward(
             if charge_spin is not None:
                 charge_spin = xp.asarray(charge_spin, device=device)
 
-            (
-                extended_coord,
-                extended_atype,
-                mapping,
-                nlist,
-            ) = extend_input_and_build_neighbor_list(
-                coord,
-                atype,
-                self.get_rcut(),
-                self.get_sel(),
-                mixed_types=self.mixed_types(),
-                box=box,
-                # exclusion is a nlist-BUILD transform (decision #18/A4);
-                # forward_common_atomic consumes a pre-excluded nlist.
-                pair_excl=self.pair_excl,
-            )
-            atomic_ret = self.forward_common_atomic(
-                extended_coord,
-                extended_atype,
-                nlist,
-                mapping=mapping,
-                fparam=fparam,
-                aparam=aparam,
-                charge_spin=charge_spin,
-            )
+            if self.uses_graph_lower():
+                nframes, nloc = atype.shape
+                # Pair exclusion is a neighbor-BUILD transform (decision
+                # #18/A4) on both routes; the graph builder folds it into
+                # ``edge_mask``.
+                graph = build_neighbor_graph(
+                    coord,
+                    atype,
+                    box,
+                    self.get_rcut(),
+                    pair_excl=self.pair_excl,
+                )
+                atomic_ret = self.forward_common_atomic_graph(
+                    graph,
+                    xp.reshape(atype, (-1,)),
+                    fparam=fparam,
+                    aparam=(
+                        xp.reshape(
+                            aparam,
+                            (nframes * nloc, self.get_dim_aparam()),
+                        )
+                        if aparam is not None
+                        else None
+                    ),
+                    charge_spin=charge_spin,
+                )
+                # The graph route works on a flat node axis; restore the
+                # per-frame layout the dense route returns.
+                atomic_ret = {
+                    kk: xp.reshape(vv, (nframes, nloc, *vv.shape[1:]))
+                    for kk, vv in atomic_ret.items()
+                }
+            else:
+                (
+                    extended_coord,
+                    extended_atype,
+                    mapping,
+                    nlist,
+                ) = extend_input_and_build_neighbor_list(
+                    coord,
+                    atype,
+                    self.get_rcut(),
+                    self.get_sel(),
+                    mixed_types=self.mixed_types(),
+                    box=box,
+                    # exclusion is a nlist-BUILD transform (decision #18/A4);
+                    # forward_common_atomic consumes a pre-excluded nlist.
+                    pair_excl=self.pair_excl,
+                )
+                atomic_ret = self.forward_common_atomic(
+                    extended_coord,
+                    extended_atype,
+                    nlist,
+                    mapping=mapping,
+                    fparam=fparam,
+                    aparam=aparam,
+                    charge_spin=charge_spin,
+                )
             # Convert outputs back to numpy arrays
             return {kk: to_numpy_array(vv) for kk, vv in atomic_ret.items()}