[Bug] DeepH loses H-only edges and cannot read current ABACUS CSR/get_s output

[goodchong]

Describe the bug

The current ABACUS–DeepH interface has two independent classes of failure:

1. Correctness: the atom-pair support of the ABACUS Hamiltonian is not always a subset of the overlap support. DeepH uses overlap support to construct the inference graph, so valid Hamiltonian edges can be absent from the graph and prediction output.
2. Current-version I/O compatibility: current ABACUS develop output cannot be consumed by either public DeepH 0.2.2 or the current documented DeepH 1.0.6 path through DeepH-dock 0.9.12.

Versions checked on 2026-08-28:

• ABACUS develop e461ffd4246e6be41189d35a242bb44c4c50eca9, runtime banner v3.11.0-beta8.
• Public DeepH-pack 141faaaba1d5e17248283ef939bcd2ead2d8e677, version 0.2.2.
• DeepH-pack documented wheel deepx-1.0.6+light.
• DeepH-dock 6fc20bb4ec00c8d4de1d192338ba116527772f7a, version 0.9.12.

1. Reproduced H/S support mismatch

ABACUS creates overlap pairs only for direct orbital overlap:

distance(i,j) < r_phi(i) + r_phi(j)

However, the nonlocal Hamiltonian can couple two orbitals through a third atom's projector:

<phi_i|beta_I> D_I <beta_I|phi_j>

Therefore, S_ij(R) == 0 does not imply H_ij(R) == 0. This follows directly from the different pair construction in overlap.cpp and nonlocal.cpp.

In the four-Si reproducer below, the NAO cutoff is 6 Bohr and the nonlocal-projector cutoff is 1.94 Bohr. Atoms 1 and 3 are 13.52296 Bohr apart, so they have no direct overlap, while each is 7 Bohr from atom 2 and couples through its projector.

At an atom-block threshold of 1e-8, current ABACUS produces:

H atom-block keys: 16
S atom-block keys: 14
H-only [0, 0, 0, 1, 3]: max|H|=5.48772265e-05 Ry, max|S|=0
H-only [0, 0, 0, 3, 1]: max|H|=5.48772265e-05 Ry, max|S|=0

These overlap blocks are structurally absent, not merely filtered numerical noise, so lowering the sparse threshold does not solve the problem.

This is deterministically harmful in the current converter: DeepH-dock first derives atom_pairs from overlap nonzeros, then extracts Hamiltonian blocks only for those overlap-derived pairs (source). Thus H-only pairs are discarded during conversion. Current DeepH documentation also permits only S/Sap graph construction during inference and states that overlap sparsity is indispensable, while training defaults to H and also supports HS (inference, training).

Possible consequences are silently missing predicted Hamiltonian blocks, incorrect bands/eigenvalues, and downstream generalized-eigenproblem failures.

2. Current ABACUS output is not readable by the documented DeepH path

Current ABACUS SCF output is:

running_scf.log
hrs1_nao.csr
hrs2_nao.csr  # nspin=2
sr_nao.csr

DeepH-dock 0.9.12 still hard-codes:

running_scf.log
data-HR-sparse_SPIN0.csr
data-HR-sparse_SPIN1.csr
data-SR-sparse_SPIN0.csr

It also parses the legacy CSR header, while ABACUS v3.9.0.25+ writes the rich CSR header with spin metadata, UnitCell data, and labeled CSR sections. The current DeepH-dock documentation explicitly targets ABACUS 3.10 LTS and shows only the legacy layout (documentation).

For overlap-only inference, an actual calculation get_s run on e461ffd produced:

OUT.<suffix>/running_get_s.log
OUT.<suffix>/sr_nao.csr

DeepH-dock and public DeepH 0.2.2 expect case-sensitive running_get_S.log and SR.csr. The latest DeepH prediction guide also still says that ABACUS dumps SR.csr (documentation).

Public DeepH 0.2.2 has additional incompatibilities: it searches case-sensitively for lattice constant (Bohr) while current ABACUS prints Lattice constant (Bohr), rejects nspin=2, and applies a separate 1e-8 atom-block filter (parser).

This conflicts with the current ABACUS documentation claim that “DeepH v1.0.0+” reads the v3.9.0.25+ CSR format (ABACUS documentation). I could not find a current released converter that implements that contract.

3. Other interface limitations that should be part of the supported-version contract

• DeepH-dock cannot distinguish two ABACUS species of the same element that use different basis/pseudopotential settings; it requires standard element-symbol labels.
• Public DeepH 0.2.2 does not support collinear nspin=2. DeepH-dock added support for the legacy two-file layout, but current hrs1_nao.csr/hrs2_nao.csr cannot reach that path.
• Current ABACUS native-binary and NPZ out_hsr modes have no documented DeepH conversion path; only text CSR should be advertised as supported.
• Public DeepH 0.2.2 and the new JAX deepx-1.0.6 are separate implementations. The compatibility documentation should name exact ABACUS, DeepH-dock, and DeepH versions.

Expected behavior

1. The graph used for training and inference must contain every Hamiltonian target/prediction edge:

H_target_keys <= graph_keys

2. Current ABACUS SCF and get_s output should be readable by the converter advertised in the ABACUS documentation.
3. The two projects should publish one exact compatibility matrix covering ABACUS version/commit, CSR format, DeepH-dock version, DeepH version, and nspin=1/2/4.

A robust fix would export or construct a dedicated graph-support/atom-pair artifact containing the union of direct-overlap and nonlocal-projector-mediated Hamiltonian pairs. This should be independent of numerical overlap values; changing the physical overlap matrix or only lowering a threshold is not sufficient.

To Reproduce

1. Build ABACUS develop at e461ffd4246e6be41189d35a242bb44c4c50eca9 with LCAO, MPI, and OpenMP enabled.
2. Create a case directory and copy these files from the ABACUS repository:

cp ../../tests/PP_ORB/Si_dojo_nsoc.upf .
cp ../../tests/PP_ORB/Si_dojo_6au_sz.orb .

3. Use the following input files.

INPUT:

INPUT_PARAMETERS
suffix            deeph_bridge4
calculation       scf
nbands            10
symmetry          0
pseudo_dir        ./
orbital_dir       ./
gamma_only        0
ecutwfc           20
scf_thr           1e-8
scf_nmax          100
basis_type        lcao
smearing_method   gauss
smearing_sigma    0.002
mixing_type       broyden
mixing_beta       0.7
mixing_gg0        0.0
out_hsr           1 8
ks_solver         scalapack_gvx

STRU:

ATOMIC_SPECIES
Si 14 Si_dojo_nsoc.upf upf201

NUMERICAL_ORBITAL
Si_dojo_6au_sz.orb

LATTICE_CONSTANT
1.0

LATTICE_VECTORS
40.0 0.0 0.0
0.0 40.0 0.0
0.0 0.0 40.0

ATOMIC_POSITIONS
Direct

Si
0.0
4
0.325000000000 0.500000000000 0.500000000000 1 1 1
0.500000000000 0.500000000000 0.500000000000 1 1 1
0.651554445663 0.587500000000 0.500000000000 1 1 1
0.500000000000 0.500000000000 0.675000000000 1 1 1

KPT:

K_POINTS
0
Gamma
1 1 1 0 0 0

4. Run with one OpenMP thread:

OMP_NUM_THREADS=1 /path/to/abacus

5. Compare 4-by-4 atom blocks in hrs1_nao.csr and sr_nao.csr at 1e-8. The two H-only pairs and values shown above are obtained.
6. Change the INPUT to calculation get_s and run again. The actual output names are running_get_s.log and sr_nao.csr, which the current DeepH-dock ABACUS validator does not recognize.

Environment

• OS: Ubuntu 22.04
• Compiler: GCC 11.4.0 through Open MPI mpicxx
• MPI: Open MPI 4.1.2
• Linear algebra: OpenBLAS, LAPACK, ScaLAPACK
• ABACUS build: MPI ON, OpenMP ON, LCAO ON, ELPA ON
• Runtime: CPU, one MPI rank, OMP_NUM_THREADS=1

The SCF converged in 84 iterations. The get_s reproduction completes without SCF in about 0.08 seconds.

Additional Context

The old warning in the public DeepH repository already states that the DeepH–ABACUS interface has an overlap sparsity-pattern bug (warning). The reproducer above identifies a concrete physical/source-code cause and shows that the warning is still applicable to the latest documented S/Sap inference workflow.

Related issue #3691 is not a duplicate. That issue concerned a spin/SOC dimension mismatch (50-by-50 H versus 25-by-25 S) and was resolved by setting the converted data as spinful. This report concerns missing atom-pair keys even when H and S have the same orbital dimension.

Suggested regression coverage:

• Assert H_target_keys <= graph_keys for the four-Si projector bridge case.
• End-to-end SCF and get_s conversion tests for nspin=1, nspin=2, and nspin=4.
• Pin exact ABACUS, DeepH-dock, and DeepH versions in CI and documentation.