Failure to Generate graph_data.npz for ABACUS LCAO Calculations

[TiacWei]

When using the HamGNN ABACUS interface (graph_data_gen_abacus.py) to generate graph_data.npz from ABACUS SCF calculations:
Success: LCAO Si (official ABACUS example) → generates graph_data.npz normally
Failure: LCAO Cu, LCAO ZnO (official ABACUS example) → script crashes with TypeError: data must be a sequence (got NoneType)
Tested ABACUS versions: v3.10.1 and v3.5.3 (same failure behavior)
Pre-requisite files generated successfully: data-H0R-sparse_SPIN0.csr / data-S0R-sparse_SPIN0.csr
Error Details
Full traceback from the script:

Processing 1 SCF outputs with 1 worker(s), worker_threads=1, chunksize=1.
Processing SCF Outputs:   0%|                                                                                                                                                             | 0/1 [00:00<?, ?it/s]Error generating Hamiltonian and overlap matrices: 29
Processing SCF Outputs:   0%|                                                                                                                                                             | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/user01/software/HamGNN-main/DFT_interfaces/abacus/graph_data_gen_abacus.py", line 757, in <module>
    main()
  File "/home/user01/software/HamGNN-main/DFT_interfaces/abacus/graph_data_gen_abacus.py", line 723, in main
    _, success, graph, payload = generate_graph(task)
  File "/home/user01/software/HamGNN-main/DFT_interfaces/abacus/graph_data_gen_abacus.py", line 646, in generate_graph
    hamiltonian=torch.FloatTensor(H),
TypeError: new(): data must be a sequence (got NoneType)

[QuantumLab-ZY]

Hello, thank you for the report. To help us further investigate the issue, could you please provide the full command and all parameters you used when running graph_data_gen_abacus.py?

In particular, we would like to confirm the following settings:

• --data-dirs
• --graph-data-folder
• --output-format
• Whether you modified NAO_MAX in the script
• Whether SOC_ENABLED was enabled
• The names of the .orb basis files used for Cu / ZnO

Based on the traceback, it appears that H becomes None during Hamiltonian assembly. One possible cause is that the script currently defaults to NAO_MAX = 13, while Cu / Zn require a larger ABACUS basis definition, such as NAO_MAX = 27 or 40.
Once we know the exact runtime parameters, we can further determine whether this is caused by a basis-dimension configuration mismatch.

[TiacWei]

Hello, thank you for the report. To help us further investigate the issue, could you please provide the full command and all parameters you used when running graph_data_gen_abacus.py?

In particular, we would like to confirm the following settings:

• --data-dirs
• --graph-data-folder
• --output-format
• Whether you modified NAO_MAX in the script
• Whether SOC_ENABLED was enabled
• The names of the .orb basis files used for Cu / ZnO

Based on the traceback, it appears that H becomes None during Hamiltonian assembly. One possible cause is that the script currently defaults to NAO_MAX = 13, while Cu / Zn require a larger ABACUS basis definition, such as NAO_MAX = 27 or 40. Once we know the exact runtime parameters, we can further determine whether this is caused by a basis-dimension configuration mismatch.

Thanks for your reply.
I finally solved this problem by change the orb file and the nao_max in graph_data_gen_abacus.py. But I still get Total length of Dataloader across ranks is zero. Please make sure that it returns at least 1 batch. error. So I think there are problems with the graph_data.npz file.

my workflow is

1. normal abacus run
2. abacus-postprocess-v353 (the running_scf_h0.log energies are all 0 for all kpoint (Cartesian) , is this wrong)
3. python DFT_interfaces
4. HamGNN2.0

the INPUT for abacus ZnO example (abacus offical example)

INPUT_PARAMETERS
pseudo_dir              ../../../tests/PP_ORB
orbital_dir                 /home/user01/software/upf/apns-orbitals-efficiency-v1
basis_type                              lcao
calculation                             scf
ntype                                   2

gamma_only                              0

ecutwfc                                 150
scf_nmax                                80
mixing_ndim                             4

out_mat_hs2  1
out_chg 1

ATOMIC_SPECIES
Zn 1.000 ./Zn.LDA.UPF
O  1.000 ./O.LDA.100.UPF

NUMERICAL_ORBITAL
Zn_gga_9au_150Ry_4s2p2d1f.orb
O_gga_6au_100Ry_2s2p1d.orb

LATTICE_CONSTANT
6.1416 

LATTICE_VECTORS
1.00    0.00    0.00
-0.5    0.866   0.00
0.00    0.00    1.6

ATOMIC_POSITIONS
Direct

Zn
0.0
1
0.00    0.00    0.00    1       1       1

O
0.0
1
0.33333 0.66667 0.50    1       1       1 

K_POINTS
0
Gamma
8 8 8 0 0 0

the OUT.ABACUS file contains

ABACUS-CHARGE-DENSITY.restart  data-S0R-sparse_SPIN0.csr  INPUT        running_scf_h0.log  STRU.cif                warning.log
data-H0R-sparse_SPIN0.csr      data-SR-sparse_SPIN0.csr   istate.info  running_scf.log     STRU_READIN_ADJUST.cif
data-HR-sparse_SPIN0.csr       device.log                 kpoints      SPIN1_CHG.cube      STRU_SIMPLE.cif

python ~/software/HamGNN-main/DFT_interfaces/abacus/graph_data_gen_abacus.py --data-dirs ./ --graph-data-folder ./ --output-format npz

HamGNN2.0 --config config.yaml > log.out 2>&1

dataset_params:
  batch_size: 1
  split_file: null
  test_ratio: 0.1
  train_ratio: 0.8
  val_ratio: 0.1
  graph_data_path: ./graph_data.npz # Directory where graph_data.npz is located

losses_metrics:
  losses:
  - loss_weight: 27.211
    metric: mae
    prediction: hamiltonian
    target: hamiltonian
  metrics:
  - metric: mae
    prediction: hamiltonian
    target: hamiltonian
optim_params:
  lr: 0.01
  lr_decay: 0.5
  lr_patience: 5
  gradient_clip_val: 0.0
  max_epochs: 3000
  min_epochs: 100
  stop_patience: 30

output_nets:
  output_module: HamGNN_out
  HamGNN_out:
    ham_only: true # true: Only the Hamiltonian H is computed; 'false': Fit both H and S
    ham_type: abacus # openmx: fit openmx Hamiltonian; abacus: fit abacus Hamiltonian
    nao_max: 27 # The maximum number of atomic orbitals in the data set, which can be 14, 19 or 27
    add_H0: True # Generally true, the complete Hamiltonian is predicted as the sum of H_scf plus H_nonscf (H0)
    symmetrize: True # if set to true, the Hermitian symmetry constraint is imposed on the Hamiltonian
    calculate_band_energy: False # Whether to calculate the energy bands to train the model
    num_k: 5 # When calculating the energy bands, the number of K points to use
    band_num_control: 8 # `dict`: controls how many orbitals are considered for each atom in energy bands; `int`: [vbm-num, vbm+num]; `null`: all bands
    k_path: null # `auto`: Automatically determine the k-point path; `null`: random k-point path; `list`: list of k-point paths provided by the user
    soc_switch: False # if true, fit the SOC Hamiltonian
    nonlinearity_type: gate # norm or gate
    # spin constrained
    spin_constrained: False
    collinear_spin: False
    minMagneticMoment: 0.5

profiler_params:
  progress_bar_refresh_rat: 1
  train_dir: ./ #The folder for saving training information and prediction results. This directory can be read by tensorboard to monitor the training process.

representation_nets:
  HamGNN_pre:
    legacy_edge_update: false
    cutoff: 26.0
    cutoff_func: cos
    edge_sh_normalization: component
    edge_sh_normalize: true
    ######## Irreps set 1 (crystal): ################
    irreps_edge_sh: 0e + 1o + 2e + 3o + 4e + 5o
    irreps_node_features: 64x0e+64x0o+32x1o+16x1e+12x2o+25x2e+18x3o+9x3e+4x4o+9x4e+4x5o+4x5e+2x6e
    num_layers: 3
    num_radial: 64
    num_types: 96
    rbf_func: bessel
    set_features: true
    radial_MLP: [64, 64]
    use_corr_prod: False
    correlation: 2
    num_hidden_features: 16
    use_kan: False
    radius_scale: 1.01
    build_internal_graph: False
    use_gradient_checkpointing: False

setup:
  GNN_Net: HamGNNpre
  accelerator: null
  ignore_warnings: true
  checkpoint_path: null # Path to the model weights file
  load_from_checkpoint: False
  resume: False
  num_gpus: 1 # null: use cpu; [i]: use the ith GPU device
  precision: 32
  property: hamiltonian
  stage: fit # fit: training; test: inference

[QuantumLab-ZY]

Hi @TiacWei,

Could you please check the size of the generated graph_data.npz first?

ls -lh graph_data.npz

If possible, also check the number of graphs inside:

import numpy as np
with np.load("graph_data.npz", allow_pickle=True) as data:
    graph = data["graph"].item() if "graph" in data else data
    print("keys:", data.files)
    print("number of graphs:", len(graph))

This will help us determine whether the file is empty/invalid.

[QuantumLab-ZY]

In addition, to locate the exact failure point in graph_data_gen_abacus.py, could you temporarily disable a few broad try/except blocks and rerun the graph generation script?

Currently, some internal errors are caught and the script only reports a later secondary error such as H is None, which hides the original cause.

In particular, please temporarily comment out or remove the try/except wrappers around:

1. Reading Hamiltonian / overlap matrices and building graph_h0, graph_h, graph_s
2. Calling generate_hamiltonian_and_overlap(...)
3. The internal try/except inside generate_hamiltonian_and_overlap(...)

Then rerun:

python ~/software/HamGNN-main/DFT_interfaces/abacus/graph_data_gen_abacus.py \
  --data-dirs ./ \
  --graph-data-folder ./ \
  --output-format npz

Please paste the new full traceback. This should let the script fail at the real source, for example a basis-dimension mismatch, missing sparse matrix entry, graph expansion mismatch, or an unexpected ABACUS sparse format issue, instead of failing later at:

torch.FloatTensor(H)

where H has already become None.

[TiacWei]

Hi,

-rw-r--r-- 1 user01 users 2.8M May 23 21:00 graph_data.npz
keys: ['graph']
number of graphs: 1

I also tested the graph_data.npz file from https://doi.org/10.5281/zenodo.11204268. And I still got the same error Total length of Dataloader across ranks is zero. Please make sure that it returns at least 1 batch. Is this due to an installation problem?

[QuantumLab-ZY]

Hi @TiacWei,

The latest error is likely not caused by the generated graph_data.npz itself or by installation.
Your graph_data.npz contains only 1 graph:

keys: ['graph']
number of graphs: 1

With your current split:

train_ratio: 0.8
val_ratio: 0.1
test_ratio: 0.1
stage: fit
batch_size: 1

HamGNN computes the split in hamgnn/data/graph_data.py as:

num_train = round(train_ratio * dataset_length)
num_val = round(val_ratio * dataset_length)
train_indices = indices[:num_train]
val_indices = indices[num_train:num_train+num_val]
test_indices = indices[num_train+num_val:]

For dataset_length = 1, this gives:

train: 1
val: 0
test: 0

Therefore, the validation and test dataloaders are empty. During stage: fit, PyTorch Lightning still requests the validation dataloader, and HamGNN also monitors validation/total_loss for checkpointing. This can lead to:

Total length of Dataloader across ranks is zero. Please make sure that it returns at least 1 batch.

So this is expected when running stage: fit with only one graph and a train/validation/test split.

For training, please generate graph data from multiple SCF calculations so that the train, validation, and test splits all contain at least one graph. For example, use multiple structures or pass multiple ABACUS output directories through --data-dirs.

With only one graph, a meaningful train/validation/test split is not possible. If you only want to check whether the file can be loaded, use a larger dataset for stage: fit, or run an inference/test workflow with a proper checkpoint instead of trying to train from a single graph.

Regarding the earlier H is None issue, changing the .orb basis file and matching NAO_MAX is consistent with a basis-dimension mismatch. The current dataloader error is a separate downstream issue.