QMCPACK Issue Draft: Complex CUDA/NCCL AFQMC Factorized SparseTensor Crash

[DingyangLyu]

Issue Body

Summary

I hit a reproducible failure in QMCPACK 4.3.0 when running a complex-valued AFQMC calculation with an HDF5 Factorized Hamiltonian on the CUDA/NCCL build.

The same small complex AFQMC input runs with the complex CPU/PHDF5 build. The complex GPU/NCCL build also runs an HDF5 THC Hamiltonian input. The failing path appears specific to the complex CUDA/HIP SparseTensor / Factorized Hamiltonian path.

I have a local workaround patch that makes the one-rank complex GPU/NCCL Factorized/Cholesky smoke test complete successfully. I am opening this issue first to confirm whether the maintainers prefer this conservative host-scratch fallback, or a proper device sparse implementation.

Environment

• QMCPACK: 4.3.0
• Upstream commit tested: bb7eede051f98ec03296664b304982e655f960c4
• Build mode:
  • QMC_COMPLEX=ON
  • BUILD_AFQMC=ON
  • ENABLE_PHDF5=ON
  • ENABLE_CUDA=ON
  • BUILD_AFQMC_WITH_NCCL=ON
• GPU: NVIDIA A100-SXM4-80GB
• CUDA: 12.9
• NCCL: 2.25
• MPI: OpenMPI via Conda environment

Minimal Run Shape

Input shape:

<simulation method="afqmc">
  <project id="qmc_short_chol" series="0"/>

  <AFQMCInfo name="info0">
    <parameter name="NMO">26</parameter>
    <parameter name="NAEA">4</parameter>
    <parameter name="NAEB">4</parameter>
  </AFQMCInfo>

  <Hamiltonian name="ham0" type="Factorized" info="info0">
    <parameter name="filetype">hdf5</parameter>
    <parameter name="filename">../choldump.h5</parameter>
  </Hamiltonian>

  <Wavefunction name="wfn0" type="MSD" info="info0">
    <parameter name="filetype">ascii</parameter>
    <parameter name="filename">../wfn.dat</parameter>
  </Wavefunction>

  <WalkerSet name="wset0">
    <parameter name="walker_type">closed</parameter>
  </WalkerSet>

  <Propagator name="prop0" info="info0"/>

  <execute wset="wset0" ham="ham0" wfn="wfn0" prop="prop0" info="info0">
    <parameter name="timestep">0.01</parameter>
    <parameter name="blocks">1</parameter>
    <parameter name="steps">1</parameter>
    <parameter name="nWalkers">2</parameter>
  </execute>
</simulation>

Command:

CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=1 \
mpiexec -n 1 --bind-to none \
  /path/to/qmcpack qmc_short_chol.in.xml --gpu

Observed Behavior

Before the local patch, the run failed in the complex CUDA/NCCL build during AFQMC propagator initialization / mean-field subtraction.

The crash mapped into the SparseTensor Factorized path, including:

• AFQMC/HamiltonianOperations/SparseTensor.hpp
• SparseTensor::vbias
• SparseTensor::vHS
• NOMSD::vMF
• PropagatorFactory::buildAFQMCPropagator

After patching only vHS and vbias, the code progressed further but then failed in the energy path with a host/device GEMV mismatch:

types: (gemv) T: std::complex<double>
  ptrA: device::device_pointer<std::complex<double>>
  ptrB: const std::complex<double>*
  ptrC: device::device_pointer<std::complex<double>>
Error: Calling qmc_cuda::gemv catch all.

This suggests that the Factorized SparseTensor path is passing CUDA device dense arrays into host sparse/dense operations, while at least some sparse tensors and vectors are still host/shared-memory objects.

Expected Behavior

A one-rank complex CUDA/NCCL AFQMC run with an HDF5 Factorized Hamiltonian should either:

1. complete correctly on GPU, or
2. use an explicit supported fallback for this sparse host-backed path, rather than crashing or entering unsupported host/device mixed operations.

Root Cause Hypothesis

In SparseTensor.hpp, the Factorized path appears to use host/shared CSR data structures such as Spvn_view and SpvnT_view, but vHS, vbias, and part of energy can receive CUDA device dense buffers.

The old code then constructs host-looking array refs or calls generic ma::product on a mix like:

• host/shared sparse matrix
• device dense matrix
• host vector or device output

That mix is not supported by the current CUDA matrix dispatch and can segfault or fall into the CUDA GEMV catch-all.

Local Workaround Patch

The local workaround is deliberately conservative:

• Add #include <algorithm>.
• Under ENABLE_CUDA || BUILD_AFQMC_HIP, in SparseTensor::vHS:
  • copy dense input X from device to a host scratch buffer,
  • copy v to host scratch only when c != 0,
  • run the existing host sparse multiply against Spvn_view,
  • copy the local result segment back to the original output buffer.
• Do the same in SparseTensor::vbias for G and SpvnT_view[k].
• In SparseTensor::energy:
  • copy Gc to host scratch,
  • compute into a host E scratch using the existing host path,
  • copy E back to the original output buffer before returning.

This is not meant to be the fastest final solution. It is a correctness-first fallback for the host-backed sparse Factorized path. A proper device sparse implementation may be preferable if maintainers want this path to stay fully GPU-resident.

Validation After Local Patch

With the local workaround patch and a rebuilt complex CUDA/NCCL AFQMC binary:

Complex AFQMC CUDA/NCCL HDF5 Factorized/Cholesky, 1 MPI rank: PASS
Complex AFQMC CUDA/NCCL HDF5 THC, 1 MPI rank: PASS
Complex AFQMC CPU/PHDF5 HDF5 Factorized/Cholesky, 1 MPI rank: PASS
Real AFQMC CUDA/NCCL HDF5 Factorized, 1 MPI rank: PASS
Regular VMC CPU smoke: PASS
Regular VMC GPU smoke: PASS

The patched complex GPU Factorized run reaches:

Energy of starting determinant:
  - Total energy    : (-10.3197575584,0)
  - One-body energy : (-10.8097667596,0)
  - Coulomb energy  : (1.46758186973,0)
  - Exchange energy : (-0.977572668585,0)

QMCPACK execution completed successfully

Separate Upstream Limitation

This issue is separate from the existing multi-rank Factorized limitation:

Error: Distributed Factorized hamiltonian not yet implemented.

That still occurs for a single complex Factorized input launched with multiple MPI ranks. The local fix above only addresses the one-rank CUDA/NCCL host/device SparseTensor failure.

Question For Maintainers

Would a PR with the conservative host-scratch fallback in SparseTensor::{energy,vHS,vbias} be acceptable, or would you prefer a different implementation strategy, such as:

• adding a true device sparse Factorized path,
• explicitly forcing this path to CPU/host execution,
• or rejecting unsupported host/device combinations earlier with a clearer error?

I can prepare a PR with the patch and a small regression test if this direction is acceptable.

Where To Submit

Submit this as a new issue at:

https://github.com/QMCPACK/qmcpack/issues

Repository confirmed from the local upstream remote:

https://github.com/QMCPACK/qmcpack.git

[prckent]

Thanks for reporting. Do all the existing AFQMC tests included with 4.3.0 work with your installation before this change? Have you noticed any other issues with AFQMC builds?