Skip to content

gromacs: add ROCm support and enhance heFFTe compatibility - #6094

Open
cedricjourdain wants to merge 1 commit into
spack:developfrom
cedricjourdain:gromacs-add-ROCm-support-enhance-other-gpu-build
Open

gromacs: add ROCm support and enhance heFFTe compatibility#6094
cedricjourdain wants to merge 1 commit into
spack:developfrom
cedricjourdain:gromacs-add-ROCm-support-enhance-other-gpu-build

Conversation

@cedricjourdain

@cedricjourdain cedricjourdain commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Add support for ROCm, improve HeFFTe’s compatibility with ROCm, improve the management of the sycl compilation (using the VkFFT FFT backend) and correctly manage CRAY MPICH linking dependencies

$ spack --version
1.2.2 (3e19345b6e12f5ff1b874f4059622fc6a1fd804a)

Tested 3 differents version with Cray Compiler :

  • gromacs@2026.3cp2kheffte+mpi+openmp+rocm build_type=Release amdgpu_target=gfx90a ^cray-fftw+mpi+openmp ^cray-libsci+mpi+openmp
  • gromacs@2026.3~cp2k+heffte+mpi+openmp+rocm build_type=Release amdgpu_target=gfx90a ^cray-fftw+mpi+openmp ^cray-libsci+mpi+openmp ^heffte@2.4.1+fftw+rocm amdgpu_target=gfx90a
  • gromacs@2026.3cp2kheffte+mpi+openmp+rocm+sycl build_type=Release amdgpu_target=gfx90a ^cray-fftw+mpi+openmp ^cray-libsci+mpi+openmp ^hipsycl@25.10.0+use_llvm_amdgpu amdgpu_target=gfx90a
Command line:
  gmx_mpi mdrun -pin off -notunepme -maxh 0.9 -ntomp 16 -nsteps 2000 -noconfout -nb gpu -v -s systWT250nsGr2018woEne.tpr -g Gromacs_BSG_2mpi-16omp-8gpu.log -resethway

GROMACS version:     2026.3-spack
Precision:           mixed
Memory model:        64 bit
MPI library:         MPI (GPU-aware: HIP)
MPI version:         MPI VERSION    : CRAY MPICH version 9.0.1.498 (ANL base 4.1.2) MPI BUILD INFO : Wed Jul 16  9:35 2025 (git hash 0848216)
OpenMP support:      enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support:         HIP
NBNxM GPU setup:     super-cluster 2x2x2 / cluster 8 (cluster-pair splitting off)
SIMD instructions:   AVX2_256
CPU FFT library:     commercial-fftw-3.3.10-sse2-avx-avx2-avx2_128
GPU FFT library:     rocFFT 1.0.32
Multi-GPU FFT:       none

Command line:
  gmx_mpi mdrun -pin off -notunepme -maxh 0.9 -ntomp 16 -nsteps 2000 -noconfout -nb gpu -v -s systWT250nsGr2018woEne.tpr -g Gromacs_BSG_2mpi-16omp-8gpu.log -resethway

GROMACS version:     2026.3-spack
Precision:           mixed
Memory model:        64 bit
MPI library:         MPI (GPU-aware: HIP)
MPI version:         MPI VERSION    : CRAY MPICH version 9.0.1.498 (ANL base 4.1.2) MPI BUILD INFO : Wed Jul 16  9:35 2025 (git hash 0848216)
OpenMP support:      enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support:         HIP
NBNxM GPU setup:     super-cluster 2x2x2 / cluster 8 (cluster-pair splitting off)
SIMD instructions:   AVX2_256
CPU FFT library:     commercial-fftw-3.3.10-sse2-avx-avx2-avx2_128
GPU FFT library:     rocFFT 1.0.32
Multi-GPU FFT:       HeFFTe 2.4.1 with rocFFT backend

Command line:
  gmx_mpi mdrun -pin off -notunepme -maxh 0.9 -ntomp 16 -nsteps 2000 -noconfout -nb gpu -v -s systWT250nsGr2018woEne.tpr -g Gromacs_BSG_2mpi-16omp-8gpu.log -resethway

GROMACS version:     2026.3-spack
Precision:           mixed
Memory model:        64 bit
MPI library:         MPI (GPU-aware: HIP)
MPI version:         MPI VERSION    : CRAY MPICH version 9.0.1.498 (ANL base 4.1.2) MPI BUILD INFO : Wed Jul 16  9:35 2025 (git hash 0848216)
OpenMP support:      enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support:         SYCL (AdaptiveCpp)
NBNxM GPU setup:     super-cluster 2x2x2 / cluster 8 (cluster-pair splitting off)
SIMD instructions:   AVX2_256
CPU FFT library:     commercial-fftw-3.3.10-sse2-avx-avx2-avx2_128
GPU FFT library:     VkFFT internal (1.3.1) with HIP backend
Multi-GPU FFT:       none

@spackbot-triage spackbot-triage Bot added new-version Modifications to packages' `depends_on()` directives dependencies Modifications with a `depends_on()` directive update-package Modifications to packages in the repository conflicts Modifications with a `conflicts()` directive labels Aug 14, 2026
al42and
al42and reviewed Aug 14, 2026
View reviewed changes
Comment thread repos/spack_repo/builtin/packages/gromacs/package.py
if self.spec.satisfies("+rocm") and not self.spec.satisfies("+sycl"):
options.append("-DGMX_GPU_FFT_LIBRARY=rocFFT")
options.append(f"-DCMAKE_CXX_FLAGS=-I{self.spec['rocfft'].headers.directories[0]}")
options.append("-DGMX_BUILD_OWN_FFTW=OFF")

@al42and al42and Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GMX_BUILD_OWN_FFTW is CPU-only and should be orthogonal to GPU FFT library choice

Comment thread repos/spack_repo/builtin/packages/gromacs/package.py
options.append("-DHeffte_ENABLE_ROCM:BOOL=ON")
options.append(f"-DHeffte_ROCM_ROOT={self.spec['hip'].prefix}")

# Use rocFFT backend to perform FFT on GPU only for pure ROCm builds

@al42and al42and Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. At least with ROCm 7.0 and earlier on MI300A and MI250X, VkFFT tends to perform somewhat better in our tests (on both SYCL/ACpp and native HIP). Is your experience different? In any case, I think it's better handle +rocm the same, regardless of whether it uses ACpp or HIP directly, since the FFT part is identical.
  2. This should depend on heffte choice: on AMD GPUs (with SYCL/ACpp and native HIP both), we cannot use HeFFTe with VkFFT, only with rocFFT.

Comment thread repos/spack_repo/builtin/packages/gromacs/package.py
@@ -96,6 +99,9 @@ class Gromacs(CMakePackage, CudaPackage):
)
depends_on("heffte +cuda", when="+heffte +cuda")
depends_on("heffte +sycl", when="+heffte +sycl")

@al42and al42and Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we build +sycl+rocm, we need heffte +rocm, so something like this (untested):

Suggested change
depends_on("heffte +sycl", when="+heffte +sycl")
depends_on("heffte +sycl", when="+heffte +sycl ~rocm")
depends_on("heffte +rocm", when="+heffte +sycl +rocm")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eirrgang eirrgang Awaiting requested review from eirrgang
@junghans junghans Awaiting requested review from junghans
@mabraham mabraham Awaiting requested review from mabraham
1 more reviewer
@al42and al42and al42and left review comments
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

conflicts Modifications with a `conflicts()` directive dependencies Modifications with a `depends_on()` directive new-version Modifications to packages' `depends_on()` directives update-package Modifications to packages in the repository

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@cedricjourdain @al42and @CedricJour