Enabling Generic HIP Builds

[Spaarsh]

Hello team,

I am Spaarsh¹ and I am trying to enable HIP/ROCm in the debian package for dbcsr under my GSoC'25 work as a contributor at Debian³. Cordell Bloor (@cgmb⁴) is my GSoC mentor.

The package can be easily built for a GPU but the current build process requires us to run the entire build multiple times for all GPU architectures (which, in case of AMD, are currently gfx906, gfx908 and gfx90a). To prevent this, cmake allows to use a flag called CMAKE_HIP_ARCHITECTURES⁵ that allows us to create a fat binary (a generic build, loosely speaking) that is compatible for multiple architectures simultaneously. But it has been manually disabled in the build process as seen here⁶. Due to this reason we need to build a separate binary for each supported architecture (libdbcsr-rocm-dev-gfxXXXX) which bloats our source package (and not to mention the increased build time).

Upon further investigation, I came to realize that the reason for the aforementioned disable was due to the custom kernel building⁷ that dbcsr does. This cannot be manipulated via cmake since those sections are built via python scripts. A non-trivial amount of changes shall be required to achieve a full generic build, if it is possible at all.

However, after analyzing the binaries built for different packages, I understand that only libdbcsr.a is dependent on the architecture (please correct me if I am wrong).

With this understanding, I propose the following change, which would result in a single binary file that shall be able to support multiple GPU architectures with lesser bloating of our debian package:
Currently the build process uses the WITH_GPU flag to build the package. Assuming I am not mistaken about only libdbcsr.a being architecture-dependent, a way to compile a libdbcsr.arch.a for each architecture would enable support for all architectures via a single binary. This can be achieved by introducing a new optional flag, say HIP_MULTI_ARCHITECTURE, in the main CMakeLists.txt⁸ file that can accept multiple architectures/target GPUs as input. In the CMakeLists.txt we parse over the inputs in the flag and loop over the libdbcsr.a compilation, changing the WITH_GPU for each iteration. This shall result in a libdbcsr.arch.a file after each iteration. Finally, all these files can be a part of the libdbcsr-rocm-dev package (you can find the MR here⁹. This would also need us to make the WITH_GPU flag optional. A new check that ensures that at least WITH_GPU or HIP_MULTI_ARCHITECTURE is defined shall be added.

The proposed change will allow us to build a generic HIP/ROCm dbcsr package for the dbcsr debian package.

Regards,
Spaarsh

1. https://salsa.debian.org/Spaarsh/
2. https://mentors.debian.net/packages/uploader/spaarshthakkar11010@gmail.com/
3. https://summerofcode.withgoogle.com/programs/2025/projects/9s4jUjV0
4. https://salsa.debian.org/cgmb
5. https://cmake.org/cmake/help/latest/variable/CMAKE_HIP_ARCHITECTURES.html#variable:CMAKE_HIP_ARCHITECTURES
6. https://github.com/cp2k/dbcsr/blob/0175815ab79318ecdfef87e208359ab984453158/CMakeLists.txt#L277-L281]
7. https://github.com/cp2k/dbcsr/blob/0175815ab79318ecdfef87e208359ab984453158/src/acc/libsmm_acc/CMakeLists.txt
8. https://github.com/cp2k/dbcsr/blob/0175815ab79318ecdfef87e208359ab984453158/CMakeLists.txt
9. https://salsa.debian.org/debichem-team/dbcsr/-/merge_requests/1

[hfp]

Thank you for sharing, we/I have yet to digest your proposal. Regarding generic package, we have an OpenCL based flavor which works just fine on all major GPU architectures like single binary (same goes for CP2K). The reason for non-generic build with CUDA/HIP are specialized kernels and in the past no fallback to generic/untuned parameters. This is now solved for CUDA/HIP but may need some extra work to expose a generic GPU package. Worst case for the latter is to always fall-back even worse to have done data transfers before entering the fallback. For OpenCL, this was never a problem since tuned parameters are entirely optional plus it can include all tuned parameters available with one binary plus having a GPU based fallback for parameters. Questioning your desire in general, DBCSR's GPU support relies on reasonable support for FP64 aka double-precision which may be badly supported by client GPUs (very unfavorable FLOPS ratio vs single-precision let alone low precision). With unfavorable I mean the GPU may yield less FLOPS then the aggregated CPU package (all cores). Even with FP64, GPU acceleration can fall short just to mention it.

[hfp]

For example, cmake -DUSE_ACCEL=opencl just works and may only depend on Khronos ICD loader which then uses whatever driver is available. If you had CUDA, it would not even need Khronos ICD loader but then depend on CUDA installation. OpenCL (C-)headers and the ICD loader are standard packages on Debian.

[Spaarsh]

Hello,

My proposal isn't aimed towards making a truly generic package (maybe I should change the discussion title?). What I am suggesting is to allow the building of a package that contains the dbcsr library for multiple supported architectures.

So the package can contain, at maximum, only three libdbcsr.arch.a libraries since we can currently only support three architectures.

I am also planning to make a PoC PR that implements my idea. If you have any other questions, please let me know!
Thanks.

[hfp]

How do you envision to select an architecture at runtime or is your idea not about selecting an implementation at runtime?

I am not against your idea. In fact, I implemented this the OpenCL backend like supporting multiple architectures by the means of loading tuned parameters according to the device name (with some fuzzy-ness). The CUDA backend was before the OpenCL backend and there was little need to support such scenario. DBCSR was envisioned to be compiled towards a dedicated HPC system with a well-defined and known (CUDA-)architecture (and by extension HIP-arch); kernels were instantiated and compiled ahead of time. In contrast, the OpenCL backend uses JIT compilation (although OpenCL may be compiled ahead of time as well), hence supporting multiple "architectures" (like different tuned parameters) is about loading the "best" set of tuned parameters according to the device's name. The OpenCL backend also has default parameters such that missing tuned parameters still allow to come up with a nice GPU-kernel.

Anyway, is your idea meant to support GPUs by providing a single upstream-package? If not, what prevents you to compile for a single (HIP-)architecture if you only want to support say three architectures at most like the set of target systems is known ahead of time?

[Spaarsh]

Anyway, is your idea meant to support GPUs by providing a single upstream-package? If not, what prevents you to compile for a single (HIP-)architecture if you only want to support say three architectures at most like the set of target systems is known ahead of time?

Yes, my goal is to make a single upstream package. As @alazzaro correctly pointed out, in #933 (comment), my goal is to have lesser compile time.

At one point, I even considered running the entire build for all architectures, then shipping only the unique files (the .so files) in a single binary. But that method is not scalable if dbcsr grows to support more architectures.

How do you envision to select an architecture at runtime or is your idea not about selecting an implementation at runtime?

I do not have plans to have runtime selection. Not right now at least. And runtime selection does not work for debian apt since a package developer shall build the package files and upload it to the main server. These files are meant to be usable as is. That said, there are discussions in my team that we may use virtual-package techniques to decide which .arch.a file to install on the user's device. But that shall be handled by debian/ files.

[oschuett]

AFAIK, the kernel parameters have not changed much with recent GPU upgrades. So, a pragmatic solution would be to just use the latest parameters (H100 and Mi300) and call it a day.

[Spaarsh]

I am not aware if the kernel parameters have changed or not. But if the other developers agree with this, I am alright with the suggestion.

[alazzaro]

I see the point why you are asking this change: compilation time. And you are definitely right, we can reuse most (definitely all Fortran files, for instance) of the built files for different GPU arch. However, as you pointed out, it requires some refactoring of the build system.
This is what the cmake currently does. We pass WITH_GPU to set the GPU type. This is passed to https://github.com/cp2k/dbcsr/blob/develop/src/acc/libsmm_acc/generate_parameters.py, which then uses it to select the tuned parameters of the kernels. Then, we fetch the arch to be used.
This info is passed to the Jitter of the kernel.

I think the only file that requires a specific recompilation per each arch is https://github.com/cp2k/dbcsr/blob/develop/src/acc/cuda_hip/calculate_norms.cpp.

(side node: the current policy is to pass all macros to all files, so -DARCH_NUMBER is passed to all compiled files, even if it is not really used)

So, ideally, we can pass WITH_GPU="list of GPUs", then generate the kernels (i.e. the python script) per each arch in separate directories, build the Fortran files only once, compile calculate_norms per each arch (or we can make a fat binary), and then link per each arch and produce a libdbcsr_.a, as you are suggesting. This is valid only for the library. I don't want to make the same machinery for the executables (tests and examples), where only a single arch must be specified (then we will pick up the right library).

Is this what you are suggesting?

Given my current commitments, I can work on that in about ~1 month...

[Spaarsh]

Is this what you are suggesting?

Yes! This is exactly what I had in mind.

Given my current commitments, I can work on that in about ~1 month...

If @hfp finds these suggestions reasonable, I am willing to work on this. I understand that you are busy but could you just help me out in my PR in case I am stuck?

Thanks!

[Spaarsh]

Hello folks!

Any consensus on this issue?

[hfp]

I look forward to see your contribution! If I can help, I can also take a look at your PR or perhaps help if you run into an obstacle.

[Spaarsh]

Fantastic! I'll get onto this soon! Thanks!

[Spaarsh]

I have made the required changes and pushed them to my fork ¹.

However, for some reason, when I run make, the compiler is unable to find the hip required references. I will talk with people from rocm-team at Debian too. However, if anyone among you has any ideas, it would be really helpful!

Errors:

In file included from /dbcsr/src/acc/cuda_hip/acc_mem.cpp:13:
/dbcsr/src/acc/cuda_hip/acc_mem.cpp: In function 'int c_dbcsr_acc_host_mem_allocate(void**, size_t, void*)':
/dbcsr/src/acc/cuda_hip/../hip/acc_hip.h:23:16: error: 'hipHostAllocDefault' was not declared in this scope; did you mean 'hipHostMallocDefault'?
   23 | #define ACC(x) hip##x
      |                ^~~
/dbcsr/src/acc/cuda_hip/acc_mem.cpp:49:23: note: in expansion of macro 'ACC'
   49 |   unsigned int flag = ACC(HostAllocDefault);
      |                       ^~~

Upon manually fixing this one, I get many more.

PS: The CUDA side of things might be a bit shaky since I was more focused on ROCm/HIP. I'll see if I can fix that up by the time I go for the PR.

Thanks!

1. https://github.com/spaarsh/dbcsr