Additional backends

[tzanio]

• Improve OCCA backend
• Add MFEM backend — how to support backends that don’t support JIT and don’t run on the host?
• Add MAGMA backend?
• Add OpenMP 4.5 backend?
• Add pure CUDA backend?
• Add HIP backend?

[jedbrown]

With the announcement that OLCF Frontier will be AMD CPU/GPU, we should try to get it into our workflow. We can use HIP (an open source CUDA-like model that can compile to CUDA and ROCm) which can be almost automatically produced from CUDA (using hipify-clang) or OpenMP-5 offload as on-node programming models. Note that HIP does not currently support run-time compilation.

HIP nominally compiles to CUDA with negligible overhead, but the toolchain needs to be installed to do so.

[tcew]

OCCA:HIP supports run-time compilation.

[jeremylt]

Our OCCA backend is in serious need of a performance overhaul, so it would be great if we can also include OCCA:HIP.

[tcew]

See: https://github.com/libocca/occa/blob/022b76829d43cbe20b719e6d5a54c9aff8fa178c/src/modes/hip/device.cpp#L230

[jedbrown]

Yes, I don't think anything special needs to be done for /gpu/occa/hip versus /gpu/occa/cuda, though the OCCA backend needs attention. My comment on run-time compilation was with regard to @YohannDudouit's native CUDA implementation.

I'm also curious about observed differences in performance characteristics between the Radeon Instinct and V100.

[tcew]

You should follow up with Noel Chalmers. I believe he has run libP experiments with the Radeon Instinct.

[jedbrown]

Thanks. @noelchalmers, can you share any experiments?

[noelchalmers]

Hi everyone. I'll try and chip in what I know for some of the points in this thread:

• In addition to hipify-clang, which ports existing CUDA code by actually looking at the code's semantics, there is also hipify-perl which is a simple script which can convert CUDA codes to HIP, and at least warn about sections it is unable to translate.

• HIP does indeed support runtime compilation in the same way CUDA does. OCCA uses analogous API calls for its runtime compilation of CUDA and HIP. I know the documentation of what is/is not currently in the HIP API is a bit sparse at the moment. The HIP Porting Guide is a good resource for the moment.

• As for V100 vs Radeon Instinct performance, in micro-benchmarking we've been seeing bandwidth numbers in the 800-900 GB/s range for the MI-60s and similar GFLOP numbers to the PCIe V100s.

• I don't have any readily available performance numbers for any CEED-relevant benchmarking. My plan is to resurrect the bake-off problems in libp and do some performance analysis to get a better sense of what the Radeons can do compared to the V100s. Libp's kernels rely heavily on things like shared memory bandwidth and cache performance so it will be a good exercise in finding out how portable they are to Radeon.