Realm CMake build system defaults to wrong compiler when compiling with Kokkos + ROCm

[rbberger]

Noticed while testing the GitLab master.

In the Spack package we're setting CMAKE_CXX_COMPILER and CMAKE_C_COMPILER to hipcc when building for ROCm. This isn't propagated to the Realm CMake build, which causes it to use some default compiler (like GCC). Since Kokkos (kokkos+rocm) injects hipcc/amdclang specific compilation flags, that then fails.

The suggested fix for this is to propagate CMAKE_CXX_COMPILER and CMAKE_C_COMPILER of Legion to the Realm build options.

The above is not true. See below. But the wrong compiler is being used or the amdclang compilation flags get applied to too many files.

https://gitlab.com/StanfordLegion/legion/-/blob/master/CMakeLists.txt#L834-867

[lightsighter]

Moved this issue over to the Realm repository. Assigning @muraj.

[rbberger]

Just for the record, it's in the main Legion CMakeLists.txt.

[lightsighter]

Ok, I'll move it back, same assignment though. ;)

[rbberger]

Another solution would be to make realm its own independent spack package and have legion just use it as dependency. Probably a large chunk of the current Legion Spack logic would go into realm, while the Legion package would shrink over time with version depreciation.

[elliottslaughter]

I think this is a fix worth having, because it will impact more than just Spack users. @muraj does it seem reasonable to you?

Whether you also split the Realm build in Spack is I guess up to you, but I wouldn't do it solely to fix this issue.

[muraj]

I had thought that because the c and cxx compiler was set in the parent project, the child projects would also get these variables through cpm no? AFAIK, cpm basically just calls add_subdirectory on the source dir. Anyway, yeah, I don't mind adding that option if it's really needed.

As to making realm a separate spack package, I was actually considering doing that or vcpkg or something anyway, but I'm not an spack expert and don't have the time to invest into it. If someone wants to build a branch for it, I'd be happy to review it :)

[rbberger]

As far as I understand it, CPM is a wrapper around FetchContent which will build dependencies with its own CMake process. So it doesn't inherit the context of the parent. That's why you would have to pass the additional options. As for the Spack package for realm, I'll see if I can make something since I need this to work for our ElCap hackathon one way or the other.

[rbberger]

Ok, after looking at it a bit more in detail, I'm taking back the statement that we're using hipcc via the Spack package. That's normally the case, but not with Legion. The build system actually does its own HIP detection and enables the HIP language support in CMake. The problem we're seeing here is that Realm actually has the same C, CXX compiler as Legion (which is gcc), adds a link dependency to Kokkos::kokkos, which injects its compilation flags for amdclang/hipcc. So all files that are in that target need to be of language HIP to use the right compiler or the CXX compiler has to be set to hipcc for Realm.

These parasitic compile options are a know issue with Kokkos. Either disable the compilation options completely (by forcefully wiping out INTERFACE_COMPILE_OPTIONS of the appropiate Kokkos target, or isolate the Kokkos files into their own target so it doesn't propagate beyond.

The hotfix is to compile everyhing in realm with hipcc.

[rbberger]

the current work on this is in https://gitlab.com/StanfordLegion/legion/-/merge_requests/1904

[muraj]

This should be resolved in merged in change:

StanfordLegion/realm#270

Will sync up with legion soon.