HTR CI build failure for Legion

[seemamirch]

Legion master fails with the following

/usr/WS1/stanf_ci/psaap-ci/artifacts/2664035/legion/runtime/realm/cuda/cuda_hook.cc.o: in function cuhook_start_task': cuda_hook.cc:(.text+0x2e8): call to __tls_init' lacks nop, can't restore toc; (plt call stub)
/usr/WS1/stanf_ci/psaap-ci/artifacts/2664035/legion/runtime/realm/cuda/cuda_hook.cc.o: in function cuhook_end_task': cuda_hook.cc:(.text+0x12f8): call to __tls_init' lacks nop, can't restore toc; (plt call stub)
cuda_hook.cc:(.text+0x138c): call to __tls_init' lacks nop, can't restore toc; (plt call stub) /usr/WS1/stanf_ci/psaap-ci/artifacts/2664035/legion/runtime/realm/cuda/cuda_hook.cc.o: in function Realm::Cuda::cuhook_stream_callback(Realm::Cuda::CUHookSymbolEnum, int, void*)':
cuda_hook.cc:(.text+0x3c88): call to `__tls_init' lacks nop, can't restore toc; (plt call stub)
/usr/tce/packages/gcc/gcc-11.2.1/rh/usr/bin/../libexec/gcc/ppc64le-redhat-linux/11/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make: *** [/usr/WS1/stanf_ci/psaap-ci/artifacts/2664035/legion/runtime/runtime.mk:1518: librealm_cuhook.so] Error 1

[elliottslaughter]

Related to https://gitlab.com/StanfordLegion/legion/-/merge_requests/1780

@Jacobfaib can we revert that until we find a proper fix?

[elliottslaughter]

Note this is on a PPC64le machine. @seemamirch told me she had tried multiple GCC versions, going up through at least 11.

[Jacobfaib]

Looking into a fix, though note that this is 100% a compiler bug.

[Jacobfaib]

@seemamirch is this error isolated to just cuhook?

[elliottslaughter]

Echoing what we said in the Legion meeting today, the concern is that even if the error is isolated for now, we may trip over this potentially in the future. A seemingly unrelated Realm change could retrigger the compiler bug. Especially if it's still broken in recent GCC versions.

My understanding is that @seemamirch went up to at least GCC 12, but she can confirm the exact versions.

[seemamirch]

@seemamirch is this error isolated to just cuhook?

It appears to be related to 'thread_local' - I manually replaced it with __thread and it compiled. That was only needed in cuda_hook.cc.
Looking at the CI build logs in detail - that file is compiled using nvcc - the module loaded is cuda/11.8.0

[elliottslaughter]

Is there a newer nvcc on this machine we could switch to?

[seemamirch]

I tried with a newer version i.e. cuda/12.2.2 (newest one) and it failed

[elliottslaughter]

I wonder if you need to use nvcc -ccbin to specify the GCC version to use? Otherwise you may not actually be using the GCC version you think you are using to compile the host code.

[Jacobfaib]

Additionally, from some googling, it seems that explicitly adding -fPIC to host flags solved the problem for some people.

[seemamirch]

This is the full compile line - from the build logs

/usr/tce/packages/cuda/cuda-11.8.0/bin/nvcc --compiler-options '-fPIC' -o /usr/WS1/stanf_ci/psaap-ci/artifacts/2667133/legion/runtime/realm/cuda/cuda_hook.cc.d -M -MT /usr/WS1/stanf_ci/psaap-ci/artifacts/2667133/legion/runtime/realm/cuda/cuda_hook.cc.o /usr/WS1/stanf_ci/psaap-ci/artifacts/2667133/legion/runtime/realm/cuda/cuda_hook.cc -std=c++17 -Xcompiler -fPIC -ccbin mpicxx -g -O0 -DVOLTA_ARCH -gencode arch=compute_70,code=sm_70 -gencode arch=compute_70,code=compute_70 -Xcudafe --diag_suppress=boolean_controlling_expr_is_constant -std=c++17 -I/usr/WS1/stanf_ci/psaap-ci/artifacts/2667133/legion/bindings/regent -I/usr/WS1/stanf_ci/psaap-ci/artifacts/2667133/legion/runtime -I/usr/WS1/stanf_ci/psaap-ci/artifacts/2667133/legion/runtime/mappers -I/usr/tce/packages/cuda/cuda-11.8.0/include -I/usr/tce/packages/cuda/cuda-11.8.0/extras/CUPTI/include -I/usr/WS1/stanf_ci/psaap-ci/codes/legion-latest/gpu/language/gasnet/release/include -I/usr/WS1/stanf_ci/psaap-ci/codes/legion-latest/gpu/language/gasnet/release/include/ibv-conduit

[elliottslaughter]

Since you're setting -ccbin to mpicxx, can you confirm what compiler version that is?

mpicxx -version

[seemamirch]

when gcc/12.2.1 was loaded
[mirchandaney1@lassen709:build]$ mpicxx --version
g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-4)

when gcc/11.2.1 was loaded
[mirchandaney1@lassen709:build]$ mpicxx --version
g++ (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)

[elliottslaughter]

If someone can make an isolated file that reproduces the issue, maybe we can check out different compiler versions with Godbolt?

I was trying to do it today, and I got the file minimized far enough to compile on its own, but then Godbolt's version of GCC for PPC64le didn't fail to build it. Maybe there's some subtle interaction with the rest of the Realm and/or CUDA code bases that matters?