Regent/Terra are incompatible with CUDA 12.8

[rohany]

I tried to build some regent applications with CUDA 12.8 and see the following error:

(legate) root@eos0143:/opt/legate/legion/language# LEGION_BACKTRACE_ON_ERROR=1 OMPI_ALLOW_RUN_AS_ROOT=1 OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 mpirun -n 1 --bind-to none entrypoint.sh ./stencil -ll:gpu 1
LD_PRELOAD: /opt/conda/envs/legate/lib/libtcmalloc.so
error in cuModuleLoadData (CUDA_ERROR_INVALID_PTX): a PTX JIT compilation failed

Investating further, I found that Regent's generated PTX contains this snippet:

;
.extern .func  (.param .b32 func_retval0) llvm.nvvm.tanh.approx.f32
(
        .param .b32 llvm.nvvm.tanh.approx.f32_param_0
)

which is invalid PTX:

(legate) root@eos0114:/opt/legate/legion/language# /usr/local/cuda/bin/ptxas stencil.ptx
ptxas stencil.ptx, line 126; fatal   : Parsing error near '.nvvm': syntax error
ptxas fatal   : Ptx assembly aborted due to errors

The issue is that this code comes from CUDA's shipped libdevice.10.bc, so it's unclear what we're supposed to do here. Adding a dead-code-elimination pass to Terra's PTX generation would be a workaround for this issue as long as an application does not need to use tanh (https://github.com/terralang/terra/blob/master/src/tcuda.cpp#L87).

I can confirm that this issue does not arise on CUDA 12.5.1, and it's on my backlog to get a tighter bound for the last working CUDA version.

[jurahul]

My guess is that this issue is a combination of CUDA 12.8 libdevice using a new LLVM intrinsic llvm.nvvm.tanh.approx.f32 which the upstream LLVM NVPTX does not support and the Regent/Terra code not using llvm::Linker::Flags::LinkOnlyNeeded flag when linking the libdevice code here https://github.com/terralang/terra/blob/master/src/tcuda.cpp#L140

This means the llvm.nvvm.tanh.approx.f32 intrinsic gets sucked into the module unnecessarily and then later in NVPTX backend treats it as any regular extern function (since LLVM upstream does not have this intrinsic).

A simple fix might be to use the llvm::Linker::Flags::LinkOnlyNeeded flag when linking libdevice into the module, so that unused functions from libdevice are not unnecessarily linked into the module. This may not only fix this issue but also improve compile time by eliminating unneeded cruft that the compiler has to cleanup later on.

For reference, XLA code that does similar things:

https://github.com/openxla/xla/blob/main/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc#L168

and Triton code:
https://github.com/triton-lang/triton/blob/main/python/src/llvm.cc#L476

[muraj]

@lightsighter this doesn't look like a realm specific bug, more like a regent/terra bug? Do we want to transfer this one back?

[rohany]

yes, this has nothing to do with realm.

[elliottslaughter]

I have transferred this one back to Legion.