BinaryBH: Intel oneAPI 2025.3 compiler segfaults when compiling `BinaryBHLevel.cpp`

[mirenradia]

Summary

When I try to build the BinaryBH example with the Intel oneAPI 2025.3 toolkit, the compiler segfaults.

Steps to reproduce

I have been able to reproduce this both on Dawn and my own my laptop. The rough steps are below.

1. Load Intel oneAPI toolkit 2025.3 into your environment (e.g. load appropriate modules or source the appropriate setvars.sh script).
2. Clone the AMReX repository and the GRTeclyn repository in the same directory.
3. I had the following relevant contents in /path/to/amrex/Tools/GNUMake/Make.local-pre on Dawn (and something similar on my laptop) but I'm not sure this is necessary.

   ifeq ($(USE_SYCL),TRUE)       
     SYCL_AOT = TRUE
     AMREX_INTEL_ARCH = pvc
     SYCL_SUB_GROUP_SIZE = 16
     SYCL_AOT_GRF_MODE = Large
     SYCL_PARALLEL_LINK_JOBS = 24
   endif

4. Change directory into /path/to/GRTeclyn/Examples/BinaryBH.
5. Run

   make -j 4 USE_SYCL=TRUE
   

   You can change the number of make jobs as appropriate on your system.

Observed outcome

When the code tries to compile BinaryBHLevel.cpp, I get the following error:

Compiling BinaryBHLevel.cpp ...
mpiicpc -MMD -MP -gline-tables-only -fdebug-info-for-profiling -O3  -Wno-tautological-constant-compare -std=c++17 -fsycl -fsycl-device-code-split=per_kernel -mlong-double-64 -Xclang -mlong-double-64  -pthread -fsycl-targets=spir64_gen  -cxx=icpx   -DBL_USE_MPI -DAMREX_USE_MPI -DAMREX_USE_SYCL -DAMREX_USE_DPCPP -DAMREX_SYCL_SUB_GROUP_SIZE=16 -DBL_NO_FORT -DAMREX_USE_GPU -DBL_COALESCE_FABS -DAMREX_GPU_MAX_THREADS=256 -DAMREX_USE_GPU_RDC -DBL_SPACEDIM=3 -DAMREX_SPACEDIM=3 -DBL_FORT_USE_UNDERSCORE -DAMREX_FORT_USE_UNDERSCORE -DBL_Linux -DAMREX_Linux -DAMREX_PARTICLES -DNDEBUG -DAMREX_NO_PROBINIT -Itmp_build_dir/s/3d.sycl.MPI.EXE -I. -I/home/miren/NR/amrex/Src/Base -I/home/miren/NR/amrex/Src/Base/Parser -I/home/miren/NR/amrex/Src/Boundary -I/home/miren/NR/amrex/Src/AmrCore -I/home/miren/NR/amrex/Src/Amr -I/home/miren/NR/amrex/Src/Particle -I/home/miren/NR/GRTeclyn/Examples/BinaryBH -I/home/miren/NR/GRTeclyn/Source/AMRInterpolator -I/home/miren/NR/GRTeclyn/Source/BlackHoles -I/home/miren/NR/GRTeclyn/Source/CCZ4 -I/home/miren/NR/GRTeclyn/Source/Grids -I/home/miren/NR/GRTeclyn/Source/GRTeclynCore -I/home/miren/NR/GRTeclyn/Source/IO -I/home/miren/NR/GRTeclyn/Source/Maths -I/home/miren/NR/GRTeclyn/Source/Tagging   -c BinaryBHLevel.cpp -o tmp_build_dir/o/3d.sycl.MPI.EXE/BinaryBHLevel.o
icpx: warning: ignoring '-gline-tables-only' option as it is not currently supported for target 'spir64_gen-unknown-unknown'; only supported for host compilation [-Woption-ignored]
PLEASE append the compiler options "-save-temps -v", rebuild the application to get the full command which is failing and submit a bug report to https://software.intel.com/en-us/support/priority-support which includes the failing command, input files for the command and the crash backtrace (if any).
Stack dump:
0.      Program arguments: /opt/intel/oneapi/compiler/2025.3/bin/compiler/clang -cc1 -triple x86_64-unknown-linux-gnu -O3 -fsycl-is-host -include-internal-header /tmp/icpx-e344993104/BinaryBHLevel-header-4e92e7.h -dependency-filter /tmp/icpx-e344993104/BinaryBHLevel-header-4e92e7.h -fsycl-enable-int-header-diags -include-internal-footer /tmp/icpx-e344993104/BinaryBHLevel-footer-53cfcf.h -dependency-filter /tmp/icpx-e344993104/BinaryBHLevel-footer-53cfcf.h -sycl-std=2020 -fsycl-unique-prefix=uid853ff89a5c5186a0 -D__SYCL_ANY_DEVICE_HAS_ANY_ASPECT__=1 -emit-obj -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name BinaryBHLevel.cpp -mrelocation-model static -fveclib=SVML -faltmathlib=SVMLAltMathLibrary -mframe-pointer=none -fapprox-func -funsafe-math-optimizations -fno-signed-zeros -mreassociate -freciprocal-math -fdenormal-fp-math=preserve-sign,preserve-sign -ffp-contract=fast-honor-pragmas -fno-rounding-math -complex-range=promoted -mlong-double-64 -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -mllvm -x86-enable-unaligned-vector-move=true -tune-cpu generic --dependent-lib=sycl-devicelib-host -fdebug-info-for-profiling -debug-info-kind=line-tables-only -dwarf-version=4 -debugger-tuning=gdb -fdebug-compilation-dir=/home/miren/NR/GRTeclyn/Examples/BinaryBH -fcoverage-compilation-dir=/home/miren/NR/GRTeclyn/Examples/BinaryBH -fclang-abi-compat=17 -resource-dir /opt/intel/oneapi/compiler/2025.3/lib/clang/21 -dependency-file tmp_build_dir/o/3d.sycl.MPI.EXE/BinaryBHLevel.d -MT tmp_build_dir/o/3d.sycl.MPI.EXE/BinaryBHLevel.o -MP -internal-isystem /opt/intel/oneapi/compiler/2025.3/bin/compiler/../../include/sycl/stl_wrappers -internal-isystem /opt/intel/oneapi/compiler/2025.3/bin/compiler/../../include -internal-isystem /opt/intel/oneapi/compiler/2025.3/bin/compiler/../../include/sycl/stl_wrappers -internal-isystem /opt/intel/oneapi/compiler/2025.3/bin/compiler/../../include -D BL_USE_MPI -D AMREX_USE_MPI -D AMREX_USE_SYCL -D AMREX_USE_DPCPP -D AMREX_SYCL_SUB_GROUP_SIZE=16 -D BL_NO_FORT -D AMREX_USE_GPU -D BL_COALESCE_FABS -D AMREX_GPU_MAX_THREADS=256 -D AMREX_USE_GPU_RDC -D BL_SPACEDIM=3 -D AMREX_SPACEDIM=3 -D BL_FORT_USE_UNDERSCORE -D AMREX_FORT_USE_UNDERSCORE -D BL_Linux -D AMREX_Linux -D AMREX_PARTICLES -D NDEBUG -D AMREX_NO_PROBINIT -I tmp_build_dir/s/3d.sycl.MPI.EXE -I . -I /home/miren/NR/amrex/Src/Base -I /home/miren/NR/amrex/Src/Base/Parser -I /home/miren/NR/amrex/Src/Boundary -I /home/miren/NR/amrex/Src/AmrCore -I /home/miren/NR/amrex/Src/Amr -I /home/miren/NR/amrex/Src/Particle -I /home/miren/NR/GRTeclyn/Examples/BinaryBH -I /home/miren/NR/GRTeclyn/Source/AMRInterpolator -I /home/miren/NR/GRTeclyn/Source/BlackHoles -I /home/miren/NR/GRTeclyn/Source/CCZ4 -I /home/miren/NR/GRTeclyn/Source/Grids -I /home/miren/NR/GRTeclyn/Source/GRTeclynCore -I /home/miren/NR/GRTeclyn/Source/IO -I /home/miren/NR/GRTeclyn/Source/Maths -I /home/miren/NR/GRTeclyn/Source/Tagging -I /opt/intel/oneapi/mpi/2021.17/include -I/opt/intel/oneapi/mpi/2021.17/include -I/opt/intel/oneapi/umf/1.0/include -I/opt/intel/oneapi/mkl/2025.3/include -c-isystem /opt/intel/oneapi/umf/1.0/include -c-isystem /opt/intel/oneapi/mkl/2025.3/include -c-isystem /opt/intel/oneapi/tbb/2022.3/include -cxx-isystem /opt/intel/oneapi/umf/1.0/include -cxx-isystem /opt/intel/oneapi/mkl/2025.3/include -cxx-isystem /opt/intel/oneapi/tbb/2022.3/include -internal-isystem /opt/intel/oneapi/compiler/2025.3/bin/compiler/../../opt/compiler/include -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15 -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/x86_64-redhat-linux -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/backward -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15 -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/x86_64-redhat-linux -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/backward -internal-isystem /opt/intel/oneapi/compiler/2025.3/lib/clang/21/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/15/../../../../x86_64-redhat-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /opt/intel/oneapi/compiler/2025.3/lib/clang/21/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/15/../../../../x86_64-redhat-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -Wno-tautological-constant-compare -std=c++17 -fdeprecated-macro -ferror-limit 19 -fmessage-length=110 -pthread -fgpu-rdc -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -fcolor-diagnostics -vectorize-loops -vectorize-slp -mlong-double-64 -mllvm -disable-hir-generate-mkl-call -mllvm -sycl-host -mllvm -loopopt=1 -floopopt-pipeline=light -mllvm -intel-abi-compatible=true -dwarf-debug-flags " --driver-mode=g++ --intel -MMD -MP -gline-tables-only -fdebug-info-for-profiling -O3 -Wno-tautological-constant-compare -std=c++17 -fsycl -fsycl-device-code-split=per_kernel -mlong-double-64 -Xclang -mlong-double-64 -pthread -fsycl-targets=spir64_gen -D BL_USE_MPI -D AMREX_USE_MPI -D AMREX_USE_SYCL -D AMREX_USE_DPCPP -D AMREX_SYCL_SUB_GROUP_SIZE=16 -D BL_NO_FORT -D AMREX_USE_GPU -D BL_COALESCE_FABS -D AMREX_GPU_MAX_THREADS=256 -D AMREX_USE_GPU_RDC -D BL_SPACEDIM=3 -D AMREX_SPACEDIM=3 -D BL_FORT_USE_UNDERSCORE -D AMREX_FORT_USE_UNDERSCORE -D BL_Linux -D AMREX_Linux -D AMREX_PARTICLES -D NDEBUG -D AMREX_NO_PROBINIT -I tmp_build_dir/s/3d.sycl.MPI.EXE -I . -I /home/miren/NR/amrex/Src/Base -I /home/miren/NR/amrex/Src/Base/Parser -I /home/miren/NR/amrex/Src/Boundary -I /home/miren/NR/amrex/Src/AmrCore -I /home/miren/NR/amrex/Src/Amr -I /home/miren/NR/amrex/Src/Particle -I /home/miren/NR/GRTeclyn/Examples/BinaryBH -I /home/miren/NR/GRTeclyn/Source/AMRInterpolator -I /home/miren/NR/GRTeclyn/Source/BlackHoles -I /home/miren/NR/GRTeclyn/Source/CCZ4 -I /home/miren/NR/GRTeclyn/Source/Grids -I /home/miren/NR/GRTeclyn/Source/GRTeclynCore -I /home/miren/NR/GRTeclyn/Source/IO -I /home/miren/NR/GRTeclyn/Source/Maths -I /home/miren/NR/GRTeclyn/Source/Tagging -c BinaryBHLevel.cpp -o tmp_build_dir/o/3d.sycl.MPI.EXE/BinaryBHLevel.o -I /opt/intel/oneapi/mpi/2021.17/include -fveclib=SVML" -D__GCC_HAVE_DWARF2_CFI_ASM=1 -fintel-compatibility -fintel-libirc-allowed -fintel-libimf-allowed -o /tmp/icpx-e344993104/BinaryBHLevel-7100b5.o -x c++ BinaryBHLevel.cpp
1.      <eof> parser at end of file
2.      Code generation
3.      Running pass 'Function Pass Manager' on module 'BinaryBHLevel.cpp'.
4.      Running pass 'Register Coalescer' on function ''
 #0 0x000055f564b4ee3d llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x6422e3d)
 #1 0x000055f564b4da07 llvm::sys::RunSignalHandlers() (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x6421a07)
 #2 0x000055f564b4f7bc SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #3 0x00007fe16f81b070 __restore_rt (/lib64/libc.so.6+0x1a070)
 #4 0x000055f562ee606b (anonymous namespace)::RegisterCoalescer::reMaterializeTrivialDef(llvm::CoalescerPair const&, llvm::MachineInstr*, bool&) RegisterCoalescer.cpp:0:0
 #5 0x000055f562ee2028 (anonymous namespace)::RegisterCoalescer::joinCopy(llvm::MachineInstr*, bool&, llvm::SmallPtrSetImpl<llvm::MachineInstr*>&) RegisterCoalescer.cpp:0:0
 #6 0x000055f562edfc43 (anonymous namespace)::RegisterCoalescer::copyCoalesceWorkList(llvm::MutableArrayRef<llvm::MachineInstr*>) RegisterCoalescer.cpp:0:0
 #7 0x000055f562edcf39 (anonymous namespace)::RegisterCoalescer::run(llvm::MachineFunction&) RegisterCoalescer.cpp:0:0
 #8 0x000055f562edc0a6 (anonymous namespace)::RegisterCoalescerLegacy::runOnMachineFunction(llvm::MachineFunction&) RegisterCoalescer.cpp:0:0
 #9 0x000055f5627d3a08 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x40a7a08)
#10 0x000055f5629e0508 llvm::FPPassManager::runOnFunction(llvm::Function&) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x42b4508)
#11 0x000055f5629dfc9c llvm::FPPassManager::runOnModule(llvm::Module&) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x42b3c9c)
#12 0x000055f562f806cc llvm::legacy::PassManagerImpl::run(llvm::Module&) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x48546cc)
#13 0x000055f562c2ba4d clang::emitBackendOutput(clang::CompilerInstance&, clang::CodeGenOptions&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream>>, clang::BackendConsumer*) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x44ffa4d)
#14 0x000055f562b84419 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x4458419)
#15 0x000055f562a9b8f7 clang::ParseAST(clang::Sema&, bool, bool) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x436f8f7)
#16 0x000055f562c291ac clang::CodeGenAction::ExecuteAction() (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x44fd1ac)
#17 0x000055f5635afd6f clang::FrontendAction::Execute() (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x4e83d6f)
#18 0x000055f5635af8be clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x4e838be)
#19 0x000055f563149bf5 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x4a1dbf5)
#20 0x000055f5633bb71e cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x4c8f71e)
#21 0x000055f56314b03b ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) driver.cpp:0:0
#22 0x000055f5633beabc clang_main(int, char**, llvm::ToolContext const&) (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x4c92abc)
#23 0x000055f56372ae09 main (/opt/intel/oneapi/compiler/2025.3/bin/compiler/clang+0x4ffee09)
#24 0x00007fe16f804575 __libc_start_call_main (/lib64/libc.so.6+0x3575)
#25 0x00007fe16f804628 __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x3628)
#26 0x000055f563d8bc1e _start /export/project/tools-build/toolchain-cross/src/glibc/csu/../sysdeps/x86_64/start.S:122:0
icpx: error: unable to execute command: Segmentation fault (core dumped)
icpx: error: clang frontend command failed due to signal (use -v to see invocation)
Intel(R) oneAPI DPC++/C++ Compiler 2025.3.1 (2025.3.1.20251023)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/intel/oneapi/compiler/2025.3/bin/compiler
Configuration file: /opt/intel/oneapi/compiler/2025.3/bin/compiler/../icpx.cfg
icpx: note: diagnostic msg: Error generating preprocessed source(s).
make: *** [/home/miren/NR/amrex/Tools/GNUMake/Make.rules:260: tmp_build_dir/o/3d.sycl.MPI.EXE/BinaryBHLevel.o] Error 1

Expected outcome

The code should compile as with previous versions of the Intel oneAPI toolkit.

Software versions

GRTeclyn: cae4124
AMReX: a7f0aed
icpx: Intel(R) oneAPI DPC++/C++ Compiler 2025.3.1 (2025.3.1.20251023), also 2025.3.0 (2025.3.0.20251010)

[mirenradia]

Changing the optimization flag from -O3 to -O1 makes the problem go away (it still happens with -O2) but this isn't an acceptable solution.

[mirenradia]

@julianakwan, are you able to open a ticket with Intel for this using your access to Intel priority support?

If you want to test yourself, note that the 2025.3 Intel oneAPI toolkit is available on Dawn at /usr/local/dawn/software/external/intel-oneapi/2025.3.

[julianakwan]

@julianakwan, are you able to open a ticket with Intel for this using your access to Intel priority support?

If you want to test yourself, note that the 2025.3 Intel oneAPI toolkit is available on Dawn at /usr/local/dawn/software/external/intel-oneapi/2025.3.

Unfortunately, I no longer have access to priority support either. 😢 Maybe you could ask @draenog to raise the issue?