[BUG] Finalization segfault if using Open MPI 5

[eddy16112]

Describe the bug
We have seen a segfault during runtime finalization when using UCX+Open MPi 5

To Reproduce
Steps to reproduce the behavior:

1. Create a conda env with Open MPI 5.0.8

conda create -n ompi5
conda activate ompi5
conda install conda-forge::openmpi

2. Compile realm with UCX

cmake ../ -DREALM_ENABLE_UCX=ON -DREALM_ENABLE_CUDA=OFF -DREALM_ENABLE_HIP=OFF -DREALM_ENABLE_OPENMP=OFF -DREALM_ENABLE_PYTHON=ON -DREALM_ENABLE_HDF5=OFF -DREALM_BUILD_TESTS=ON

3. Run any of the realm program, I picked memspeed and remove everything except runtime init and shutdown.

memspeed

Expected behavior
Here is the backtrace

Thread 6 (Thread 0x7fffe0fddc00 (LWP 1607363) "memspeed" (Exiting)):
#0  0x00007ffff73af5b0 in ?? ()
#1  0x00007ffff7894be1 in advise_stack_range (guardsize=<optimized out>, pd=140736968121344, size=<optimized out>, mem=0x7fffe0edb000) at ./nptl/allocatestack.c:195
#2  start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:551
#3  0x00007ffff79268c0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7ffff01a2c00 (LWP 1607362) "memspeed" (Exiting)):
#0  0x00007ffff73af5b0 in ?? ()
#1  0x00007ffff7894be1 in advise_stack_range (guardsize=<optimized out>, pd=140737221635072, size=<optimized out>, mem=0x7ffff00a0000) at ./nptl/allocatestack.c:195
#2  start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:551
#3  0x00007ffff79268c0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7ffff13ff640 (LWP 1607357) "cuda0000380000f"):
#0  0x00007ffff7918c3f in __GI___poll (fds=0x555555f15e50, nfds=3, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007ffff284164f in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007ffff290f18f in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007ffff283c233 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#4  0x00007ffff7894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#5  0x00007ffff79268c0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7ffff7ab3c00 (LWP 1607353) "memspeed"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=1607363, futex_word=0x7fffe0fdded0) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=1607363, futex_word=0x7fffe0fdded0) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fffe0fdded0, expected=1607363, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007ffff7896624 in __pthread_clockjoin_ex (threadid=140736968121344, thread_return=0x0, clockid=0, abstime=0x0, block=<optimized out>) at ./nptl/pthread_join_common.c:105
#4  0x000055555577b2cb in Realm::KernelThread::join (this=0x555555fadf60) at /home/weiwu/realm/src/realm/threads.cc:1062
#5  0x0000555555aaf17d in Realm::BackgroundWorkThread::join (this=0x555555d8ced0) at /home/weiwu/realm/src/realm/bgwork.cc:154
#6  0x0000555555ab0318 in Realm::BackgroundWorkManager::stop_dedicated_workers (this=0x555555d8d300) at /home/weiwu/realm/src/realm/bgwork.cc:335
#7  0x0000555555707b4d in Realm::RuntimeImpl::wait_for_shutdown (this=0x555555d8d020) at /home/weiwu/realm/src/realm/runtime_impl.cc:2826
#8  0x00005555556fc998 in Realm::Runtime::wait_for_shutdown (this=0x7fffffffde40) at /home/weiwu/realm/src/realm/runtime_impl.cc:734
#9  0x000055555559782a in main (argc=5, argv=0x7fffffffdf78) at /home/weiwu/realm/tests/memspeed.cc:584

If we remove the MPI_Finaliza, the segfault is gone.

Here is a branch that could reproduce the segfault even without UCX. https://github.com/StanfordLegion/realm/commits/debug-mpi
d4d4718
In this branch, we explicitly initialize the MPI bootstrap during initialization and close it during the finalization, then we can produce the segfault using

memspeed  -ll:networks none

The MPI bootstrap dlopen the mpi wrapper, which calls MPI_Init_thread and MPI_Finalize. If we replace the dlopen with a direct call to MPI, then the error is gone.

[manopapad]

Fixing this would allow us to remove the openmpi<5 restriction on Legate packages.

[eddy16112]

@lightsighter @apryakhin @muraj Any ideas? This looks so weird to me because we does not launch any work, but just initialize and finalize the runtime and the core module.

[manopapad]

also ping @SeyedMir

[muraj]

Is this only with UCX or also with the MPI network?

[marcinz]

I think that what may be happening is that when we dlclose the OpenMPI DSO, it's .fini gets called and messes up the stacks. I think that OpenMPI may not be designed to be destructed before all the thread related stuff is finished. Some steps to test this could be:

• dlclose OpenMPI only after all thread joins have been performed (e.g., after bgwork.stop_dedicated_workers())
• Do not dlclose OpenMPI at all
• Use RTLD_NODELETE in dlopen

I think that realm should not be touching any pthread stuff after OpenMPI .fini is called, or we are asking for trouble. Probably the "cleanest" thing would be to dlclose after all pthread stuff, but if we don't call OpenMPI's .fini it should not be a big deal?

[eddy16112]

@muraj The MPI Module works. The tricky parts are that

1. if I do not enable UCX network, but only enable the MPI bootstrap, it also crashes. See d4d4718 for what I did.
2. If I do not dlopen the bootstrap, but just call MPI_Init/Finalize directly, see the same commit above but undef OH_MPI_CRASH, then it also works.

@marcinz It does not matter with thread join. I moved the MPI_Finalize to the end of the realm runtime shutdown, then I am getting another segfault

Thread 1 "memspeed" received signal SIGSEGV, Segmentation fault.
0x00007ffff73af6c0 in ?? ()
(gdb) thread apply all bt

Thread 3 (Thread 0x7ffff13ff640 (LWP 1761890) "cuda0000380000f"):
#0  0x00007ffff7918c3f in __GI___poll (fds=0x555555ef9980, nfds=3, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007ffff284164f in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007ffff290f18f in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007ffff283c233 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#4  0x00007ffff7894ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#5  0x00007ffff79268c0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7ffff7ab3c00 (LWP 1761874) "memspeed"):
#0  0x00007ffff73af6c0 in ?? ()
#1  0x0000555555789d95 in Realm::SharedMemoryInfo::~SharedMemoryInfo (this=0x555555d6de10, __in_chrg=<optimized out>) at /home/weiwu/realm/src/realm/shm.cc:262
#2  0x00005555556b6a4a in std::pair<unsigned long long const, Realm::SharedMemoryInfo>::~pair (this=0x555555d6de08, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/stl_pair.h:211
#3  0x00005555556b6a6e in __gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<unsigned long long const, Realm::SharedMemoryInfo>, false> >::destroy<std::pair<unsigned long long const, Realm::SharedMemoryInfo> > (this=0x555555d711e8, __p=0x555555d6de08) at /usr/include/c++/11/ext/new_allocator.h:168
#4  0x00005555556b5218 in std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<unsigned long long const, Realm::SharedMemoryInfo>, false> > >::destroy<std::pair<unsigned long long const, Realm::SharedMemoryInfo> > (__a=..., __p=0x555555d6de08) at /usr/include/c++/11/bits/alloc_traits.h:535
#5  0x00005555556b2bd5 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<unsigned long long const, Realm::SharedMemoryInfo>, false> > >::_M_deallocate_node (this=0x555555d711e8, __n=0x555555d6de00) at /usr/include/c++/11/bits/hashtable_policy.h:1894
#6  0x00005555557242d9 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<unsigned long long const, Realm::SharedMemoryInfo>, false> > >::_M_deallocate_nodes (this=0x555555d711e8, __n=0x0) at /usr/include/c++/11/bits/hashtable_policy.h:1916
#7  0x000055555571c0d2 in std::_Hashtable<unsigned long long, std::pair<unsigned long long const, Realm::SharedMemoryInfo>, std::allocator<std::pair<unsigned long long const, Realm::SharedMemoryInfo> >, std::__detail::_Select1st, std::equal_to<unsigned long long>, std::hash<unsigned long long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::clear (this=0x555555d711e8) at /usr/include/c++/11/bits/hashtable.h:2320
#8  0x0000555555712f64 in std::_Hashtable<unsigned long long, std::pair<unsigned long long const, Realm::SharedMemoryInfo>, std::allocator<std::pair<unsigned long long const, Realm::SharedMemoryInfo> >, std::__detail::_Select1st, std::equal_to<unsigned long long>, std::hash<unsigned long long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::~_Hashtable (this=0x555555d711e8, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/hashtable.h:1532
#9  0x000055555570f1d8 in std::unordered_map<unsigned long long, Realm::SharedMemoryInfo, std::hash<unsigned long long>, std::equal_to<unsigned long long>, std::allocator<std::pair<unsigned long long const, Realm::SharedMemoryInfo> > >::~unordered_map (this=0x555555d711e8, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/unordered_map.h:102
#10 0x00005555556feb23 in Realm::RuntimeImpl::~RuntimeImpl (this=0x555555d70fc0, __in_chrg=<optimized out>) at /home/weiwu/realm/src/realm/runtime_impl.cc:1076
#11 0x00005555556fb607 in Realm::Runtime::wait_for_shutdown (this=0x7fffffffdec0) at /home/weiwu/realm/src/realm/runtime_impl.cc:737
#12 0x000055555559682a in main (argc=3, argv=0x7fffffffdff8) at /home/weiwu/realm/tests/memspeed.cc:584

[eddy16112]

@marcinz using RTLD_NODELETE works.

[marcinz]

@marcinz It does not matter with thread join. I moved the MPI_Finalize to the end of the realm runtime shutdown, then I am getting another segfault

I don't mean MPI_Finalize. I mean the .fini from the DSO.

BTW, there have been some changes in the OpenMPI feedstock. The feedstock enabled UCX by default in late 2024. Packages up through 5.0.5 shipped a post-link hook that wrote pml = ^ucx and osc = ^ucx into etc/openmpi-mca-params.conf, so UCX was present but dormant and no UCX progress thread ever ran. Commit 66bb1402 (24 Oct 2024) removed that hook; starting with the 5.0.6 builds the default config no longer disables UCX, so UCX’s background thread is active and you hit the destructor/join race as soon as Realm dlcloses the plugin. I wonder if this may have something to do with the issue.

[muraj]

@marcinz using RTLD_NODELETE works.

When it comes to OpenMPI, it really doesn't like to be torn down until the app does, and likes to deep bind everything. I think to work around this we should have the bootstrap dlopen itself with RTLD_NODELETE, therefore keeping a reference to itself until application exit. I don't think we should RTLD_NODELETE the bootstraps within the ucx module though, as this seems to be openmpi specific.

[eddy16112]

so UCX’s background thread is active and you hit the destructor/join race as soon as Realm dlcloses the plugin. I wonder if this may have something to do with the issue.

Actually, even if I ran the MPI with --mca pml ^ucx, it still crashes. I tried the verbose run, and I did see that the pml_ucx is not picked up. However, if we ran with the old version such as 5.0.3, then the crash is gone. There may be some other background thread that is already active? Or the ucx background is active even if the UCX is not loaded?

[marcinz]

I am not sure, but I guess debugging this is probably not worth it at the end. Like @muraj said, we should not tear down (Open)MPI until the very end or at least not until all pthread stuff is done since this is where the crash comes from. Or, we can not tear it down at all with RTLD_NODELETE, although I am not sure if there are any potential implications to this like problems running other OpenMPI apps later on.

[SeyedMir]

UCX itself doesn't have any background progress thread. The only background thread that UCX spawns is for the UD transport AFAIK. You can disable that by UCX_TLS=^ud if you want to debug this further.
We can ping Open MPI folks on this if you think you can make a small reproducer (which I think won't be easy).

[lightsighter]

Is there any downside to just dlopen-ing with RTLD_NODELETE? What do we lose by having MPI always loaded into the address space (if anything)?