prov/shm: Intermittent MPI collective hang traced to the `shm` provider's connreq handshake

[Sbozzolo]

Preface

This is the result of several days of investigation into occasional hangs in MPI collectives which led me to the shm provider in libfabric. I used a large language model to guide me through the libfabric source, and the final root-cause determination is the model's. I can't personally judge whether that root cause is correct, but my aim is to provide enough reproducible data for maintainers to make the call. I've tried to clearly separate what I measured from what the model inferred and I hope that this will be helpful in investigating this issue. I am happy to share the full diagnostic logs, the instrumentation patch, and rerun with additional logging/patches, or anything else that would help.

Summary

• A few percent of my 384-rank (192 ranks/node × 2 nodes) Palace jobs on AWS r8g.48xlarge (ARM64) hang at the first MPI collective.
• The hang seems much less frequent when running on 4 nodes
• The hang is eliminated by either FI_EFA_ENABLE_SHM_TRANSFER=0 or FI_EFA_USE_SM2=1, which possibly points at the shm provider's connection setup specifically.
• Instrumentation shows shm connection requests ("connreqs") being dropped when the peer's shared-memory region isn't yet mappable (-ENOENT), with the corresponding senders stuck forever.
• The model attributes this to a one-shot connreq that is never retried when the drop happens; I'd appreciate maintainers' judgment on whether that's correct.

The problem

Palace is an MPI code developed by AWS to simulate quantum chips. I'm working on internal infrastructure improvements that involve running Palace simulations in containers, and I want to run with EFA.

I run containers through Apptainer, and I noticed that a few percent of my jobs on r8g.48xlarge instances stall at the first MPI collective.

Software stack

• Containers built with Spack via this GitHub action
  • Dockerfile template
  • Spack environment
• OpenMPI 5.0.10, libfabric 2.5.1, with fabrics=efa,lnx,mrail,rxm,shm,sockets,tcp,udp
• Apptainer 1.5.2
• I also tested OpenMPI main and libfabric 2.6 and saw no difference.

Diagnosis

I don't have a minimal reproducer. The hang needs a fresh job and only appears a few percent of the time. My debugging strategy was to submit 100 jobs trying to trigger deadlocks. Then, I instrumented Palace and finally patched libfabric to print additional stats.

What pointed me at shm in libfabric

After investigating obvious problems (e.g., a rank aborting/erroring) and trying various versions of the application/libraries, I attached gdb to the hung ranks of a stuck job. The call tree captured via gdb from a stuck rank on the head node was:

  #0  efa_rdm_cq_readfrom ()              ← spinning in libfabric
  #1  ompi_mtl_ofi_progress_no_inline ()  ← OpenMPI OFI MTL calling libfabric
  #2  opal_progress ()                    ← OpenMPI progress engine
  #3  mca_pml_cm_recv ()                  ← cm PML blocking receive
  #4  ompi_coll_base_barrier_intra_tree() ← collective algorithm
  #5  PMPI_Barrier ()                     ← the MPI call that deadlocks

And when caught inside ofi_peer_cq_read (the SHM path within efa_rdm_cq_readfrom):

  #0  smr_ep_progress ()                  ← SHM provider progress
  #1  ofi_cq_progress ()                  ← util CQ progress
  #2  ofi_peer_cq_read ()                 ← FI_PEER CQ read (SHM → EFA)
  #3  efa_rdm_cq_readfrom ()              ← EFA RDM CQ read entry
  #4  ompi_mtl_ofi_progress_no_inline ()
  #5  opal_progress ()
  #6  ompi_request_default_wait ()
  #7  ompi_coll_base_reduce_generic ()    ← reduce inside allreduce
  #8  ompi_coll_base_reduce_intra_binomial ()
  #9  ompi_coll_tuned_allreduce_intra_dec_fixed ()
  #10 mca_coll_han_comm_create_new ()     ← HAN creating sub-communicators
  #11 mca_coll_han_barrier_intra_simple ()
  #12 PMPI_Barrier ()

This pointed me to libfabric as the place to look, and to shm specifically.

To further strengthen the identification of the problematic component, I found that either of these two settings would make the deadlock go away: FI_EFA_ENABLE_SHM_TRANSFER=0 or FI_EFA_USE_SM2=1.

Separately, I also looked at EFA device hardware counters and found that they the are clean and symmetric on both nodes even when the job is stuck (rx_pkts/tx_pkts mirror each other, rx_drops=0, retrans=0, no errors).

The instrumentation and what it caught

At this point, I asked the LLM to patch libfabric 2.5.1 to add instrumentation. After several attempts and disproven hypotheses, the model settled on logging the shm connreq handshake to /tmp/smr_diag_<pid>.log. On a stuck run (head node 192.168.92.216), the receiver side logged 8 dropped connreqs, e.g.:

connreq: pid=6775 DROPPED connreq from peer name=fe80::...0005... tx_id=6  idx=7   map_err=-2
connreq: pid=6845 DROPPED connreq from peer name=fe80::...0001... tx_id=46 idx=47  map_err=-2

map_err=-2 is -ENOENT. The senders whose connreqs were dropped logged a matching permanent stall:

verify_peer: pid=6776 peer=6  STUCK 1M attempts: name_sent=1 peer_id=-1 region_mapped=1
verify_peer: pid=6846 peer=46 STUCK 1M attempts: name_sent=1 peer_id=-1 region_mapped=1

name_sent=1 (the connreq was sent), peer_id=-1 (no acknowledgement ever came back), region_mapped=1 (the peer's shared-memory region did become mappable later, after the failure). There were zero "command queue full" events, so this is a mapping failure, not queue exhaustion (this was a previous hypothesis that the model had).

The code-level root cause the model identified (I can't personally verify this)

⚠️ This section is the model's reading of the code and goes beyond my own understanding of libfabric. I'm including it in case it's a useful starting point, but please treat it as a hypothesis to check rather than a claim I can stand behind. ⚠️

For each local rank pair, shm establishes a connection once via a connreq. Line references are to libfabric 2.5.1, unchanged on current main.

1. Sender: smr_send_name() (prov/shm/src/smr_ep.c:172) puts a connreq into the peer's command queue and sets name_sent = 1 (line 196), so it will never send another connreq to that peer (guarded at line 181).
2. Receiver:smr_progress_connreq() (prov/shm/src/smr_progress.c:827) consumes the connreq and calls smr_map_to_region() to map the sender's /dev/shm region (line 838). If that returns -ENOENT (the sender's shm file isn't visible yet), it does goto out (line 842) and skips writing the acknowledgement back into the sender's region (the smr_peer_data(peer_smr)[cmd->hdr.tx_id].id = idx at line 869 is never reached). The command-queue slot is then released regardless (smr_progress_cmd(), line 1348), so the connreq is gone.
3. Sender: spins in smr_verify_peer() (prov/shm/src/smr_ep.c:200) returning EAGAIN forever: its peer id was never set (early-out at line 210 never taken) and name_sent=1 blocks any retry (line 181).

This matches the logs: the receiver's map_err=-2 drop and the sender's name_sent=1 peer_id=-1 stall are the two ends of the same broken handshake.

Why it's intermittent, not always fatal (again, the model's reasoning): if the peer later sends to the stuck sender, it issues a connreq in the reverse direction; processing that reverse connreq reaches line 860 (smr_peer_data(ep->region)[idx].id = cmd->hdr.tx_id) and completes the id write on the originally-stuck side, unsticking it. So a dropped connreq would only strand a link permanently when traffic on that edge is effectively one-directional. This is at least consistent with only some runs hanging, and only some ranks within them.

[zachdworkin]

@Sbozzolo thank you for finding this! I think you're right that there is an issue here if the peer fails to progress the connreq. I would like to try to reproduce it and am having a little trouble understanding the full code flow you're talking about.

It seems like you're saying that the sender is stuck waiting to hear the acknowledgement from its peer of the established connreq "smr_peer_data(peer_smr)[cmd->hdr.tx_id].id = idx;". The peer never writes this ack because it always takes the "goto out;" line inside the "if (!peer_smr)" block because the smr_map_to_region function fails. If this isn't the path then I am very confused...

Are you able to share the return code from smr_map_to_region? Is it FI_ENOENT or -errno (and which errno)?
Additionally since there are several of the same return codes in smr_map_to_region do you know which check inside creates the failure? And when it fails is it consistently the same check failing?
Are you seeing the FI_WARN "Could not map peer region" line when you set FI_LOG_LEVEL=warn?
Does it fail the assert(peer_smr) line on a debug build (if it gets to that point)?

Do you see the issue when running with just shm (without efa)? [If you run this you'll have to limit ranks<256 and run on 1 node because that is the current MAX_PEERS supported by shm]
And finally can you share the collective you're running? That way I can build and run with the same one to work on a fix.

[Sbozzolo]

Thanks for looking at this so quickly!

Yes, you've got the path exactly right: the sender is stuck waiting for the peer to write smr_peer_data(peer_smr)[cmd->hdr.tx_id].id = idx;, and the peer never writes it because smr_progress_connreq takes the goto out inside if(!peer_smr) when smr_map_to_region fails.

Once again, a caveat before the specifics: I've never worked in the libfabric source, and an LLM guided the investigation and wrote the instrumentation, and I'm relaying what the logs showed plus what it inferred. So please treat the code-flow narrative as a hypothesis to verify.

Here are the answers to each of your questions:

Are you able to share the return code from smr_map_to_region? Is it FI_ENOENT or -errno (and which errno)?

It returns -FI_ENOENT, caused by shm_open failing with errno 2 (ENOENT).

Additionally since there are several of the same return codes in smr_map_to_region do you know which check inside creates the failure?

It's the very first check, shm_open itself (line 134 in smr_av.c). It never reaches the ftruncate size check, mmap, or the peer->pid == 0 check. We instrumented all five failure paths and it's always shm_open returning ENOENT.

And when it fails is it consistently the same check failing?

Yes, 100% of observed failures are shm_open with errno 2. We've captured this across hundreds of runs and it's always the same path.

Are you seeing the FI_WARN "Could not map peer region" line when you set FI_LOG_LEVEL=warn?

With FI_LOG_LEVEL=warn we see:

  shm:av:smr_map_to_region():134<warn> shm_open error: name fe80::432:b0ff:fed3:5957_002f_62a491c6_eddf errno 2

Does it fail the assert(peer_smr) line on a debug build (if it gets to that point)?

It never reaches the assert. The if (ret) check after smr_map_to_region fires first and takes the goto out path, so peer_smr is never dereferenced.

Do you see the issue when running with just shm (without efa)? [If you run this you'll have to limit ranks<256 and run on 1 node because that is the current MAX_PEERS supported by shm]

Yes, and I found this is actually the easiest way to reproduce it. We tested with 192 ranks on a single ARM64 node (aarch64, r8g.48xlarge) and found a much higher rate of failures.

And finally can you share the collective you're running? That way I can build and run with the same one to work on a fix.

This is a standard MPI_Barrier(the first collective called after MPI_Init), but I also found the same problem with MPI_Allreduce. The application does:

  Mpi::Init(argc, argv);
  // ... prints a banner (rank 0 only, no collective) ...
  Mpi::Barrier(world_comm);  // <-- deadlocks here

[zachdworkin]

Thank you for providing background context and the full picture so I can debug it. Can you provide your build and run commands? And just to confirm, you are using upstream/main or upstream/v2.6.x correct?

Here are the common run commands I use:
Build Libfabric: "./autogen.sh; ./configure --prefix=/path/to/prefix --enable-shm --enable-debug; make -j install"
Build OMPI: "./autogen.sh; ./configure --prefix=/path/to/prefix --with-libfabric=/path/to/libfabric/install --with-slurm --disable-oshmem --without-hcoll --disable-mpi-fortran --enable-debug --enable-mem-debug CFLAGS="-g -O0" CXXFLAGS="-g -O0"; make -j install"
Run: "FI_PROVIDER=shm mpirun -x LD_LIBRARY_PATH --mca mtl ofi --mca pml cm --mca opal_common_ofi_provider_include 'shm' -n 192 --map-by numa --bind-to core --report-bindings /path/to/osu/libexec/osu-micro-benchmarks/mpi/collective/osu_barrier"

Do these match your build/run commands?

[Sbozzolo]

It's more complicated than that. I build everything with spack and run through Apptainer. Let me try simplifying the setup and collecting more details on the build flags. I'll get back to you

[Sbozzolo]

@zachdworkin, thanks for the patience.

As I was trying to simplify the setup, I think I might have stumbled upon the root cause: an external /dev/shm sweep by systemd-logind's RemoveIPC.

Our compute nodes have RemoveIPC=yes (the systemd default). When a uid's last login session on a node closes, logind runs clean_ipc_by_uid(), which shm_unlinks every POSIX shm object owned by that uid. Our MPI jobs run in a fairly complex setup that involves a batch scheduler/ssh/containers, and I am thinking that this wipes the job's shm regions mid-startup (eg., our scheduler doesn't run jobs on a longin session).

Here's some evidence:

• The EFA *_eddf regions and OpenMPI's sm_segment.* BTL files are deleted together, in the same ~2 ms window.
• Instrumenting all of libfabric's shm_unlink sites (smr_free, smr_map retry, smr_create error) shows that none fire for the swept regions.
• An LD_PRELOAD interposer on unlink/unlinkat/shm_unlink, loaded in every job process, records nothing. Meanwhile inotify on /dev/shm shows the real DELETEs.

As a confirmation, I tested running with loginctl enable-linger and I could not reproduce the hang with 200 runs.

The LLM I am using to help me in the investigation suggests this angle for libfabric:

Even though the trigger is external, the shm provider turns a transient "peer region temporarily gone" into a permanent hang, because smr_verify_peer returns before smr_send_name (so the connreq is never (re)sent) and there's no shm to device fallback on EFA. If you think it's in scope, hardening the provider to treat a swept-region ENOENT as retryable (and/or an EFA device fallback) would make libfabric more resilient.

I'll let you take the next step here: I am happy to close, or to help test a hardening patch if you'd like to pursue it.

Thank you!

[zachdworkin]

@Sbozzolo wow thats quite an interesting feature xD. I would like to support a retry in this event. Not only to fix your problem but also because we used to have an application use case with peers connecting and disconnecting where this could become an issue. I will look into fixing it and ping you on the PR or hardening testing. Will probably happen next week. Thanks for catching this!