MPI error with custom ElementDeletionModifier in parallel: MPI_Allreduce/MPI_Bcast failed during mesh initialization

[CJL-CFDCSD]

**##

Description**

When running my MOOSE-based application (roshan-opt) in parallel with mpiexec -n 4, I encounter MPI collective communication errors during mesh initialization. The same input file runs successfully with mpiexec -n 1.

## Error Message

ElementDeletionModifier initialization...
Generic Warning: In ../Parallel/MPI/vtkMPICommunicator.cxx, line 72
MPI had an error

Other MPI error, error stack:
PMPI_Allreduce(450)...........: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x59cde1bbfaa0, count=1, datatype=MPI_DOUBLE, op=MPI_MAX, comm=MPI_COMM_WORLD) failed
PMPI_Allreduce(436)...........:
MPIR_Allreduce_impl(293)......:
MPIR_Allreduce_intra_auto(178):
MPIR_Allreduce_intra_auto(84).:
MPIR_Bcast_impl(310)..........:
MPIR_Bcast_intra_auto(223)....:
MPIR_Bcast_intra_binomial(182): Failure during collective

application called MPI_Abort(MPI_COMM_WORLD, 542712079) - process 0
Generic Warning: In ../Parallel/MPI/vtkMPICommunicator.cxx, line 72
MPI had an error

Other MPI error, error stack:
PMPI_Allreduce(450)...........: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x568c494447f0, count=1, datatype=MPI_DOUBLE, op=MPI_MAX, comm=MPI_COMM_WORLD) failed
PMPI_Allreduce(436)...........:
MPIR_Allreduce_impl(293)......:
MPIR_Allreduce_intra_auto(178):
MPIR_Allreduce_intra_auto(84).:
MPIR_Bcast_impl(310)..........:
MPIR_Bcast_intra_auto(223)....:
MPIR_Bcast_intra_binomial(182): Failure during collective

application called MPI_Abort(MPI_COMM_WORLD, 876159247) - process 1
Generic Warning: In ../Parallel/MPI/vtkMPICommunicator.cxx, line 72
MPI had an error

Message truncated, error stack:
PMPI_Allreduce(450).....................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7ffe309da1fc, count=1, datatype=MPI_UNSIGNED, op=MPI_MAX, comm=MPI_COMM_WORLD) failed
PMPI_Allreduce(436).....................:
MPIR_Allreduce_impl(293)................:
MPIR_Allreduce_intra_auto(178)..........:
MPIR_Allreduce_intra_auto(84)...........:
MPIR_Bcast_impl(310)....................:
MPIR_Bcast_intra_auto(223)..............:
MPIR_Bcast_intra_binomial(112)..........:
MPIDI_CH3_

application called MPI_Abort(MPI_COMM_WORLD, 409542926) - process 2
Generic Warning: In ../Parallel/MPI/vtkMPICommunicator.cxx, line 72
MPI had an error

Other MPI error, error stack:
PMPI_Allreduce(450)...........: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x61abb9596b70, count=1, datatype=MPI_DOUBLE, op=MPI_MAX, comm=MPI_COMM_WORLD) failed
PMPI_Allreduce(436)...........:
MPIR_Allreduce_impl(293)......:
MPIR_Allreduce_intra_auto(178):
MPIR_Allreduce_intra_auto(84).:
MPIR_Bcast_impl(310)..........:
MPIR_Bcast_intra_auto(223)....:
MPIR_Bcast_intra_binomial(123): message sizes do not match across processes in the collective routine: Received 4 but expected

application called MPI_Abort(MPI_COMM_WORLD, 205070607) - process 3

Files(pictures) for understand

Context

The error happens during the initialization of my custom ElementDeletionModifierCCANYAddSide (a MeshModifier that deletes elements and potentially adds sides).
I suspect the issue is related to parallel mesh consistency — possibly the modifier modifies the mesh differently across MPI ranks, causing subsequent collective operations (like MPI_Allreduce or MPI_Bcast on mesh data) to fail due to mismatched message sizes or inconsistent states.

Questions

Are there specific requirements for ensuring MeshModifier operations are consistent across all MPI ranks in MOOSE?
Should ElementDeletionModifier (or similar modifiers) be run only on rank 0 and then broadcast, or does MOOSE handle synchronization automatically?
What is the best practice for debugging MPI mesh synchronization issues in MOOSE?

Environment

OS: WSL2 (Ubuntu 20.04.6 LTS)
MOOSE: Installed via conda environment
MPI: MPICH (bundled with conda environment)
Application: Custom XXX-opt with custom mesh modifiers

[loganharbour]

What's the minimal number of MPI ranks that it fails with? Will it fail with 3? 2?

To debug MPI ranks, first build with debug:

METHOD=dbg make -j 8

After that, you can use a few convenient command line arguments. You can run with either:

• --start-in-debugger=[lldb,gdb] which will open a xterm window for each rank, with the debugger attached
• --stop-for-debugger=30 which will have the app stop for the value (30 here) in seconds, printing out ranks so that you can attach to it with a debugger

In either case, you should be able to get a stack trace to help us see what each rank is doing.

https://mooseframework.inl.gov/application_development/debugging.html is a useful reference as well

[CJL-CFDCSD]

First, we built with debug mode using METHOD=dbg make -j 4.
Then we tested with different MPI ranks:

With 2 ranks:

mpiexec -n 2 ../../roshan-dbg -i CarbanTestCCYuanHuForPaperMeshDeletion2.i --stop-for-debugger=30

The simulation runs until Time Step 12 (time = 8.07812), the nonlinear solve converges, but then fails during ElementDeletionModifierCCANYAddSide initialization with an MPI error:

Generic Warning: In ../Parallel/MPI/vtkMPICommunicator.cxx, line 72
MPI_Allreduce failed
application called MPI_Abort(MPI_COMM_WORLD, ...) – process 1

With 3 ranks:

mpiexec -n 3 ../../roshan-dbg -i CarbanTestCCYuanHuForPaperMeshDeletion2.i --stop-for-debugger=30

Same behavior — reaches Time Step 12, solve converges, then fails at the same ElementDeletionModifierCCANYAddSide initialization with the same MPI_Allreduce error, this time on process 2.

With 4 ranks:

mpiexec -n 4 ../../roshan-dbg -i CarbanTestCCYuanHuForPaperMeshDeletion2.i --stop-for-debugger=30
Again fails during ElementDeletionModifierCCANYAddSide initialization. All 4 processes abort. The error message shows:

Failure during collective
message sizes do not match across processes in the collective routine: Received 4 but expected

It seems the issue consistently occurs during the ElementDeletionModifierCCANYAddSide initialization after the nonlinear solve converges at step 12, regardless of the number of MPI ranks (tested 2, 3, and 4). The MPI_Allreduce / collective communication failure suggests some ranks may have different data sizes or states when entering this modifier.First，we use：METHOD=dbg make -j 4
then，

[loganharbour]

When you run with --stop-for-debugger, MOOSE will print out a statement with a listing of processor tasks that you can attach to with a debugger. Are you familiar with using a debugger? If you can attach to the tasks, you can help show where the inconsistencies are between tasks by showing a trace that specifies where in code each task is.

Can you do this? If you're not familiar with using a debugger, we can assist with more guidance.