Skip to content

Add MPI_APPNUM support for MPMD setups#695

Open
maximilian-tech wants to merge 3 commits into
Nek5000:nextfrom
maximilian-tech:add_mpi_appnum_support
Open

Add MPI_APPNUM support for MPMD setups#695
maximilian-tech wants to merge 3 commits into
Nek5000:nextfrom
maximilian-tech:add_mpi_appnum_support

Conversation

@maximilian-tech
Copy link
Copy Markdown

Add MPI_APPNUM support for MPMD setups.

In certain (albeit seldom) cases one likes to start nekRS in a MPMD setup (e.g. mpirun -np 2 <nekRS> : -np 2 <other_exe> for e.g. coupling applications.
While the actual coupling will probably happen via User Defined Function (UDF), the communicator passed to UDF_Setup0 is a copy of MPI_COMM_WORLD, which spans both executables (<nekRS> & <other_exe>), which is too large/contains too many processes.

This MR splits MPI_COMM_WORLD based on the color MPI_APPNUM, an attribute of MPI_COMM_WORLD, which returns the index of the program on the command line (e.g. 0 for <nekRS> and 1 for <other_exe>). After that split the communicator passed to UDF_Setup0 is the actual "cut down" nekRS communicator.

Hidden behind a CMake option, as there are probably funky launchers that do not set MPI_APPNUM.

This MR also fixes two bugs:

  1. Fail, if MPI cannot provide the requested Thread Level
  2. Free commGlobal

@stgeke
Copy link
Copy Markdown
Collaborator

stgeke commented Mar 19, 2026

I don’t think this is strictly necessary — we already support multiple sessions. For coupling different codes, a dedicated coupling (driver) application can be used. This approach avoids relying on MPMD, which is not consistently supported across all systems.

What are you trying to do?

@maximilian-tech
Copy link
Copy Markdown
Author

I’m using this to couple NekRS to custom post-processing running on another node. It relies on simple MPI point-to-point communication over MPI_COMM_WORLD in the UDF, so I needed this support for my setup -- no additional driver. I’m aware this is a niche use case, but I decided to upstream it anyway.

If not merged the two bug fixes should be merged though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants