Install MPI ABI into separate prefix

[eschnett]

Open MPI (head of the main branch) installs the header file for the new MPI ABI into $prefix/include/standard_abi, while installing OpenMPI ABI header file into $prefix/include. Since both header files are called mpi.h this is confusing and dangerous; if $prefix/include accidentally ends up on the include path then one might accidentally build against the wrong ABI.

There are also two different compiler drivers, mpicc and mpicc_abi. Neither autoconf nor cmake know about this. Many existing packages have (for better or worse) their own hand-rolled configure system (e.g. PETSc), and they also do not know about this. This again makes it very easy to accidentally compile or link against the wrong ABI. Unix is know to be abysmally bad at detecting such inconsistencies.

Having separate install directories would eliminate this risk. For example, installing all the files for the MPI ABI into $prefix/mpiabi (include/mpi.h, lib/libmpi_abi.so, and bin/mpicc) would do so.

Another option would be to install only either the OpenMPI or the MPI ABI files, not both.

[hppritcha]

this was discussed some at our meeting today. having two different prefixes for a single build/install sounds rather unusual.
How about if we add in a configure option to NOT build the OMPI ABI libraries. That way one could do two separate installs

--prefix=mpi_abi_only --enable-standard-abi --disable-ompi-abi
--prefix=ompi_abi_only --disable-standard-abi

[eschnett]

This is an excellent idea.

[jsquyres]

This seems like a reasonable compromise.

@hppritcha If you go down this approach, be sure to find all the places in the docs where we say that the OMPI ABI is always built.

[bosilca]

Two things I'd like to separate out from the install-layout discussion, because both survive whatever happens with #14426.

What we can and can't rename

Worth being precise about what's actually up for grabs here, since MPI-5.0 fixes both names:

The ABI must be implemented using a header named mpi.h. The MPI library that implements the standard ABI must be named mpi_abi. [...] ABI-compliant implementations must not require more than mpi_abi or its versioned variant as the sole direct dependency of the application binary.

So the libraries can't collide however we package things — a Forum-ABI install ships libmpi_abi, never libmpi. The header is the only genuine overlap, and the paper behind that chapter is explicit that a separate path plus helper files is the expected answer:

Implementations can provide this header in a different path from their own header, and perhaps help users with appropriate pkg-config definitions or compiler wrapper scripts, e.g., mpi_abi.pc or mpicc_abi, but these aspects of MPI are not standardized, nor are they part of the ABI.

which is what we ship today. The distinct library name also wasn't an accident anyone is working around — it was picked precisely so both ABIs could coexist in one prefix, since implementations were expected to keep their own ABI under the existing name.

None of that makes @eschnett's complaint invalid, but it does bound the fix: #14426 removes the two-mpi.h problem for an install that deliberately gives up the Open MPI ABI, and that's genuinely useful for packagers. It does nothing for the default install, which still has both headers and both wrappers, and that's the configuration almost everyone will actually get.

The discovery problem is in the tools, and it's worse than it looks

The autoconf/CMake/PETSc complaint is the real pain, so I went and read CMake's FindMPI. It searches for wrappers by name:

set(_MPI_C_GENERIC_COMPILER_NAMES mpicc mpcc mpicc_r mpcc_r)

No mpicc_abi, and no component or option anywhere in the module to ask for the Forum ABI. find_package(MPI REQUIRED) finds mpicc and quietly hands you the implementation-specific ABI. Note this is unaffected by install layout: separate prefixes don't help, because CMake still picks up whichever mpicc is on PATH and still has no vocabulary for "I want the Forum ABI."

There is a pkg-config fallback, and this part surprised me. Look at the module names it tries:

  if(LANG STREQUAL "C")
    set(_MPI_PKG "mpi-c")
  elseif(LANG STREQUAL "CXX")
    set(_MPI_PKG "mpi-cxx")
  elseif(LANG STREQUAL "Fortran")
    set(_MPI_PKG "mpi-fort")
  endif()

We ship ompi-c.pc, ompi-cxx.pc, ompi-fort.pc. Those have never matched, so that branch has never once resolved against an Open MPI install, under either ABI. It's gated on wrapper detection having failed first, so it only fires with MPI_<LANG>_SKIP_COMPILER_WRAPPER or when nothing is on PATH — but when it does fire it finds nothing. MPICH's installer docs never mention pkg-config either, so I suspect it's dead for them too, though I haven't checked their tree. This smells like an aliasing oversight rather than anybody's decision.

Three things I'd like us to do

1. Ship mpi-c.pc / mpi-cxx.pc / mpi-fort.pc aliases for the Open MPI ABI, so that fallback resolves at all. We already symlink .pc aliases in install-data-hook-always for ompi-f77.pc and ompi-f90.pc, so this is a few lines in a style we already use.

2. Settle on one portable pkg-config name for the Forum ABI with the ABI working group — mpi-abi-c.pc, or the mpi_abi.pc the paper suggests — and ship it next to ompi-abi-c.pc. Right now a hand-rolled configure has to know the string ompi-abi-c, which is exactly the implementation-specific knowledge this whole effort exists to eliminate. The wrapper half of that suggestion (mpicc_abi) got adopted by everyone; the pkg-config half just never happened, and it's the half that helps projects like PETSc that don't drive our wrappers.

3. Possibly worth approaching Kitware about FindMPI, if someone has the appetite for it. Both MPICH 5.0 and Open MPI 6.0 now ship a wrapper spelled exactly mpicc_abi and a library spelled exactly libmpi_abi, the latter because the standard requires it, so there's at least a reasonable case that the module could grow a way to ask for the Forum ABI. I have no idea how receptive they'd be, and it isn't work we control, so I'd treat it as a nice-to-have rather than something to depend on.

The first two are cheap and entirely in our hands. I'd rather land those than assume a packaging change will fix a detection problem that lives in somebody else's build system.

[eschnett]

In addition to the above, I suggest shipping mpicc and mpifort for the ABI wrappers as well, not mpicc_abi (i.e. without the _abi suffix). None of the current tools are looking for mpicc_abi, and if things are shipped in a separate prefix then there is no need to go away from mpicc there.

[bosilca]

I'm not advocating for this, in fact I think we should do the opposite to avoid having two different meanings for mpicc: either native or ABI API depending on the installation. Either mpicc -abi or mpicc_abi, i any case a clear indicator of the ABI requirement.

[eschnett]

A plain mpicc does not indicate any ABI. Every MPI implementation uses mpicc.

The ABI is not really user visible. This -abi flag (or the new tool name) would be basically the only place people would need to know which ABI they are using.