OpenMPI: Update to 3.0.5+|3.1.5+|4.0.1+ or Use ROMIO for IO

[ax3l]

A note on using openPMD-api, especially the parallel HDF5 backend, with OpenMPI:

OpenMPI's default for its IO backend is OMPIO, starting with 2.x.

The issues below are fixed in OpenMPI versions:

• v2.0: affected, not fixed (end-of-life)
• v2.x: affected, not fixed (end-of-life)
• v3.0.4 or newer
• v3.1.4 or newer
• v4.0.1 or newer

Unfortunately, that backend contains severe bugs leading to data corruption and sporadic crashes as of the latest releases (affected: 2.X to 3.1.3 and 4.0.0). We saw those issues so far with parallel HDF5, but since other MPI-IO-parallel methods such as ADIOS use the same MPI -IO API they are potentially affected as well. Please see open-mpi/ompi#6285 for details.

As a work-around for all systems that rely on OpenMPI (and its derivatives, such as BullMPI), disable the "OMPIO" default IO backend and fallback to the existing ROMIO backend for MPI-I/O until fixed versions are available.

Available runtime switches:

export OMPI_MCA_io=^ompio
mirun ...

or

mpirun --mca io ^ompio ...

Other MPI implementations such as MPICH, and MPICH-based flavors such as IntelMPI, use ROMIO by default (they develop ROMIO) and are not affected.

[ax3l]

Another OpenMPI issue limited file writes to 2GByte (per rank).
Update to 3.0.5+, 3.1.5+. Status in 4.0.* release series unknown. -> 4.0.3+ it seems