[ci] Modernize OpenMPI CI images to Ubuntu 26.04 / OpenMPI 5 (fix flaky mpi_group ORTE session-dir race) - #144
Merged
Merged
Conversation
Bump the github_ci base image to ubuntu:26.04, which ships OpenMPI 5.0.10
(PRRTE/PMIx). OpenMPI 5 removes the ORTE per-host session directory that
concurrent mpirun invocations (ctest -j) raced on, producing flaky
'orte_session_dir failed / mkdir ... No such file or directory' aborts
in MPI tests (e.g. mpi_group_np4).
- FROM ubuntu:24.04 -> 26.04
- ARG LLVM 20 -> 22 (clang builds clair)
- gcc-14 -> gcc-16: clang-22 selects the highest gcc install dir for
libstdc++ (dir 16 on 26.04); installing gcc-16 populates it so clair
links without a --gcc-install-dir pin.
- Python 3.12 -> 3.14 (26.04 default); parametrize the hardcoded
python3.12 paths via ${PYTHON_VERSION} so they don't drift again
- Replace OMPI_MCA_rmaps_base_oversubscribe=yes (removed in OpenMPI 5,
silently ignored -> np>cores fails 'not enough slots') with the PRRTE
equivalent PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe
- pip: add --break-system-packages (PEP 668 on 26.04; harmless with --target)
Validated on amd64 under emulation: base builds, clair builds+installs
(clang-22/gcc-16), Python 3.14 layout, and mpiexec oversubscribe works via
the PRRTE env only (fails without it). Full triqs compile OOMs under
qemu at -j8 (emulation memory limit, not a config issue); the native
amd64 CI build is the authoritative test.
Mirror the github_ci base bump for the openmpi image.
- FROM ubuntu:24.04 -> 26.04 (OpenMPI 5.0.10 / PRRTE; removes the ORTE
concurrent-mpirun session-dir race behind the flaky mpi_group_np4 etc.)
- ARG LLVM 20 -> 22 (main build uses clang + -stdlib=libc++)
- Add g++-16: clair is built with CXXFLAGS="" (libstdc++, not libc++), and
clang-22 selects the highest gcc install dir (16) for libstdc++; g++-16
populates it so clair links.
- Python 3.12 -> 3.14 (26.04 default); parametrize the pkgs-stage
python path via ${PYTHON_VERSION}; the base-stage pip --target predates
that ENV so it is set to 3.14 explicitly.
- Replace OMPI_MCA_rmaps_base_oversubscribe=yes (removed in OpenMPI 5) with
PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe
- pip: --break-system-packages (PEP 668 on 26.04)
Note: the openmpi image cannot be fully built locally on the arm64 host
under x86 emulation (qemu ENOSYS in wannier90's tar --xform extraction,
and triqs OOMs at make -j under emulation). Validated by analogy with the
github_ci build plus: 26.04 apt set resolves, clang-22 -stdlib=libc++
compiles/links, and the PRRTE oversubscribe env works. Authoritative test
is the native amd64 CI build on the PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Modernize the two OpenMPI-based GitHub Actions CI images to
ubuntu:26.04,which ships OpenMPI 5.0.10 (PRRTE/PMIx), clang 22, gcc 16 and Python 3.14.
This fixes intermittent MPI-test failures in the image builds, e.g.:
Root cause: OpenMPI 4's ORTE uses one per-host session directory
(
/tmp/ompi.<host>.<uid>) shared across concurrentmpiruninvocations. Withctest -j, two MPI tests run at once; one's cleanup removes the shared parentwhile another is creating its
pid.*subdir → a race that aborts a random MPItest at init. It's nondeterministic (one image happened to pass, the other
failed on the identical test). OpenMPI 5 replaces ORTE with PRRTE and no longer
has this shared-session-dir race, so
ctestcan stay parallel.Changes (both
github_ci_dockerfileandopenmpi_dockerfile)FROM ubuntu:24.04→ubuntu:26.04ARG LLVM20 → 22python3.12paths parametrizedvia
${PYTHON_VERSION}so they don't drift againOMPI_MCA_rmaps_base_oversubscribe=yeswas removed inOpenMPI 5 (silently ignored →
np>coresfails "not enough slots"). Replacedwith the PRRTE equivalent
PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe.pip installgains--break-system-packages(PEP 668 on 26.04; harmless with--target)gcc-14→gcc-16becauseclang-22 (used for clair) selects the highest gcc install dir for libstdc++
(dir 16 on 26.04) — installing gcc-16 populates it so clair links without a pin.
-stdlib=libc++, but clair is still built withlibstdc++; added
g++-16for the same reason.mpich_dockerfileis intentionally left unchanged (MPICH/Hydra has no ORTEsession-dir race).
Validation
Built locally on arm64 under x86 emulation:
and crucially
mpi_group_np2andmpi_group_np4both pass on OpenMPI 5 — thepreviously flaky tests. The one local failure (
nda_basic_array_and_view, SIGTRAPat 0.05 s, isolated among all passing neighbours) is a suspected qemu-emulation
artifact and should be confirmed on the native amd64 runner here.
ENOSYSinwannier90's
tar --xform, unrelated to these changes), but its 26.04 package setresolves,
clang-22 -stdlib=libc++compiles/links, and the PRRTE oversubscribeenv works.
Note: the Jenkins CI continues to run against Ubuntu 24.04, so this keeps
cross-distro coverage (24.04 on Jenkins, 26.04 on the GitHub images).