Skip to content

[ci] Modernize OpenMPI CI images to Ubuntu 26.04 / OpenMPI 5 (fix flaky mpi_group ORTE session-dir race) - #144

Merged
the-hampel merged 2 commits into
unstablefrom
ci-ubuntu2604-openmpi5
Jul 24, 2026
Merged

[ci] Modernize OpenMPI CI images to Ubuntu 26.04 / OpenMPI 5 (fix flaky mpi_group ORTE session-dir race)#144
the-hampel merged 2 commits into
unstablefrom
ci-ubuntu2604-openmpi5

Conversation

@the-hampel

Copy link
Copy Markdown
Member

Summary

Modernize the two OpenMPI-based GitHub Actions CI images to ubuntu:26.04,
which ships OpenMPI 5.0.10 (PRRTE/PMIx), clang 22, gcc 16 and Python 3.14.

This fixes intermittent MPI-test failures in the image builds, e.g.:

26/377 Test #27: mpi_group_np4 ...***Failed
  mkdir /tmp/ompi.buildkitsandbox.0/pid.XXXX : No such file or directory
  orte_session_dir failed  (session_dir.c:107)

Root cause: OpenMPI 4's ORTE uses one per-host session directory
(/tmp/ompi.<host>.<uid>) shared across concurrent mpirun invocations. With
ctest -j, two MPI tests run at once; one's cleanup removes the shared parent
while another is creating its pid.* subdir → a race that aborts a random MPI
test at init. It's nondeterministic (one image happened to pass, the other
failed on the identical test). OpenMPI 5 replaces ORTE with PRRTE and no longer
has this shared-session-dir race, so ctest can stay parallel.

Changes (both github_ci_dockerfile and openmpi_dockerfile)

  • FROM ubuntu:24.04ubuntu:26.04
  • ARG LLVM 20 → 22
  • Python 3.12 → 3.14 (26.04 default); hardcoded python3.12 paths parametrized
    via ${PYTHON_VERSION} so they don't drift again
  • Oversubscribe env: OMPI_MCA_rmaps_base_oversubscribe=yes was removed in
    OpenMPI 5
    (silently ignored → np>cores fails "not enough slots"). Replaced
    with the PRRTE equivalent PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe.
  • pip install gains --break-system-packages (PEP 668 on 26.04; harmless with --target)
  • github_ci builds the stack with gcc; bumped gcc-14gcc-16 because
    clang-22 (used for clair) selects the highest gcc install dir for libstdc++
    (dir 16 on 26.04) — installing gcc-16 populates it so clair links without a pin.
  • openmpi builds with clang + -stdlib=libc++, but clair is still built with
    libstdc++; added g++-16 for the same reason.

mpich_dockerfile is intentionally left unchanged (MPICH/Hydra has no ORTE
session-dir race).

Validation

Built locally on arm64 under x86 emulation:

  • github_ci image builds through the full triqs compile; 376/377 tests pass,
    and crucially mpi_group_np2 and mpi_group_np4 both pass on OpenMPI 5 — the
    previously flaky tests. The one local failure (nda_basic_array_and_view, SIGTRAP
    at 0.05 s, isolated among all passing neighbours) is a suspected qemu-emulation
    artifact and should be confirmed on the native amd64 runner here.
  • openmpi image can't be fully built under emulation (qemu ENOSYS in
    wannier90's tar --xform, unrelated to these changes), but its 26.04 package set
    resolves, clang-22 -stdlib=libc++ compiles/links, and the PRRTE oversubscribe
    env works.

Note: the Jenkins CI continues to run against Ubuntu 24.04, so this keeps
cross-distro coverage (24.04 on Jenkins, 26.04 on the GitHub images).

Bump the github_ci base image to ubuntu:26.04, which ships OpenMPI 5.0.10
(PRRTE/PMIx). OpenMPI 5 removes the ORTE per-host session directory that
concurrent mpirun invocations (ctest -j) raced on, producing flaky
'orte_session_dir failed / mkdir ... No such file or directory' aborts
in MPI tests (e.g. mpi_group_np4).

- FROM ubuntu:24.04 -> 26.04
- ARG LLVM 20 -> 22 (clang builds clair)
- gcc-14 -> gcc-16: clang-22 selects the highest gcc install dir for
  libstdc++ (dir 16 on 26.04); installing gcc-16 populates it so clair
  links without a --gcc-install-dir pin.
- Python 3.12 -> 3.14 (26.04 default); parametrize the hardcoded
  python3.12 paths via ${PYTHON_VERSION} so they don't drift again
- Replace OMPI_MCA_rmaps_base_oversubscribe=yes (removed in OpenMPI 5,
  silently ignored -> np>cores fails 'not enough slots') with the PRRTE
  equivalent PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe
- pip: add --break-system-packages (PEP 668 on 26.04; harmless with --target)

Validated on amd64 under emulation: base builds, clair builds+installs
(clang-22/gcc-16), Python 3.14 layout, and mpiexec oversubscribe works via
the PRRTE env only (fails without it). Full triqs compile OOMs under
qemu at -j8 (emulation memory limit, not a config issue); the native
amd64 CI build is the authoritative test.
Mirror the github_ci base bump for the openmpi image.

- FROM ubuntu:24.04 -> 26.04 (OpenMPI 5.0.10 / PRRTE; removes the ORTE
  concurrent-mpirun session-dir race behind the flaky mpi_group_np4 etc.)
- ARG LLVM 20 -> 22 (main build uses clang + -stdlib=libc++)
- Add g++-16: clair is built with CXXFLAGS="" (libstdc++, not libc++), and
  clang-22 selects the highest gcc install dir (16) for libstdc++; g++-16
  populates it so clair links.
- Python 3.12 -> 3.14 (26.04 default); parametrize the pkgs-stage
  python path via ${PYTHON_VERSION}; the base-stage pip --target predates
  that ENV so it is set to 3.14 explicitly.
- Replace OMPI_MCA_rmaps_base_oversubscribe=yes (removed in OpenMPI 5) with
  PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe
- pip: --break-system-packages (PEP 668 on 26.04)

Note: the openmpi image cannot be fully built locally on the arm64 host
under x86 emulation (qemu ENOSYS in wannier90's tar --xform extraction,
and triqs OOMs at make -j under emulation). Validated by analogy with the
github_ci build plus: 26.04 apt set resolves, clang-22 -stdlib=libc++
compiles/links, and the PRRTE oversubscribe env works. Authoritative test
is the native amd64 CI build on the PR.
@the-hampel
the-hampel merged commit 69d5bd5 into unstable Jul 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant