COMM_WORLD.Get_size() returns 1 reguardless of num threads launched on Github Runner

[jpmorgan98]

mpi4py in a Github runner seems to have stopped being able to allocate multiple threads. We have tests that need exactly 4 threads to pass and mpi4py from setup-mpi seems to have stopped allocating anything other then 1, even tho it will still execute a given script n times. For an example I am attaching a runner script, a sample mpi4py test, the incorrect, and the expected outputs.

For some more background see our issue tracking this problem

Thanks in advance!

mpi_error.yml

name: MPI CI Error

on: [push, pull_request]

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
        fail-fast: false
        matrix:
            os: ["ubuntu-latest"]
    steps:
    - uses: actions/checkout@v3
    - name: Set up python 3.11
      uses: actions/setup-python@v3
      with:
        python-version: "3.11"
    - name: debug
      run: |
        pwd
        ls
    - uses: mpi4py/setup-mpi@v1
    - name: Install dependencies
      run: |
        pip install mpi4py
    - name: Run MPI
      run : |
        lscpu
        mpiexec -n 4 python -m mpi4py.bench helloworld
        mpiexec -n 4 python test_mpi.py

test_mpi.py

from mpi4py import MPI
print(MPI.COMM_WORLD.Get_size())

output when github runner CI is used

Hello, World! I am process 0 of 1 on fv-az1308-741.
Hello, World! I am process 0 of 1 on fv-az1308-741.
Hello, World! I am process 0 of 1 on fv-az1308-741.
Hello, World! I am process 0 of 1 on fv-az1308-741.

1
1
1
1

The expected output (commands run on a local machine configured with mpi4py)

Hello, World! I am process 0 of 4 on 
Hello, World! I am process 1 of 4 on 
Hello, World! I am process 2 of 4 on 
Hello, World! I am process 3 of 4 on 

4
4
4
4

[dalcinl]

There is nothing I can do about it. The root issue is the broken MPICH package in Ubuntu 24.04, six month after and the issue is still unresolved. IIRC, the required fix is available in 24.10, but it has not been backported to the 24.04 LTS release. This issue effectively renders MPICH unusable on Ubuntu 22.04.
cf. https://bugs.launchpad.net/ubuntu/+source/mpich/+bug/2072338

The only thing you can do for the time being is replacing ubuntu-latest -> ubuntu-22.04 in your workflow file.

[TysonRayJones]

I wish this were advertised on the README while the bug is active, or that the setup script issued a warning about it - or even that this issue wasn't closed! Quite nasty to be bitten by one's MPI job silently duplicating.

[dalcinl]

@TysonRayJones I usually close issues about things that a) I'm not responsible off, and b) I do not plan to take active action. Generally speaking, the issue trackers in my projects are to help me to organize my (for free) open-source work, not to help others with their own business. 90% of my open-source work is related to dealing with third-party bugs, and I'm sick of it. I wish people spend their time contributing towards solutions, rather than dropping very politically correct and nicely written complaints about stuff I done for free in my spare time I should rather dedicate to my family.

Once again, I must clarify: All the work I do in the mpi4py organization is for my own amusement and enlightenment. I have no obligations to any individual or organization. When you use open-source software written by individuals like me, you definitely get more value than what you pay for, so entitled complaints are totally out place. Those who do not like these simple facts are most welcome to fork my BSD-license repositories and adapt things to whatever workflow fit their needs, and even make everything open-source again for the benefit of the world and deal with the maintenance nightmares.

All that being said, I'm reopening this issue. I'm annoyed enough about this bug that will try to do something about it on my side for the need of my projects. I've been working on it during the last couple weeks, still deciding where to install MPICH from a Ubuntu PPA from a newer release, or use a conda-forge build. There is no ETA, though.

[TysonRayJones]

@dalcinl I totally get the sentiment and your frustration is justified - sincere apologies if I've contributed to it! Your GA is super useful, hence its integration in other free and open-source work, and its role in a community of developers trying to help one another (in which you may be a heavy-lifter!). I agree there is no place for entitled complaints, I hope mine didn't come across as such! I believe it remains worthwhile to warn about insidious problems that users of the GA may be harmed by, even when they're not your fault at all, nor your responsibility to address. Not as a matter of liability, but of community benefit.

I think re-opening the issue is a great idea. Although it's very noble you're willing to try solve the problem, it is (as you point out) not your responsibility and might be a poor use of time. I was content myself to roll-back the runner version to ubuntu-22.04 as you proposed. The only issue remains discoverability of this solution - one could consider a README notice like

Caution

A bug in MPICH on ubuntu-24.04 (presently ubuntu-latest) makes mpiexec and mpirun launch duplicated, single-node jobs, despite compilation succeeding. Avoid this by explicitly choosing 'openmpi' or 'intelmpi', or using ubuntu-22.04.

Of course the issue being open helps discoverability a lot - it's the first place many look when finding strange behaviour.

In any case, thanks for all your brilliant efforts on this GA so far!