Error when testing MOOSE on the UCSD Expanse cluster

[lichanghao]

Hi there,

I tried to use mamba to configure the MOOSE compiling environment and the compilation can pass on the Expanse cluster. However, when I run the tests, multiple items cannot pass due to similar error for the mpiexec:

pre
misc/mpi_setup.basic_mpirun_works: Working Directory: /home/cli1/moose_projects/moose/test/tests/misc/mpi_setup
misc/mpi_setup.basic_mpirun_works: Running command: mpirun -n 2 hostname
misc/mpi_setup.basic_mpirun_works: [mpiexec@exp-7-15] HYDU_sock_write (utils/sock/sock.c:254): write error (Bad file descriptor)
misc/mpi_setup.basic_mpirun_works: [mpiexec@exp-7-15] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:176): unable to write data to proxy
misc/mpi_setup.basic_mpirun_works: [mpiexec@exp-7-15] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:42): unable to send signal downstream
misc/mpi_setup.basic_mpirun_works: [mpiexec@exp-7-15] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
misc/mpi_setup.basic_mpirun_works: [mpiexec@exp-7-15] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:160): error waiting for event
misc/mpi_setup.basic_mpirun_works: [mpiexec@exp-7-15] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion
misc/mpi_setup.basic_mpirun_works: 
misc/mpi_setup.basic_mpirun_works ........................................................... FAILED (TIMEOUT)

If you can give me a hint on how to deal with this error, or you need me to provide further details, just let me know. Thank you so much for your help.

[milljm]

Its almost as if you are not "allowed" to use MPI. You might be constrained to only use the provided MPI wrapper designed for that cluster (do not use Mamba while on HPC). This may be a "SLURM" cluster for example. Where you are only allowed to run in parallel with srun (instead of mpirun).

One trick that might work, if this is indeed a SLURM cluster:

export MOOSE_MPI_COMMAND=`srun`

And then attempt to run tests.

Another possible reason for the error; most clusters will want you to be operating on a compute node, instead of a head node. I cannot tell which one you are using.

[lichanghao]

Thank you for the rapid reply. I double checked and the HPC environment does not allow explicit call of mpirun, and srun can be only used to apply interactive nodes.

Now I use the HPC-installed compilers to recompile moose, then submit the parallel job by the slurm script. Everything works well.