Test suite runs on head vs fails on compute node

[mntbighker]

I'm an OpenHPC cluster manager trying to help my team install MOOSE. I'm to the point where the test suite runs with few failures (mostly lack of CPUs). But on the compute nodes pretty much every test is skipped, and ultimately MPI errors kill the run. I'm not having much success debugging this. OpenHPC has no MOOSE meta package. Most of the MOOSE dependencies are already part of OpenHPC, but so many of the packages are built with MOOSE compile dependencies missing that we pretty much built it per the instructions. PETSc being a prime example. Some examples of things I had to add to get MOOSE tests to work: libtirpc-devel, python36-devel, and...

python -m pip install --upgrade pip
python -m pip install -t /home/shared/petsc/python3.6 pandas
python -m pip install -t /home/shared/petsc/python3.6 matplotlib
python -m pip install -t /home/shared/petsc/python3.6 sympy
python -m pip install -t /home/shared/petsc/python3.6 deepdiff
python -m pip install -t /home/shared/petsc/python3.6 numpy
python -m pip install -t /home/shared/petsc/python3.6 scipy
python -m pip install -t /home/shared/petsc/python3.6 vtk

Most of the python stuff has OpenHPC packages, that don't work with MOOSE either.

I have re-run the libmesh upgrade script a few times as these things were added, in case that was necessary.

[GiudGiud]

Hello

Could you please give us the test log for a few of the errors?
Can you run one of the failing test in an interactive job and paste the error here if the log isnt available?

Guillaume

[mntbighker]

vectorpostprocessors/work_balance.work_balance/distributed ..................... [MESH_MODE!=DISTRIBUTED] SKIP
[c1.Thor2.local:37446] [[44689,0],0] ORTE_ERROR_LOG: Data unpack had inadequate space in file util/show_help.c at line 513
[c1.Thor2.local:37446] [[44689,0],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file util/show_help.c at line 507
[c1.Thor2.local:37446] [[44689,0],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file util/show_help.c at line 501
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  getting local rank failed
  --> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_ess_init failed
  --> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "No permission" (-17) instead of "Success" (0)
--------------------------------------------------------------------------
misc/check_error.steady_no_converge: Working Directory: /home/mwmoorcroft/moose/test/tests/misc/check_error
misc/check_error.steady_no_converge: Running command: /home/mwmoorcroft/moose/test/moose_test-opt -i steady_no_converge.i --error --error-unused --error-override --no-gdb-backtrace
misc/check_error.steady_no_converge: *** An error occurred in MPI_Init_thread
misc/check_error.steady_no_converge: *** on a NULL communicator
misc/check_error.steady_no_converge: *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
misc/check_error.steady_no_converge: ***    and potentially your MPI job)
misc/check_error.steady_no_converge: [c1.Thor2.local:37573] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
misc/check_error.steady_no_converge: ################################################################################
misc/check_error.steady_no_converge:
misc/check_error.steady_no_converge: Unable to match the following pattern against the program's output:
misc/check_error.steady_no_converge:
misc/check_error.steady_no_converge: Aborting as solve did not converge
misc/check_error.steady_no_converge:
misc/check_error.steady_no_converge: ################################################################################
misc/check_error.steady_no_converge: Tester failed, reason: EXPECTED OUTPUT MISSING
misc/check_error.steady_no_converge:
misc/check_error.steady_no_converge ......................................... FAILED (EXPECTED OUTPUT MISSING)
misc/check_error.steady_no_converge: Working Directory: /home/mwmoorcroft/moose/test/tests/misc/check_error
misc/check_error.steady_no_converge: Running command: /home/mwmoorcroft/moose/test/moose_test-opt -i steady_no_converge.i --error --error-unused --error-override --no-gdb-backtrace
misc/check_error.steady_no_converge: *** An error occurred in MPI_Init_thread
misc/check_error.steady_no_converge: *** on a NULL communicator
misc/check_error.steady_no_converge: *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
misc/check_error.steady_no_converge: ***    and potentially your MPI job)
misc/check_error.steady_no_converge: [c1.Thor2.local:37561] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
misc/check_error.steady_no_converge: ################################################################################
misc/check_error.steady_no_converge:
misc/check_error.steady_no_converge: Unable to match the following pattern against the program's output:
misc/check_error.steady_no_converge:
misc/check_error.steady_no_converge: Aborting as solve did not converge
misc/check_error.steady_no_converge:
misc/check_error.steady_no_converge: ################################################################################
misc/check_error.steady_no_converge: Tester failed, reason: EXPECTED OUTPUT MISSING
misc/check_error.steady_no_converge:
misc/check_error.steady_no_converge ......................................... FAILED (EXPECTED OUTPUT MISSING)
misc/check_error.missing_mesh_test: Working Directory: /home/mwmoorcroft/moose/test/tests/misc/check_error
misc/check_error.missing_mesh_test: Running command: /home/mwmoorcroft/moose/test/moose_test-opt -i missing_mesh_test.i --error --error-unused --error-override --no-gdb-backtrace
misc/check_error.missing_mesh_test: *** An error occurred in MPI_Init_thread
misc/check_error.missing_mesh_test: *** on a NULL communicator
misc/check_error.missing_mesh_test: *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
misc/check_error.missing_mesh_test: ***    and potentially your MPI job)
misc/check_error.missing_mesh_test: [c1.Thor2.local:37576] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
misc/check_error.missing_mesh_test: ################################################################################
misc/check_error.missing_mesh_test:
misc/check_error.missing_mesh_test: Unable to match the following pattern against the program's output:
misc/check_error.missing_mesh_test:
misc/check_error.missing_mesh_test: Unable to open file \S+
misc/check_error.missing_mesh_test:
misc/check_error.missing_mesh_test: ################################################################################
misc/check_error.missing_mesh_test: Tester failed, reason: EXPECTED ERROR MISSING
misc/check_error.missing_mesh_test:
misc/check_error.missing_mesh_test ........................................... FAILED (EXPECTED ERROR MISSING)
misc/check_error.function_file_test14: Working Directory: /home/mwmoorcroft/moose/test/tests/misc/check_error
misc/check_error.function_file_test14: Running command: /home/mwmoorcroft/moose/test/moose_test-opt -i function_file_test14.i --error --error-unused --error-override --no-gdb-backtrace
misc/check_error.function_file_test14: *** An error occurred in MPI_Init_thread
misc/check_error.function_file_test14: *** on a NULL communicator
misc/check_error.function_file_test14: *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
misc/check_error.function_file_test14: ***    and potentially your MPI job)
misc/check_error.function_file_test14: [c1.Thor2.local:37560] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
misc/check_error.function_file_test14: ################################################################################
misc/check_error.function_file_test14:
misc/check_error.function_file_test14: Unable to match the following pattern against the program's output:
misc/check_error.function_file_test14:
misc/check_error.function_file_test14: In \S+: Read more than two rows of data from file '\S+'.  Did you mean to use "format = columns" or set "xy_in_file_only" to false?
misc/check_error.function_file_test14:
misc/check_error.function_file_test14: ################################################################################
misc/check_error.function_file_test14: Tester failed, reason: EXPECTED ERROR MISSING
misc/check_error.function_file_test14:
misc/check_error.function_file_test14 ........................................ FAILED (EXPECTED ERROR MISSING)
kernels/ad_max_dofs_per_elem_error.ad_max_dofs_per_elem_error ...................... [AD_MODE!=NONSPARSE] SKIP
kernels/ad_max_dofs_per_elem_error.ad_max_dofs_per_elem_error ...................... [AD_MODE!=NONSPARSE] SKIP
kernels/ad_max_dofs_per_elem_error.ad_max_dofs_per_elem_error ...................... [AD_MODE!=NONSPARSE] SKIP
kernels/ad_max_dofs_per_elem_error.ad_max_dofs_per_elem_error ...................... [AD_MODE!=NONSPARSE] SKIP
minimal_app.minimal ........................................................................ [NO DISPLAY] SKIP
minimal_app.minimal ........................................................................ [NO DISPLAY] SKIP
minimal_app.minimal ........................................................................ [NO DISPLAY] SKIP
minimal_app.minimal ........................................................................ [NO DISPLAY] SKIP
[c1.Thor2.local:37446] 3 more processes have sent help message help-orte-runtime / orte_init:startup:internal-failure
[c1.Thor2.local:37446] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[c1.Thor2.local:37446] 3 more processes have sent help message help-mpi-runtime.txt / mpi_init:startup:internal-failure
misc/check_error.dynamic_check_name_block_mismatch_test .............................................. RUNNING
misc/check_error.missing_mesh_test .................................................................

[mntbighker]

[mntbighker]

[mwmoorcroft@c4 test]$ python -m pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
aiohttp (3.8.1)
aiosignal (1.2.0)
async-timeout (4.0.2)
asynctest (0.13.0)
attrs (21.4.0)
charset-normalizer (2.0.12)
cycler (0.11.0)
deepdiff (5.7.0)
frozenlist (1.2.0)
idna (3.3)
idna-ssl (1.1.0)
kiwisolver (1.3.1)
matplotlib (3.3.4)
mpi4py (3.1.3)
mpmath (1.2.1)
multidict (5.2.0)
numpy (1.19.5)
ordered-set (4.0.2)
pandas (1.1.5)
Pillow (8.4.0)
pip (9.0.3)
pyparsing (3.0.9)
python-dateutil (2.8.2)
pytz (2022.1)
PyYAML (3.12)
scipy (1.5.4)
setuptools (39.2.0)
six (1.16.0)
sympy (1.9)
typing-extensions (4.1.1)
vtk (9.1.0)
wslink (1.6.5)
yarl (1.7.2)

[GiudGiud]

is there any errors in the parallel run?
The log is for the serial run

[mntbighker]

parallel_test.txt

[mntbighker]

Differences in python modules between head and node. I already added:

python -m pip install -t /home/shared/petsc/python3.6 pandas
python -m pip install -t /home/shared/petsc/python3.6 matplotlib
python -m pip install -t /home/shared/petsc/python3.6 sympy
python -m pip install -t /home/shared/petsc/python3.6 deepdiff
python -m pip install -t /home/shared/petsc/python3.6 numpy
python -m pip install -t /home/shared/petsc/python3.6 scipy
python -m pip install -t /home/shared/petsc/python3.6 vtk

[mwmoorcroft@thor2-head test]$ diff -w head.txt node.txt
1,2d0
< Package
< -------------------
8,11d5
< backcall
< cached-property
< cffi
< chardet
13,15d6
< ClusterShell
< configobj
< cryptography
17,18d7
< dbus-python
< decorator
20d8
< dnspython
22,24d9
< future
< gpg
< h5py
27,30d11
< ipython
< ipython-genutils
< isc
< jedi
32,33d12
< libcomps
< lxml
35d13
< mock
39d16
< nftables
41d17
< openmc
44,47d19
< parso
< perf
< pexpect
< pickleshare
50,57d21
< ply
< prompt-toolkit
< ptyprocess
< pycparser
< pycurl
< Pygments
< PyGObject
< pyOpenSSL
59,60d22
< PySocks
< python-config
62d23
< python-linux-procfs
64d24
< pyudev
66,68d25
< requests
< rpm
< schedutils
70,72d26
< selinux
< sepolicy
< setools
75,77d28
< slip
< slip.dbus
< SSSDConfig
79,84c30
< syspurpose
< systemd-python
< traitlets
< typing_extensions
< uncertainties
< urllib3
---
> typing-extensions
86d31
< wcwidth
```

[GiudGiud]

ok distributed actually doesnt look that much better, even though it was at least able to print the help message.
@milljm will know if any of these python packages end up being crucial to running the tests

[mntbighker]

I'll check, but....

[mwmoorcroft@c4 test]$ prun ./moose_test-opt -i ./tests/mesh/concentric_circle_mesh/concentric_circ
le_mesh.i --mesh-only
[prun] Master compute host = c4
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun ./moose_test-opt -i ./tests/mesh/concentric_circle_mesh/concentric_circle_mesh.i --mesh-only (family=openmpi4)
In UnstructuredMesh::stitch_meshes:
This mesh has 53 nodes on boundary 1.
Other mesh has 53 nodes on boundary 3.
Minimum edge length on both surfaces is 0.002625.
In UnstructuredMesh::stitch_meshes:
Found 53 matching nodes.

In UnstructuredMesh::stitch_meshes:
This mesh has 105 nodes on boundary 2.
Other mesh has 105 nodes on boundary 4.
Minimum edge length on both surfaces is 0.002625.
In UnstructuredMesh::stitch_meshes:
Found 105 matching nodes.

Mesh Information:
elem_dimensions()={2}
spatial_dimension()=2
n_nodes()=1225
n_local_nodes()=228
n_elem()=1212
n_local_elem()=202
n_active_elem()=1212
n_subdomains()=9
n_partitions()=6
n_processors()=6
n_threads()=1
processor_id()=0
is_prepared()=true
is_replicated()=true

Mesh Bounding Box:
Minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Maximum: (x,y,z)=(0.710315, 0.710315, 0)
Delta: (x,y,z)=( 1.42063, 1.42063, 0)

Mesh Element Type(s):
QUAD4

Mesh Nodesets:
Nodeset 1, 7 nodes
Bounding box minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Bounding box delta: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Nodeset 2, 7 nodes
Bounding box minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Bounding box delta: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Nodeset 3, 7 nodes
Bounding box minimum: (x,y,z)=(0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Bounding box delta: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Nodeset 4, 7 nodes
Bounding box minimum: (x,y,z)=(-0.710315, 0.710315, 0)
Bounding box maximum: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Bounding box delta: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)

Mesh Sidesets:
Sideset 1 (left), 6 sides (EDGE2), 6 elems (QUAD4), 7 nodes
Side volume: 1.42063
Bounding box minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(-0.710315, 0.710315, 0)
Bounding box delta: (x,y,z)=(2.22045e-16, 1.42063, 0)
Sideset 2 (bottom), 6 sides (EDGE2), 6 elems (QUAD4), 7 nodes
Side volume: 1.42063
Bounding box minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(0.710315, -0.710315, 0)
Bounding box delta: (x,y,z)=( 1.42063, 3.33067e-16, 0)
Sideset 3 (right), 6 sides (EDGE2), 6 elems (QUAD4), 7 nodes
Side volume: 1.42063
Bounding box minimum: (x,y,z)=(0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(0.710315, 0.710315, 0)
Bounding box delta: (x,y,z)=(2.22045e-16, 1.42063, 0)
Sideset 4 (top), 6 sides (EDGE2), 6 elems (QUAD4), 7 nodes
Side volume: 1.42063
Bounding box minimum: (x,y,z)=(-0.710315, 0.710315, 0)
Bounding box maximum: (x,y,z)=(0.710315, 0.710315, 0)
Bounding box delta: (x,y,z)=( 1.42063, 1.11022e-16, 0)

Mesh Edgesets:
None

Mesh Subdomains:
Subdomain 1: 276 elems (QUAD4, 276 active), 289 active nodes
Volume: 0.201323
Bounding box minimum: (x,y,z)=( -0.2546, -0.2546, 0)
Bounding box maximum: (x,y,z)=( 0.2546, 0.2546, 0)
Bounding box delta: (x,y,z)=( 0.5092, 0.5092, 0)
Subdomain 2: 144 elems (QUAD4, 144 active), 161 active nodes
Volume: 0.150984
Bounding box minimum: (x,y,z)=( -0.3368, -0.3368, 0)
Bounding box maximum: (x,y,z)=( 0.3368, 0.3368, 0)
Bounding box delta: (x,y,z)=( 0.6736, 0.6736, 0)
Subdomain 3: 96 elems (QUAD4, 96 active), 120 active nodes
Volume: 0.0502081
Bounding box minimum: (x,y,z)=( -0.36, -0.36, 0)
Bounding box maximum: (x,y,z)=( 0.36, 0.36, 0)
Bounding box delta: (x,y,z)=( 0.72, 0.72, 0)
Subdomain 4: 96 elems (QUAD4, 96 active), 120 active nodes
Volume: 0.0502251
Bounding box minimum: (x,y,z)=( -0.3818, -0.3818, 0)
Bounding box maximum: (x,y,z)=( 0.3818, 0.3818, 0)
Bounding box delta: (x,y,z)=( 0.7636, 0.7636, 0)
Subdomain 5: 96 elems (QUAD4, 96 active), 101 active nodes
Volume: 0.0252443
Bounding box minimum: (x,y,z)=( -0.3923, -0.3923, 0)
Bounding box maximum: (x,y,z)=( 0.3923, 0.3923, 0)
Bounding box delta: (x,y,z)=( 0.7846, 0.7846, 0)
Subdomain 6: 48 elems (QUAD4, 48 active), 72 active nodes
Volume: 0.0251788
Bounding box minimum: (x,y,z)=( -0.4025, -0.4025, 0)
Bounding box maximum: (x,y,z)=( 0.4025, 0.4025, 0)
Bounding box delta: (x,y,z)=( 0.805, 0.805, 0)
Subdomain 7: 48 elems (QUAD4, 48 active), 72 active nodes
Volume: 0.021476
Bounding box minimum: (x,y,z)=( -0.411, -0.411, 0)
Bounding box maximum: (x,y,z)=( 0.411, 0.411, 0)
Bounding box delta: (x,y,z)=( 0.822, 0.822, 0)
Subdomain 8: 144 elems (QUAD4, 144 active), 162 active nodes
Volume: 0.176113
Bounding box minimum: (x,y,z)=( -0.475, -0.475, 0)
Bounding box maximum: (x,y,z)=( 0.475, 0.475, 0)
Bounding box delta: (x,y,z)=( 0.95, 0.95, 0)
Subdomain 9: 264 elems (QUAD4, 264 active), 288 active nodes
Volume: 1.31744
Bounding box minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(0.710315, 0.710315, 0)
Bounding box delta: (x,y,z)=( 1.42063, 1.42063, 0)
Global mesh volume = 2.01819

[mwmoorcroft@c4 test]$ module unload pmix
[mwmoorcroft@c4 test]$ prun ./moose_test-opt -i ./tests/mesh/concentric_circle_mesh/concentric_circ
le_mesh.i --mesh-only
[prun] Master compute host = c4
[prun] Resource manager = slurm
[prun] Launch cmd = mpirun ./moose_test-opt -i ./tests/mesh/concentric_circle_mesh/concentric_circle_mesh.i --mesh-only (family=openmpi4)
In UnstructuredMesh::stitch_meshes:
This mesh has 53 nodes on boundary 1.
Other mesh has 53 nodes on boundary 3.
Minimum edge length on both surfaces is 0.002625.
In UnstructuredMesh::stitch_meshes:
Found 53 matching nodes.

In UnstructuredMesh::stitch_meshes:
This mesh has 105 nodes on boundary 2.
Other mesh has 105 nodes on boundary 4.
Minimum edge length on both surfaces is 0.002625.
In UnstructuredMesh::stitch_meshes:
Found 105 matching nodes.

Mesh Information:
elem_dimensions()={2}
spatial_dimension()=2
n_nodes()=1225
n_local_nodes()=228
n_elem()=1212
n_local_elem()=202
n_active_elem()=1212
n_subdomains()=9
n_partitions()=6
n_processors()=6
n_threads()=1
processor_id()=0
is_prepared()=true
is_replicated()=true

Mesh Bounding Box:
Minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Maximum: (x,y,z)=(0.710315, 0.710315, 0)
Delta: (x,y,z)=( 1.42063, 1.42063, 0)

Mesh Element Type(s):
QUAD4

Mesh Nodesets:
Nodeset 1, 7 nodes
Bounding box minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Bounding box delta: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Nodeset 2, 7 nodes
Bounding box minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Bounding box delta: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Nodeset 3, 7 nodes
Bounding box minimum: (x,y,z)=(0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Bounding box delta: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Nodeset 4, 7 nodes
Bounding box minimum: (x,y,z)=(-0.710315, 0.710315, 0)
Bounding box maximum: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)
Bounding box delta: (x,y,z)=(-1.79769e+308, -1.79769e+308, -1.79769e+308)

Mesh Sidesets:
Sideset 1 (left), 6 sides (EDGE2), 6 elems (QUAD4), 7 nodes
Side volume: 1.42063
Bounding box minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(-0.710315, 0.710315, 0)
Bounding box delta: (x,y,z)=(2.22045e-16, 1.42063, 0)
Sideset 2 (bottom), 6 sides (EDGE2), 6 elems (QUAD4), 7 nodes
Side volume: 1.42063
Bounding box minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(0.710315, -0.710315, 0)
Bounding box delta: (x,y,z)=( 1.42063, 3.33067e-16, 0)
Sideset 3 (right), 6 sides (EDGE2), 6 elems (QUAD4), 7 nodes
Side volume: 1.42063
Bounding box minimum: (x,y,z)=(0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(0.710315, 0.710315, 0)
Bounding box delta: (x,y,z)=(2.22045e-16, 1.42063, 0)
Sideset 4 (top), 6 sides (EDGE2), 6 elems (QUAD4), 7 nodes
Side volume: 1.42063
Bounding box minimum: (x,y,z)=(-0.710315, 0.710315, 0)
Bounding box maximum: (x,y,z)=(0.710315, 0.710315, 0)
Bounding box delta: (x,y,z)=( 1.42063, 1.11022e-16, 0)

Mesh Edgesets:
None

Mesh Subdomains:
Subdomain 1: 276 elems (QUAD4, 276 active), 289 active nodes
Volume: 0.201323
Bounding box minimum: (x,y,z)=( -0.2546, -0.2546, 0)
Bounding box maximum: (x,y,z)=( 0.2546, 0.2546, 0)
Bounding box delta: (x,y,z)=( 0.5092, 0.5092, 0)
Subdomain 2: 144 elems (QUAD4, 144 active), 161 active nodes
Volume: 0.150984
Bounding box minimum: (x,y,z)=( -0.3368, -0.3368, 0)
Bounding box maximum: (x,y,z)=( 0.3368, 0.3368, 0)
Bounding box delta: (x,y,z)=( 0.6736, 0.6736, 0)
Subdomain 3: 96 elems (QUAD4, 96 active), 120 active nodes
Volume: 0.0502081
Bounding box minimum: (x,y,z)=( -0.36, -0.36, 0)
Bounding box maximum: (x,y,z)=( 0.36, 0.36, 0)
Bounding box delta: (x,y,z)=( 0.72, 0.72, 0)
Subdomain 4: 96 elems (QUAD4, 96 active), 120 active nodes
Volume: 0.0502251
Bounding box minimum: (x,y,z)=( -0.3818, -0.3818, 0)
Bounding box maximum: (x,y,z)=( 0.3818, 0.3818, 0)
Bounding box delta: (x,y,z)=( 0.7636, 0.7636, 0)
Subdomain 5: 96 elems (QUAD4, 96 active), 101 active nodes
Volume: 0.0252443
Bounding box minimum: (x,y,z)=( -0.3923, -0.3923, 0)
Bounding box maximum: (x,y,z)=( 0.3923, 0.3923, 0)
Bounding box delta: (x,y,z)=( 0.7846, 0.7846, 0)
Subdomain 6: 48 elems (QUAD4, 48 active), 72 active nodes
Volume: 0.0251788
Bounding box minimum: (x,y,z)=( -0.4025, -0.4025, 0)
Bounding box maximum: (x,y,z)=( 0.4025, 0.4025, 0)
Bounding box delta: (x,y,z)=( 0.805, 0.805, 0)
Subdomain 7: 48 elems (QUAD4, 48 active), 72 active nodes
Volume: 0.021476
Bounding box minimum: (x,y,z)=( -0.411, -0.411, 0)
Bounding box maximum: (x,y,z)=( 0.411, 0.411, 0)
Bounding box delta: (x,y,z)=( 0.822, 0.822, 0)
Subdomain 8: 144 elems (QUAD4, 144 active), 162 active nodes
Volume: 0.176113
Bounding box minimum: (x,y,z)=( -0.475, -0.475, 0)
Bounding box maximum: (x,y,z)=( 0.475, 0.475, 0)
Bounding box delta: (x,y,z)=( 0.95, 0.95, 0)
Subdomain 9: 264 elems (QUAD4, 264 active), 288 active nodes
Volume: 1.31744
Bounding box minimum: (x,y,z)=(-0.710315, -0.710315, 0)
Bounding box maximum: (x,y,z)=(0.710315, 0.710315, 0)
Bounding box delta: (x,y,z)=( 1.42063, 1.42063, 0)
Global mesh volume = 2.01819

[mntbighker]

Ran 3232 tests in 2382.5 seconds. Average test time 3.9 seconds, maximum test time 300.0 seconds.
3230 passed, 116 skipped, 0 pending, 2 FAILED

[GiudGiud]

The ImageMeshGenerator relies on vtk libraries so you could try to install that one see if it fixes the problem.

[mntbighker]

As you can see above the vtk 9.1.0 python module is installed. So what else is needed?

[GiudGiud]

Ok it's not that. Looked at the code and it's erroring running the file command. Does this command not work on the machine?

[mntbighker]

Yeah, file wasn't in the node boot image.

[mntbighker]

For the record there has been a bug in OpenHPC (maybe it's slurm) that prevented pmix from working correctly.

Apparently they STILL have not fixed pmix.

[]openhpc/ohpc#1320

They pushed the milestone from OHPC 2.5 to OHPC 2.6. :-( The only answer is to rebuild openmpi yourself. I have have done that, but then patches don't arrive with dnf.

edit: I'm told OpenMPI PMIx may get fixed after July.