Some tests fail

[bylore]

Hi, I am building moose on HPC cluster. Follwing the construction, I arrive at the test step. After excuting the ./run_test -j 6. I get the following.

mesh/split_uniform_refine.square ................................................. [min_cpus=3] FAILED (CRASH)
mesh/mesh_only.mesh_only_checkpoint .............................................. [min_cpus=3] FAILED (CRASH)
postprocessors/num_elems.test_split .............................................. [min_cpus=4] FAILED (CRASH)
mesh/nemesis.nemesis_repartitioning_test ......................................... [min_cpus=4] FAILED (CRASH)
partitioners/petsc_partitioner.ptscotch ........................ [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)
partitioners/petsc_partitioner.ptscotch_weight_elment .......... [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)
postprocessors/num_nodes.test_split .............................................. [min_cpus=4] FAILED (CRASH)
partitioners/petsc_partitioner.ptscotch_weight_side ............ [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)
partitioners/petsc_partitioner.ptscotch_weight_both ............ [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)
partitioners/petsc_partitioner.parmetis ........................ [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)
partitioners/petsc_partitioner.parmetis_weight_element ......... [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)
partitioners/petsc_partitioner.parmetis_weight_side ............ [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)
partitioners/petsc_partitioner.parmetis_weight_both ............ [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)

---

Ran 3919 tests in 526.5 seconds. Average test time 0.7 seconds, maximum test time 23.6 seconds.
3906 passed, 200 skipped, 0 pending, 13 FAILED

Do you have any advice? Can I just ignore thest failures, or what these failure mean? Thank you!

[bylore]

It seems these failures related to my mpich (4.0.2) mpich/install/lib/libucs.so.0

[bylore]

configure --with-ucx=embedded

[milljm]

If you happen to have the exact errors produced, can you paste those?

[bylore]

run_tests --re=mesh/split_uniform_refine.square -v for example.

mesh/split_uniform_refine.square: Working Directory: */moose/test/tests/mesh/split_uniform_refine
mesh/split_uniform_refine.square: Running command: mpiexec -n 3 */moose/test/moose_test-opt -i 3d_diffusion.i -r 1 Mesh/skip_refine_when_use_split=false --use-split --split-file square --error --error-override --no-gdb-backtrace
mesh/split_uniform_refine.square:
mesh/split_uniform_refine.square: ===================================================================================
mesh/split_uniform_refine.square: = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
mesh/split_uniform_refine.square: = PID 21523 RUNNING AT ln3
mesh/split_uniform_refine.square: = EXIT CODE: 139
mesh/split_uniform_refine.square: = CLEANING UP REMAINING PROCESSES
mesh/split_uniform_refine.square: = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
mesh/split_uniform_refine.square: ===================================================================================
mesh/split_uniform_refine.square: YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
mesh/split_uniform_refine.square: This typically refers to a problem with your application.
mesh/split_uniform_refine.square: Please see the FAQ page for debugging suggestions
mesh/split_uniform_refine.square: [ln3:21523:0:21523] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
mesh/split_uniform_refine.square: ==== backtrace (tid: 21523) ====
mesh/split_uniform_refine.square: 0 /data/home/yongzhang/software/mpich/install/lib/libucs.so.0(ucs_handle_error+0x2a4) [0x7f087cbd3674]
mesh/split_uniform_refine.square: 1 /mpich/install/lib/libucs.so.0(+0x2f86c) [0x7f087cbd386c]
mesh/split_uniform_refine.square: 2 /mpich/install/lib/libucs.so.0(+0x2fb36) [0x7f087cbd3b36]
mesh/split_uniform_refine.square: 3 /lib64/libpthread.so.0(+0xf630) [0x7f087c56f630]
mesh/split_uniform_refine.square: 4 */moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0(+0xcb4388) [0x7f0886a65388]
mesh/split_uniform_refine.square: 5 */moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0(_ZN7libMesh11Partitioner22set_node_processor_idsERNS_8MeshBaseE+0x10c1) [0x7f0886a71431]
mesh/split_uniform_refine.square: 6 */moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0(_ZN7libMesh11Partitioner9partitionERNS_8MeshBaseEj+0x61) [0x7f0886a71b91]
mesh/split_uniform_refine.square: 7 */moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0(_ZN7libMesh8MeshBase15prepare_for_useEv+0x6a) [0x7f08867da6ea]
mesh/split_uniform_refine.square: 8 */moose/framework/libmoose-opt.so.0(_ZN10Adaptivity13uniformRefineEP9MooseMeshj+0x5a) [0x7f088af1fb2a]
mesh/split_uniform_refine.square: 9 */moose/framework/libmoose-opt.so.0(_ZN23SetupMeshCompleteAction3actEv+0x3e8) [0x7f088ac17278]
mesh/split_uniform_refine.square: 10 */moose/framework/libmoose-opt.so.0(_ZN6Action8timedActEv+0x2b) [0x7f088ac1470b]
mesh/split_uniform_refine.square: 11 */moose/framework/libmoose-opt.so.0(_ZN15ActionWarehouse24executeActionsWithActionERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x51) [0x7f088ac20451]
mesh/split_uniform_refine.square: 12 */moose/framework/libmoose-opt.so.0(_ZN15ActionWarehouse17executeAllActionsEv+0x1a1) [0x7f088ac20ad1]
mesh/split_uniform_refine.square: 13 */moose/framework/libmoose-opt.so.0(_ZN8MooseApp12runInputFileEv+0x88) [0x7f088aed9428]
mesh/split_uniform_refine.square: 14 */moose/framework/libmoose-opt.so.0(_ZN8MooseApp3runEv+0x302) [0x7f088aed8b02]
mesh/split_uniform_refine.square: 15 */moose/test/moose_test-opt(main+0x6f) [0x40236f]
mesh/split_uniform_refine.square: 16 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f087ab21555]
mesh/split_uniform_refine.square: 17 */moose/test/moose_test-opt() [0x4025c1]
mesh/split_uniform_refine.square: =================================
mesh/split_uniform_refine.square:
mesh/split_uniform_refine.square: ===================================================================================
mesh/split_uniform_refine.square: = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
mesh/split_uniform_refine.square: = PID 21523 RUNNING AT ln3
mesh/split_uniform_refine.square: = EXIT CODE: 139
mesh/split_uniform_refine.square: = CLEANING UP REMAINING PROCESSES
mesh/split_uniform_refine.square: = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
mesh/split_uniform_refine.square: ===================================================================================
mesh/split_uniform_refine.square: YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
mesh/split_uniform_refine.square: This typically refers to a problem with your application.
mesh/split_uniform_refine.square: Please see the FAQ page for debugging suggestions
mesh/split_uniform_refine.square: [ln3:21523:0:21523] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
mesh/split_uniform_refine.square: ==== backtrace (tid: 21523) ====
mesh/split_uniform_refine.square: 0 /install/lib/libucs.so.0(ucs_handle_error+0x2a4) [0x7f087cbd3674]
mesh/split_uniform_refine.square: 1 /install/lib/libucs.so.0(+0x2f86c) [0x7f087cbd386c]
mesh/split_uniform_refine.square: 2 /install/lib/libucs.so.0(+0x2fb36) [0x7f087cbd3b36]
mesh/split_uniform_refine.square: 3 /lib64/libpthread.so.0(+0xf630) [0x7f087c56f630]
mesh/split_uniform_refine.square: 4 */moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0(+0xcb4388) [0x7f0886a65388]
mesh/split_uniform_refine.square: 5 */moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0(_ZN7libMesh11Partitioner22set_node_processor_idsERNS_8MeshBaseE+0x10c1) [0x7f0886a71431]
mesh/split_uniform_refine.square: 6 */moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0(_ZN7libMesh11Partitioner9partitionERNS_8MeshBaseEj+0x61) [0x7f0886a71b91]
mesh/split_uniform_refine.square: 7 */moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0(_ZN7libMesh8MeshBase15prepare_for_useEv+0x6a) [0x7f08867da6ea]
mesh/split_uniform_refine.square: 8 */moose/framework/libmoose-opt.so.0(_ZN10Adaptivity13uniformRefineEP9MooseMeshj+0x5a) [0x7f088af1fb2a]
mesh/split_uniform_refine.square: 9 */moose/framework/libmoose-opt.so.0(_ZN23SetupMeshCompleteAction3actEv+0x3e8) [0x7f088ac17278]
mesh/split_uniform_refine.square: 10 */moose/framework/libmoose-opt.so.0(_ZN6Action8timedActEv+0x2b) [0x7f088ac1470b]
mesh/split_uniform_refine.square: 11 */moose/framework/libmoose-opt.so.0(_ZN15ActionWarehouse24executeActionsWithActionERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x51) [0x7f088ac20451]
mesh/split_uniform_refine.square: 12 */moose/framework/libmoose-opt.so.0(_ZN15ActionWarehouse17executeAllActionsEv+0x1a1) [0x7f088ac20ad1]
mesh/split_uniform_refine.square: 13 */moose/framework/libmoose-opt.so.0(_ZN8MooseApp12runInputFileEv+0x88) [0x7f088aed9428]
mesh/split_uniform_refine.square: 14 */moose/framework/libmoose-opt.so.0(_ZN8MooseApp3runEv+0x302) [0x7f088aed8b02]
mesh/split_uniform_refine.square: 15 */moose/test/moose_test-opt(main+0x6f) [0x40236f]
mesh/split_uniform_refine.square: 16 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f087ab21555]
mesh/split_uniform_refine.square: 17 */moose/test/moose_test-opt() [0x4025c1]
mesh/split_uniform_refine.square: =================================
mesh/split_uniform_refine.square:
mesh/split_uniform_refine.square:
mesh/split_uniform_refine.square: Exit Code: 139
mesh/split_uniform_refine.square: ################################################################################
mesh/split_uniform_refine.square: Tester failed, reason: CRASH
mesh/split_uniform_refine.square:
mesh/split_uniform_refine.square ....................................... [min_cpus=3,OVERSIZED] FAILED (CRASH)

[bylore]

I just tried mpich-4.1.2 configure --with-ucx=embeded, and these failure tests all passed! So there seems to be some mismatch between mpich-4.0.2 (--with-ucx=embeded) and the latest moose(2023.11.09).