New MOOSE build on HPC - some tests failing #17975

crb3874 · 2021-05-31T20:39:05Z

crb3874
May 31, 2021

Hello,

I have set up a new installation of MOOSE on my university's HPC following the instructions at https://mooseframework.inl.gov/getting_started/installation/hpc_install_moose.html. I am able to compile the tests and run them, and something like 98% of them are passing with no issue, but a subset of them are failing. The failed tests all seem to be for similar reasons (CRASH) - when I look at their outputs, many of them exit with code 11 due to segfaults in one or more of the MPI processes. I've included the full console output from the tests, but at a glance the following tests are failing for me:

dgkernels/ad_dg_diffusion.test ................................................................ FAILED (CRASH)
dgkernels/2d_diffusion_dg.test ................................................................ FAILED (CRASH)
dgkernels/adaptivity.dg_adaptivity ............................................................ FAILED (CRASH)
bcs/dmg_periodic.check ....................................................... FAILED (EXPECTED ERROR MISSING)
bcs/periodic.all_periodic_trans_test .......................................................... FAILED (CRASH)
adaptivity/recompute_markers_during_cycles.test ............................................... FAILED (CRASH)
adaptivity/cycles_per_step.test ............................................................... FAILED (CRASH)
postprocessors/time_extreme_value.time_extreme_pps ............................................ FAILED (CRASH)
postprocessors/nodal_extreme_value.nodal_extreme .............................................. FAILED (CRASH)
controls/time_periods/dgkernels.test .......................................................... FAILED (CRASH)
mesh/periodic_node_map.1D ..................................................................... FAILED (CRASH)
markers/error_tolerance_marker.adapt_test ..................................................... FAILED (CRASH)
markers/two_circle_marker.group/two_circle_marker_coarsen ..................................... FAILED (CRASH)
meshgenerators/distributed_rectilinear/dmg_displaced_mesh.pbc_adaptivity .... FAILED (EXPECTED OUTPUT MISSING)
userobjects/toggle_mesh_adaptivity.toggle_mesh_adaptivity/toggle_mesh_adaptivity_gaussian_ic .. FAILED (CRASH)
bcs/periodic.auto_wrap_2d_test ................................................................ FAILED (CRASH)
mesh/periodic_node_map.2D ..................................................................... FAILED (CRASH)
markers/two_circle_marker.group/two_circle_marker_gaussian_ic ................................. FAILED (CRASH)
bcs/periodic.auto_wrap_2d_test_non_generated .................................................. FAILED (CRASH)
auxkernels/time_derivative.time_derivative_nl ................................................. FAILED (CRASH)
mesh/periodic_node_map.3D ..................................................................... FAILED (CRASH)
userobjects/toggle_mesh_adaptivity.toggle_mesh_adaptivity/toggle_mesh_adaptivity_gaussian_ic_stop_time FAILED (CRASH)
dgkernels/2d_diffusion_dg.no_mallocs_during_scaling ........................................... FAILED (CRASH)
bcs/periodic.auto_wrap_3d_test ................................................................ FAILED (CRASH)
mesh/named_entities.test_periodic_names ....................................................... FAILED (CRASH)
bcs/periodic.orthogonal_pbc_on_square_test .................................................... FAILED (CRASH)
bcs/dmg_periodic.check_one_step .................................................. [min_cpus=2] FAILED (CRASH)
geomsearch/patch_update_strategy.always-grid-sequencing ....................................... FAILED (CRASH)
mortar/continuity-2d-non-conforming.sequencing-stateful-soln-continuity ....................... FAILED (CRASH)
bcs/periodic.parallel_pbc_using_trans_test .................................................... FAILED (CRASH)
bcs/periodic.testlevel1 ....................................................................... FAILED (CRASH)
materials/stateful_prop.adaptivity/spatially_const ............................................ FAILED (CRASH)
outputs/exodus.discontinuous .................................................................. FAILED (CRASH)
bcs/periodic.testperiodic ..................................................................... FAILED (CRASH)
bcs/periodic.testperiodic_dp ..................................................... [min_cpus=2] FAILED (CRASH)
bcs/dmg_periodic.2d .............................................................. [min_cpus=2] FAILED (CRASH)
materials/stateful_prop.adaptivity/spatially_varying .......................................... FAILED (CRASH)
bcs/periodic.testtrapezoid .................................................................... FAILED (CRASH)
bcs/dmg_periodic.3d .............................................................. [min_cpus=2] FAILED (CRASH)
bcs/dmg_periodic.1d .............................................................. [min_cpus=2] FAILED (CRASH)
bcs/periodic.testwedge ........................................................................ FAILED (CRASH)
bcs/periodic.testwedgesys ..................................................................... FAILED (CRASH)
bcs/periodic.auto_dir_repeated_id ............................................................. FAILED (CRASH)
bcs/periodic.no_add_scalar .................................................................... FAILED (CRASH)
partitioners/petsc_partitioner.ptscotch ........................ [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)
meshgenerators/distributed_rectilinear/dmg_displaced_mesh.pbc/adaptivity_nemesis . [min_cpus=4] FAILED (CRASH)
partitioners/petsc_partitioner.ptscotch_weight_elment .......... [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)
partitioners/petsc_partitioner.ptscotch_weight_side ............ [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)
partitioners/petsc_partitioner.ptscotch_weight_both ............ [min_cpus=4] FAILED (EXPECTED OUTPUT MISSING)

In this case, 2583 of the tests passed before maximum failures was reached. Thank you for your help with this - the full console output from the tests is attached. Please let me know if any other information would help.

tests_log.txt

Answered by GiudGiud

Jun 1, 2021

Hello

Someone might know what is in common between all these tests. But if not,

Do these tests pass when ran individually?
If not, could you please compile in debug mode and get us the backtrace?

Guillaume

View full answer

GiudGiud · 2021-06-01T01:49:34Z

GiudGiud
Jun 1, 2021
Collaborator

Hello

Someone might know what is in common between all these tests. But if not,

Do these tests pass when ran individually?
If not, could you please compile in debug mode and get us the backtrace?

Guillaume

6 replies

crb3874 Jun 21, 2021
Author

I was able to get a backtrace by running one of the tests with moose_test-opt instead - I ran /tests/dgkernels/ad_dg_diffusion/2d_diffusion_ad_dg_test.i. The backtrace follows:
#0 0x00002aaaaf731f9f in libMesh::GenericProjector<libMesh::OldSolutionValue<double, &(void libMesh::FEMContext::point_value<double>(unsigned int, libMesh::Point const&, double&, double) const)>, libMesh::OldSolutionValue<libMesh::VectorValue<double>, &(void libMesh::FEMContext::point_gradient<libMesh::VectorValue<double> >(unsigned int, libMesh::Point const&, libMesh::VectorValue<double>&, double) const)>, double, libMesh::VectorSetAction<double> >::SubProjector::construct_projection(std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, unsigned int, libMesh::Node const*, libMesh::FEGenericBase<double> const&) () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0 #1 0x00002aaaaf7315dd in libMesh::GenericProjector<libMesh::OldSolutionValue<double, &(void libMesh::FEMContext::point_value<double>(unsigned int, libMesh::Point const&, double&, double) const)>, libMesh::OldSolutionValue<libMesh::VectorValue<double>, &(void libMesh::FEMContext::point_gradient<libMesh::VectorValue<double> >(unsigned int, libMesh::Point const&, libMesh::VectorValue<double>&, double) const)>, double, libMesh::VectorSetAction<double> >::ProjectInteriors::operator()(libMesh::StoredRange<__gnu_cxx::__normal_iterator<libMesh::Elem const* const*, std::vector<libMesh::Elem const*, std::allocator<libMesh::Elem const*> > >, libMesh::Elem const*> const&) () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0 #2 0x00002aaaaf7301aa in void libMesh::Threads::parallel_reduce<libMesh::StoredRange<__gnu_cxx::__normal_iterator<libMesh::Elem const* const*, std::vector<libMesh::Elem const*, std::allocator<libMesh::Elem const*> > >, libMesh::Elem const*>, libMesh::GenericProjector<libMesh::OldSolutionValue<double, &(void libMesh::FEMContext::point_value<double>(unsigned int, libMesh::Point const&, double&, double) const)>, libMesh::OldSolutionValue<libMesh::VectorValue<double>, &(void libMesh::FEMContext::point_gradient<libMesh::VectorValue<double> >(unsigned int, libMesh::Point const&, libMesh::VectorValue<double>&, double) const)>, double, libMesh::VectorSetAction<double> >::ProjectInteriors>(libMesh::StoredRange<__gnu_cxx::__normal_iterator<libMesh::Elem const* const*, std::vector<libMesh::Elem const*, std::allocator<libMesh::Elem const*> > >, libMesh::Elem const*> const&, libMesh::GenericProjector<libMesh::OldSolutionValue<double, &(void libMesh::FEMContext::point_value<double>(unsigned int, libMesh::Point const&, double&, double) const)>, libMesh::OldSolutionValue<libMesh::VectorValue<double>, &(void libMesh::FEMContext::point_gradient<libMesh::VectorValue<double> >(unsigned int, libMesh::Point const&, libMesh::VectorValue<double>&, double) const)>, double, libMesh::VectorSetAction<double> >::ProjectInteriors&) () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0 #3 0x00002aaaaf727513 in libMesh::GenericProjector<libMesh::OldSolutionValue<double, &(void libMesh::FEMContext::point_value<double>(unsigned int, libMesh::Point const&, double&, double) const)>, libMesh::OldSolutionValue<libMesh::VectorValue<double>, &(void libMesh::FEMContext::point_gradient<libMesh::VectorValue<double> >(unsigned int, libMesh::Point const&, libMesh::VectorValue<double>&, double) const)>, double, libMesh::VectorSetAction<double> >::project(libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> const&) () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0 #4 0x00002aaaaf6ead0d in libMesh::System::project_vector(libMesh::NumericVector<double> const&, libMesh::NumericVector<double>&, int) const () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0 #5 0x00002aaaaf6e9ec1 in libMesh::System::project_vector(libMesh::NumericVector<double>&, int) const () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0 #6 0x00002aaaaf6bef3b in libMesh::System::restrict_vectors() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0 #7 0x00002aaaaf667c72 in libMesh::EquationSystems::reinit_solutions() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0 #8 0x00002aaaacdd8c20 in FEProblemBase::meshChangedHelper(bool) () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0 #9 0x00002aaaacdd870e in FEProblemBase::adaptMesh() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0 #10 0x00002aaaad919273 in Steady::execute() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0 #11 0x00002aaaadc91ed3 in MooseApp::executeExecutioner() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0 #12 0x00002aaaab89ce85 in MooseTestApp::executeExecutioner() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/test/lib/libmoose_test-opt.so.0 #13 0x00002aaaadc888b3 in MooseApp::run() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0 #14 0x00000000004043a0 in main ()

crb3874 Jun 21, 2021
Author

I ran another one of the tests to obtain a backtrace - this time, ./tests/bcs/dmg_periodic/dmg_periodic_bc.i. I'm not sure what is common between these - I can get backtraces from more of them if needed.

#0 0x00002aaaaeb3fc21 in libMesh::FEGenericBase::compute_periodic_constraints(libMesh::DofConstraints&, libMesh::DofMap&, libMesh::PeriodicBoundaries const&, libMesh::MeshBase const&, libMesh::PointLocatorBase const*, unsigned int, libMesh::Elem const*) () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0
#1 0x00002aaaaed37ab1 in libMesh::FEInterface::compute_periodic_constraints(libMesh::DofConstraints&, libMesh::DofMap&, libMesh::PeriodicBoundaries const&, libMesh::MeshBase const&, libMesh::PointLocatorBase const*, unsigned int, libMesh::Elem const*) () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0
#2 0x00002aaaae97954c in (anonymous namespace)::ComputeConstraints::operator()(libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> const&) const ()
from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0
#3 0x00002aaaae978b42 in void libMesh::Threads::parallel_for<libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*>, (anonymous namespace)::ComputeConstraints>(libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> const&, (anonymous namespace)::ComputeConstraints const&) ()
from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0
#4 0x00002aaaae977b41 in libMesh::DofMap::create_dof_constraints(libMesh::MeshBase const&, double) ()
from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0
#5 0x00002aaaaf6c4fdc in libMesh::System::reinit_constraints() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0
#6 0x00002aaaaf6c4243 in libMesh::System::init_data() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0
#7 0x00002aaaaf6c50fb in libMesh::System::init() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0
#8 0x00002aaaaf66a5ce in libMesh::EquationSystems::init() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/scripts/../libmesh/installed/lib/libmesh_opt.so.0
#9 0x00002aaaace17957 in FEProblemBase::init() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0
#10 0x00002aaaad5edac6 in InitProblemAction::act() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0
#11 0x00002aaaad53c318 in Action::timedAct() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0
#12 0x00002aaaad53bf9c in ActionWarehouse::executeActionsWithAction(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) ()
from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0
#13 0x00002aaaad53b9b5 in ActionWarehouse::executeAllActions() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0
#14 0x00002aaaadc91647 in MooseApp::runInputFile() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0
#15 0x00002aaaadc8887a in MooseApp::run() () from /work2/07954/cbrennan/MOOSE_Shared/moose_proj/moose/framework/libmoose-opt.so.0
#16 0x00000000004043a0 in main ()

GiudGiud Jun 21, 2021
Collaborator

These are linked to libmesh.
Does the devel build pass the tests?
It s very odd that opt would fail and dbg would pass.

GiudGiud Jun 24, 2021
Collaborator

I would also rebuild everything (at least on the moose side, with make clobberall then make) then try again with opt

GiudGiud Jul 25, 2021
Collaborator

Closing this.
If debug passes and opt fails, it's likely that an old library got linked, and that it needs to be rebuilt.
make clobberall OR commit everything then git clean -xfd will take care of that (the second better than the first).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New MOOSE build on HPC - some tests failing #17975

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 6 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

New MOOSE build on HPC - some tests failing #17975

Uh oh!

crb3874 May 31, 2021

Replies: 1 comment · 6 replies

Uh oh!

GiudGiud Jun 1, 2021 Collaborator

Uh oh!

Uh oh!

crb3874 Jun 21, 2021 Author

Uh oh!

Uh oh!

crb3874 Jun 21, 2021 Author

Uh oh!

GiudGiud Jun 21, 2021 Collaborator

Uh oh!

GiudGiud Jun 24, 2021 Collaborator

Uh oh!

GiudGiud Jul 25, 2021 Collaborator

crb3874
May 31, 2021

Replies: 1 comment 6 replies

GiudGiud
Jun 1, 2021
Collaborator

crb3874 Jun 21, 2021
Author

crb3874 Jun 21, 2021
Author

GiudGiud Jun 21, 2021
Collaborator

GiudGiud Jun 24, 2021
Collaborator

GiudGiud Jul 25, 2021
Collaborator