Error compile MOOSE-based application on Lonestar6 cluster

[chunhuizhao478]

Check these boxes if you have followed the posting rules.

• I have consulted the Posting Guidelines.
• I have searched the Discussions Forum and MOOSE Framework Troubleshooting and have not found what I was looking for
• Q&A Getting Started is the most appropriate category for my question (trouble installing, beginner user, ...)

Issue or question about MOOSE

Hi all, I tried to build moose following the assistant help from Lonestar6 using the following bash script:

#!/bin/bash

ml reset
ml gcc/11.2.0
ml impi/19.0.9
ml cuda/12.0
ml eigen/3.4.0
ml hdf5/1.14.6
ml netcdf/4.9.2 
ml cmake/3.29.5
#ml petsc/3.23

echo $CC $CXX $FC $F90 $F77
export CC=mpicc CXX=mpicxx FC=mpif90 F90=mpif90 F77=mpif77

# CLONE MOOSE REPO

MOOSE_ROOT=`pwd`/moose-src

if [[ 1 == 1 ]]; then
git clone https://github.com/idaholab/moose moose-src

  cd moose-src
  git checkout master
fi 

# BUILD PETSC

export MOOSE_JOBS=6 METHODS=opt
export HDF5_DIR=$TACC_HDF5_DIR
#export PETSC_DIR=$TACC_PETSC_DIR

if [[ 1 == 1 ]]; then
  cd $MOOSE_ROOT
  bash scripts/update_and_rebuild_petsc.sh
fi

# BUILD LIBMESH

export libmesh_CXXFLAGS="-fpermissive "

if [[ 1 == 1 ]]; then
  cd $MOOSE_ROOT
  bash scripts/update_and_rebuild_libmesh.sh
  #bash scripts/update_and_rebuild_libmesh.sh --with-petsc=$TACC_PETSC_DIR
fi

# BUILD WASP

export PATH=`pwd`/petsc/arch-moose/bin:$PATH

if [[ 1 == 1 ]]; then
  cd $MOOSE_ROOT
  bash scripts/update_and_rebuild_wasp.sh
fi

# BUILD MOOSE TEST

if [[ 1 == 1 ]]; then
  cd $MOOSE_ROOT/test
  make -j $MOOSE_JOBS
fi

MOOSE is able to build without error, however after I try to compile my own application, it returned errors about libmesh, I wonder if this indicates the moose build actually fails and how to have a workaround with it.

login2.ls6(563)$ make -j 4
Rebuilding symlinks in /work/10024/zhaochun/ls6/projects/farms_cdms/build/header_symlinks
Symlinking MOOSE configure /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/MooseConfig.h
Creating Unity Directory /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/interfacekernels_slipweakening3d_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/interfacekernels_ratestate_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/timesteppers_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/kernels_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/kernels_adnonlocaldamage_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/kernels_core_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/kernels_threefieldporodynamics_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/kernels_twofieldHM_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/kernels_nonlocaldamage_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/kernels_phasefield_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/kernels_cdbm_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/interfaces_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/bcs_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/bcs_core_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/markers_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/bounds_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/auxkernels_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/auxkernels_core_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/auxkernels_energybudget_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/auxkernels_adaptivity_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/auxkernels_slipweakening3d_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/auxkernels_cdbm_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/base_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/utils_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/postprocessors_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/userobjects_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/userobjects_core_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/userobjects_nonlocaldamage_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/functions_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/functions_pulsepower_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/functions_slipweakening_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/functions_ratestate2d_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/functions_ratestate3d_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/functions_cdbm_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_porousflowmatprops_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_threefieldporodynamics_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_phasefield_function_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_phasefield_smalldeform_nonAD_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_phasefield_function_nonAD_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_phasefield_hardening_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_slipweakeningczm2d_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_slipweakening_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_phasefield_largedeform_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_ratestate2d_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_nonlocaldamage_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_cdbm_explicit_3d_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_ratestate3d_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_slipweakening3d_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_phasefield_smalldeform_Unity.C
Creating Unity /work/10024/zhaochun/ls6/projects/farms_cdms/build/unity_src/materials_cdbm_Unity.C
Compiling C++ (in opt mode) /work/10024/zhaochun/ls6/projects/farms_cdms/src/main.C...
Compiling C++ (in opt mode) /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/FDCompVarRatev2.C...
Compiling C++ (in opt mode) /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/CompTJumpRateState.C...
Compiling C++ (in opt mode) /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/CompDispRateState.C...
icpc: command line warning #10006: ignoring unknown option '-felide-constructors'
icpc: command line warning #10006: ignoring unknown option '-felide-constructors'
In file included from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/Moose.h(12),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/MooseApp.h(27),
                 from /work/10024/zhaochun/ls6/projects/farms_cdms/build/header_symlinks/farmsTestApp.h(12),
                 from /work/10024/zhaochun/ls6/projects/farms_cdms/src/main.C(10):
/work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/libMeshReducedNamespace.h(16): catastrophic error: cannot open source file "libmesh/libmesh.h"
  #include "libmesh/libmesh.h"
                              ^

compilation aborted for /work/10024/zhaochun/ls6/projects/farms_cdms/src/main.C (code 4)
make: *** [/work/10024/zhaochun/ls6/projects/moose/framework/build.mk:159: /work/10024/zhaochun/ls6/projects/farms_cdms/src/main.x86_64-pc-linux-gnu.opt.lo] Error 1
make: *** Waiting for unfinished jobs....
icpc: command line warning #10006: ignoring unknown option '-felide-constructors'
icpc: command line warning #10006: ignoring unknown option '-felide-constructors'
In file included from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/Moose.h(12),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/MooseTypes.h(12),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/MooseUtils.h(13),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/MooseObject.h(13),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/AuxKernel.h(12),
                 from /work/10024/zhaochun/ls6/projects/farms_cdms/build/header_symlinks/FDCompVarRatev2.h(5),
                 from /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/FDCompVarRatev2.C(1):
/work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/libMeshReducedNamespace.h(16): catastrophic error: cannot open source file "libmesh/libmesh.h"
  #include "libmesh/libmesh.h"
                              ^

compilation aborted for /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/FDCompVarRatev2.C (code 4)
In file included from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/Moose.h(12),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/MooseTypes.h(12),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/MooseUtils.h(13),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/MooseObject.h(13),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/AuxKernel.h(12),
                 from /work/10024/zhaochun/ls6/projects/farms_cdms/build/header_symlinks/CompDispRateState.h(3),
                 from /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/CompDispRateState.C(1):
/work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/libMeshReducedNamespace.h(16): catastrophic error: cannot open source file "libmesh/libmesh.h"
  #include "libmesh/libmesh.h"
                              ^

compilation aborted for /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/CompDispRateState.C (code 4)
In file included from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/Moose.h(12),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/MooseTypes.h(12),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/MooseUtils.h(13),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/MooseObject.h(13),
                 from /work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/AuxKernel.h(12),
                 from /work/10024/zhaochun/ls6/projects/farms_cdms/build/header_symlinks/CompTJumpRateState.h(5),
                 from /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/CompTJumpRateState.C(1):
/work/10024/zhaochun/ls6/projects/moose/framework/build/header_symlinks/libMeshReducedNamespace.h(16): catastrophic error: cannot open source file "libmesh/libmesh.h"
  #include "libmesh/libmesh.h"
                              ^

compilation aborted for /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/CompTJumpRateState.C (code 4)
make: *** [/work/10024/zhaochun/ls6/projects/moose/framework/build.mk:159: /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/FDCompVarRatev2.x86_64-pc-linux-gnu.opt.lo] Error 1
make: *** [/work/10024/zhaochun/ls6/projects/moose/framework/build.mk:159: /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/CompDispRateState.x86_64-pc-linux-gnu.opt.lo] Error 1
make: *** [/work/10024/zhaochun/ls6/projects/moose/framework/build.mk:159: /work/10024/zhaochun/ls6/projects/farms_cdms/test/src/auxkernels/CompTJumpRateState.x86_64-pc-linux-gnu.opt.lo] Error 1

---

I do start with the similar build procedures as I did on Expanse: #26162, but it turns out to give -ludev not found error, after asking the IT to install the libdev package, it could take a lot of time for review, see below the original script and error I encountered:

Dear consultant,

I would like to install MOOSE framework on lonestar6 cluster, and encounter problems at libmesh step:

/opt/apps/gcc/11.2.0/bin/ld: cannot find -ludev
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:549: libnetcdf.la] Error 1
make[4]: Leaving directory '/scratch/10024/zhaochun/projects/moose/libmesh/build/contrib/netcdf/netcdf-c-4.6.2/liblib'
make[3]: *** [Makefile:695: all-recursive] Error 1
make[3]: Leaving directory '/scratch/10024/zhaochun/projects/moose/libmesh/build/contrib/netcdf/netcdf-c-4.6.2'
make[2]: *** [Makefile:540: all] Error 2
make[2]: Leaving directory '/scratch/10024/zhaochun/projects/moose/libmesh/build/contrib/netcdf/netcdf-c-4.6.2'
make[1]: *** [Makefile:1056: all-recursive] Error 1
make[1]: Leaving directory '/scratch/10024/zhaochun/projects/moose/libmesh/build/contrib'
make: *** [Makefile:35206: all-recursive] Error 1

I checked the discussion forum on MOOSE, these errors may imply a missing package: libudev-devel,. I wonder if there is a workaround, below is the build process on Frontera that works, here I use "mvapich2/2.3.7" instead of "mvapich2-x", and encounter the above error at the step: "bash scripts/update_and_rebuild_libmesh.sh", thanks for your support.

module swap intel gcc
module swap impi mvapich2-x
module load cuda
echo $CC $CXX $FC $F90 $F77
export CC=mpicc CXX=mpicxx FC=mpif90 F90=mpif90 F77=mpif77
Clone the Moose repository.
mkdir -p ~/projects
cd ~/projects
git clone https://github.com/idaholab/moose.git
cd moose
git checkout master

Inside the scripts directory, run:
cd ~/projects/moose/scripts
export MOOSE_JOBS=6 METHODS=opt
# Build petsc and libmesh
bash scripts/update_and_rebuild_petsc.sh --download-cmake=1
bash scripts/update_and_rebuild_libmesh.sh
# Making use of Cmake already being available via PETSc downloading it earlier:
export PATH=`pwd`/petsc/arch-moose/bin:$PATH
# Build wasp
bash scripts/update_and_rebuild_wasp.sh
# Build MOOSE test
cd test
make -j $MOOSE_JOBS
cd ../test/
make -j 6

(Optional) code in question / simulation log / errors

No response

Encountering Errors? Please include diagnostic output

No response

[GiudGiud]

hello

I think you just need to set the LIBMESH_DIR environment variable before building your app.

Guillaume

[chunhuizhao478]

Hi @GiudGiud, I tried to add export LIBMESH_DIR=/work/10024/zhaochun/ls6/projects/moose/libmesh before build, but it returns error:

/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/libtool: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/libtool: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
Compiling C++ (in opt mode) /work/10024/zhaochun/ls6/projects/moose/framework/contrib/pcre/src/pcre_stringpiece.cc...
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/libtool: No such file or directory
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/contrib/bin/libmesh-config: No such file or directory
Compiling C++ (in opt mode) /work/10024/zhaochun/ls6/projects/moose/framework/contrib/pcre/src/pcrecpp.cc...
make: *** [/work/10024/zhaochun/ls6/projects/moose/framework/build.mk:134: /work/10024/zhaochun/ls6/projects/moose/framework/contrib/pcre/src/pcre_stringpiece..opt.lo] Error 127
make: *** Waiting for unfinished jobs....
/bin/sh: /work/10024/zhaochun/ls6/projects/moose/libmesh/libtool: No such file or directory
make: *** [/work/10024/zhaochun/ls6/projects/moose/framework/build.mk:135: /work/10024/zhaochun/ls6/projects/moose/framework/contrib/pcre/src/pcrecpp..opt.lo] Error 127

I checked the libmesh folder, it only has libmesh-config.in file, or the libmesh is built somewhere else by using the bash script?

[GiudGiud]

Yes these errors are consistent with libmesh having been built somewhere else.

Your LIBMESH_DIR should point to where it actually got built. So look at the original script output to see where it got built (by update_and_rebuild_libmesh)

[chunhuizhao478]

Thanks @GiudGiud, I figured it out, I shouldn't rename moose-src to moose after build, now it is working