Failed Installation on HPC Cluster with VTK

[DamynChipman]

Bug Description

I am attempting to install MOOSE on my university's cluster for local usage (i.e., not cluster wide). I am following the instructions listed at HPC Cluster.

Steps to Reproduce

The issue seems to appear after running ./scripts/update_and_rebuild_petsc.sh --download-vtk. After appearing to successfully run the build script for PETSc, I run make PETSC_DIR=/bsuhome/dchipman/packages/moose/moose/scripts/../petsc PETSC_ARCH=arch-moose check and get the following output:

(base) (11:59:21) [dchipman@borah-login petsc] >>> make PETSC_DIR=/bsuhome/dchipman/packages/moose/moose/scripts/../petsc PETSC_ARCH=arch-moose check
Running check examples to verify correct installation
Using PETSC_DIR=/bsuhome/dchipman/packages/moose/moose/scripts/../petsc and PETSC_ARCH=arch-moose
*******************Error detected during compile or link!*******************
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/bsuhome/dchipman/packages/moose/moose/petsc/src/snes/tutorials ex19
*********************************************************************************
mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -O  -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -O    -I/bsuhome/dchipman/packages/moose/moose/petsc/include -I/bsuhome/dchipman/packages/moose/moose/petsc/arch-moose/include     ex19.c  -Wl,-rpath,/bsuhome/dchipman/packages/moose/moose/scripts/../petsc/arch-moose/lib -L/bsuhome/dchipman/packages/moose/moose/scripts/../petsc/arch-moose/lib -Wl,-rpath,/bsuhome/dchipman/packages/moose/moose/petsc/arch-moose/lib -L/bsuhome/dchipman/packages/moose/moose/petsc/arch-moose/lib -Wl,-rpath,/cm/shared/apps/mpich/ge/gcc/64/3.3.2/lib -L/cm/shared/apps/mpich/ge/gcc/64/3.3.2/lib -Wl,-rpath,/cm/shared/apps/blas/gcc/3.8.0/lib64 -L/cm/shared/apps/blas/gcc/3.8.0/lib64 -Wl,-rpath,/cm/shared/apps/slurm/current/lib64 -L/cm/shared/apps/slurm/current/lib64 -Wl,-rpath,/cm/shared/apps/gcc/8.2.0/lib/gcc/x86_64-pc-linux-gnu/8.2.0 -L/cm/shared/apps/gcc/8.2.0/lib/gcc/x86_64-pc-linux-gnu/8.2.0 -Wl,-rpath,/cm/shared/apps/gcc/8.2.0/lib64 -L/cm/shared/apps/gcc/8.2.0/lib64 -Wl,-rpath,/cm/shared/apps/lapack/gcc/64/3.8.0 -L/cm/shared/apps/lapack/gcc/64/3.8.0 -Wl,-rpath,/cm/shared/apps/slurm/current/lib64/slurm -L/cm/shared/apps/slurm/current/lib64/slurm -Wl,-rpath,/cm/shared/apps/gcc/8.2.0/lib -L/cm/shared/apps/gcc/8.2.0/lib -lpetsc -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lstrumpack -lscalapack -lsuperlu_dist -lflapack -lfblas -lptesmumps -lptscotchparmetis -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lpthread -lparmetis -lmetis -lm -lstdc++ -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lrt -lquadmath -lstdc++ -ldl -o ex19
/usr/bin/ld: warning: libgfortran.so.3, needed by /cm/shared/apps/mpich/ge/gcc/64/3.3.2/lib/libmpifort.so, may conflict with libgfortran.so.5
/usr/bin/ld: warning: libgfortran.so.3, needed by /cm/shared/apps/mpich/ge/gcc/64/3.3.2/lib/libmpifort.so, may conflict with libgfortran.so.5
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with hypre
C/C++ example src/snes/tutorials/ex19 run successfully with mumps
C/C++ example src/snes/tutorials/ex19 run successfully with superlu_dist
Completed test examples

Continuing with the installation to install libmesh also fails with the following output:

(base) (11:59:39) [dchipman@borah-login moose] >>> ./scripts/update_and_rebuild_libmesh.sh 
Submodule path 'libmesh': checked out '34519df7787ee1d80fe2d7c9e53a1048895a730a'
fatal: reference is not a tree: 9a40214a48657d152ba6f1de2b6296b8553890a3
Submodule path 'libmesh/contrib/timpi': checked out '2397962a0b2c4d19c53b1a45f0304eaea999e024'
Submodule path 'libmesh/contrib/timpi/m4/autoconf-submodule': checked out 'a993d797c7c0dbf9728f39fd5fbcbac758b151fa'
Submodule path 'libmesh/m4/autoconf-submodule': checked out 'a993d797c7c0dbf9728f39fd5fbcbac758b151fa'
Unable to checkout '9a40214a48657d152ba6f1de2b6296b8553890a3' in submodule path 'libmesh/contrib/metaphysicl'
Failed to recurse into submodule path 'libmesh'
git submodule command failed, are your proxy settings correct?
mkdir: cannot create directory ‘libmesh’: File exists

Impact

As I cannot install, I cannot use MOOSE 😢.

[loganharbour]

Transferred to a discussion.

Ping @roystgnr - how are we missing a commit for metaphysicl?

[loganharbour]

Yeah. I'm not sure what's going on.

@DamynChipman my easiest suggestion: nuke /bsuhome/dchipman/packages/moose/moose and re-clone. With a fresh clone on that commit, I have no issues locally parsing that failed commit in libmesh/m4/autoconf-submodule.

[roystgnr]

This is weird. When I check out Moose bc61efd (to get something with the same libMesh hash as reported above), doing git submodule update --init --recursive libmesh/ afterward succeeds ... but, even though libmesh/.gitmodules has the correct url for MetaPhysicL, when I actually go to contrib/metaphysicl/ and do a git remote -v it seems to be pointing to the old location.

Ugh, even if I go to a fully updated Moose origin/master tracking branch I get the same inconsistency. I'm not sure what's going on here.

[roystgnr]

That might have been something peculiar to my pre-existing Moose checkout? If I do a fresh clone and a submodule update from there then everything looks fine. Looking around it seems that git submodule sync is the way to properly update an existing clone, but at this point I second the "Just nuke it from orbit. It's the only way to be sure." advice.

[DamynChipman]

Thanks for the assistance. I got rid of my clone of moose and recloned. With a fresh clone, I reran all of the update_and_rebuild_* scripts for petsc and libmesh. It got much further along, but ultimately gave an error when running the libmesh script. It's a linking error: /usr/bin/ld: cannot find -ludev. I've had this issue before installing a different package on this cluster, so I'll take it up with the system admins (unless you guys know something off the top of your head).

Feel free to close this discussion, I'll open another issue if I run into more problems after I figure out the ludev issue.

[roystgnr]

I'm kind of new to Github discussions; I've copied the answer somewhere more visible and marked it as such but I'm not sure how to close this thread like you would an issue.

cannot find -ludev means that something in your build stack (probably your MPI implementation or PETSc install?) thinks your system needs you to link to a libudev.foo (.so on Linux, .dylib I think on OSX) shared library, but then the linker (in the middle of the configure script, I guess/hope?) can't find one. Solution might be as simple as your sysadmins running apt install libudev-dev, or might be much more complicated depending on how they have things set up.

[roystgnr]

For posterity's sake: it looks like the workaround answer was "Nuke and restart from a Moose clone that postdates the submodule move from roystgnr/MetaPhysicL to libMesh/MetaPhysicL on Github. People more git-submodule-savvy than I could probably come up with a much better answer; possibly git submodule sync or something?

[gambka]

I just ran into this issue on the INL HPC clusters and

cd libmesh
git submodule sync
cd ../
./scripts/update_and_rebuild_libmesh.sh

fixed the issue for me.