Unexpected download/extract termination when creating moose-dev on Expanse Cluster #26162
-
|
Hi all, I tried to build moose on UCSD Expanse Cluster, after installing mamba, I tried to install moose-dev as follows:
The download/extract progress terminated somehow without any error: If I try to do I appreciate any suggestions, thanks! |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 45 replies
-
|
Hello Did the download fail the first time? Guillaume |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @milljm, I think it indicates there is enough room? |
Beta Was this translation helpful? Give feedback.
-
|
hmm Indeed, that worked. You can delete that file: rm ~/zero_file.junkAs for why you can't install mamba create -n moose moose-libmesh moose-toolsThis is substantially a smaller (256MB) amount of Conda packages. Basically, we're installing everything except Peacock dependencies (additional 426MB). |
Beta Was this translation helpful? Give feedback.
-
|
What I want to do is try and determine what package in our channel that is causing this to fail. The dependency tree is as follows:
(if you install insert that package here you will get the children listed below it) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @milljm, thanks for your help, just to update, after using interactive node, all steps can be execuated successfully. Do you think if I should redo the normal build under interactivate session or is there anything we would like to check first? |
Beta Was this translation helpful? Give feedback.
-
|
I would proceed to build using an interactive |
Beta Was this translation helpful? Give feedback.
-
|
Are you attempting to do the build on a login node or a cluster node? For such a large build I would request an interactive session on a cluster node with sufficient time and memory to complete the job. Specifying enough memory is crucial. |
Beta Was this translation helpful? Give feedback.
-
|
+1 but make sure cluster nodes have access to the internet. On INL clusters they dont |
Beta Was this translation helpful? Give feedback.
-
|
I don't expect that to be a problem on Expanse. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @hzclarksm, yes I'm building on a login node, I should try interactive session |
Beta Was this translation helpful? Give feedback.
-
|
This seems to be the cause of failed test build as suggested by @milljm, after using a cluster node, it is able to build the moose inside test_conda without any error, thanks @hzclarksm for the advice |
Beta Was this translation helpful? Give feedback.
-
|
After using the interactive node, I'm able to build moose env in mamba, after downloading moose and build test, I received the following error: I'm not sure if there is conflict in the modules or something else. Please help, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
no these are MPI errors |
Beta Was this translation helpful? Give feedback.
-
|
This "Expanse" Cluster probably wants you using their MPI stack and not one provided by Conda. Also just a guess, but this cluster may be using Slurm as its job scheduler as well. Which means you'll want to submit work using that tool ( The TestHarness (the (run this command while not on an interactive node. e.g Please run this command while on the head node instead) cd /home/czhao1/projects/moose/test/tests/vectorpostprocessors/work_balance
srun --mincpus 2 /home/czhao1/projects/moose/test/moose_test-opt -i work_balance.i --error --error-override --no-gdb-backtraceI don't think any of us here are Slurm folks, so our knowledge will be limited in that regard. |
Beta Was this translation helpful? Give feedback.
-
|
Another discussion which might be related to this issues in this discussion: #26029 (reply in thread) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @milljm, the second command indeed pops up a srun error, this might lead to the timeout error: I tried your suggestion #26029, in an interactive node, but received error: Previously I tried also to build using compilers from Expanse follows https://mooseframework.inl.gov/getting_started/installation/hpc_install_moose.html, but it's not successful either. |
Beta Was this translation helpful? Give feedback.
-
|
what modules do you have available? Perhaps we can guide you there, but yeah... sorry about throwing you an invalid command. I know its going to be I honestly do not know much about Slurm. But I am betting you'll need to utilize it in order to use Expanse correctly. However, I think we can help you build MOOSE on this cluster. Can you provide a list of modules available? We need to choose a stack, and go from there. module avail...might print a very large list. Please don't hesitate to post the entire thing! Also, that bit about what you exported, you only need to supply literal 'srun'. When you encapsulate the command in ticks: `` as you did, you are actually executing the command at that moment, and whatever that command returns, is what you are assigning to export MOOSE_MPI_COMMAND=srunBut this is neither here, nor there. We first need to build MOOSE properly, not using our Conda compilers! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @milljm for your help! After logining into Expanse, I run |
Beta Was this translation helpful? Give feedback.
-
|
I am being asked to direct you to creating a help desk ticket with them... Please email: consult@sdsc.edu The person that emailed me has studied this thread, and they believe they know the issue; You are using a GPU node instead of a CPU compute node. Also, the MPI stack I asked you to load is not the preferred stack they recommend. Edit: Also what might help, when you create a ticket, you should probably reference my issue number with them: #32170 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @milljm. I just created a ticket #32171 and cc you and people from our project, let's see how they reply. |
Beta Was this translation helpful? Give feedback.
-
|
From their help desk:
|
Beta Was this translation helpful? Give feedback.
-
|
I was able to successfully build and test MOOSE on Expanse. Here was my workflow:
cd ~/
git clone https://github.com/idaholab/moose
cd moose
git checkout master
module load gcc/10.2.0 python/3.8.12 py-virtualenv/16.7.6 py-pip/21.1.2 openmpi/4.1.3
export CC=mpicc CXX=mpicxx FC=mpif90 F90=mpif90 F77=mpif77 MOOSE_JOBS=6 METHODS=opt METHOD=opt
pip install packaging pyyamlAgain, you only need to perform the above once.
If you are told you need to update libMesh for example, then you must rebuild libMesh, then MOOSE (and then your application if you are building your own application). If you are told to rebuild PETSc, then you must rebuild PETSc, libMesh, and then MOOSE (and then your application if you are building your own application). Starting from the beginning, everything is needed to be built of course: scripts/update_and_rebuild_petsc.sh --download-cmake=1
scripts/update_and_rebuild_libmesh.sh
# Making use of Cmake already being available via PETSc downloading it earlier:
export PATH=`pwd`/petsc/arch-moose/bin:$PATH
scripts/update_and_rebuild_wasp.sh
cd test
make -j $MOOSE_JOBS
diff --git a/python/TestHarness/testers/RunApp.py b/python/TestHarness/testers/RunApp.py
index 7a67222f52..0d4616ed8a 100644
--- a/python/TestHarness/testers/RunApp.py
+++ b/python/TestHarness/testers/RunApp.py
@@ -207,7 +207,7 @@ class RunApp(Tester):
elif nthreads > 1:
command = command + ' --n-threads=' + str(nthreads)
- if self.force_mpi or options.parallel or ncpus > 1:
+ if self.force_mpi or ncpus > 1:
command = self.mpi_command + ' -n ' + str(ncpus) + ' ' + command
return command(Please don't do this, or accept this as a solution. I am only posting here for informational purposes). This doesn't make parallel tests run. It skips them (but doesn't say so). Regardless, if it makes you feel better to see no errors after running With the above modification: ./run_tests -p 1 -j $MOOSE_JOBS
<trimmed>
meshgenerators/distributed_rectilinear/partition.3D_3 ........................................ [min_cpus=3] OK
meshgenerators/distributed_rectilinear/partition.3D_4 ........................................ [min_cpus=4] OK
meshgenerators/distributed_rectilinear/partition.3D_8 ................... [insufficient slots,min_cpus=8] SKIP
--------------------------------------------------------------------------------------------------------------
Ran 3935 tests in 707.4 seconds. Average test time 0.8 seconds, maximum test time 42.6 seconds.
3935 passed, 254 skipped, 0 pending, 0 failedIn order to run a large problem on this cluster, in parallel, you will instead interface with SLURM's job scheduler. Which is beyond the scope of this discussion. However, MOOSE is ready to do so if following the above instructions. But very quickly, here is a sample: Mesh refinement set to 5 to add some computational requirements ( cd tests/geomsearch/quadrature_penetration_locator
srun --partition=shared --nodes=1 --ntasks=6 --cpus-per-task 6 --account=<redacted> --mem=12G --time=00:30:00 mpirun -n 6 /home/me/moose/test/moose_test-opt -r 5 -i quadrature_penetration_locator.i
<snippet>
Framework Information:
MOOSE Version: git commit 0f3efa9423 on 2023-12-20
LibMesh Version: 7c03afd53c9c7fd7fcb6e3e20bae37cf1b124986
PETSc Version: 3.20.1
SLEPc Version: 3.20.0
Current Time: Wed Dec 20 13:23:56 2023
Executable Timestamp: Wed Dec 20 12:34:13 2023
Parallelism:
Num Processors: 6 <---- working in parallel
Num Threads: 1 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @milljm for your patience and answer! It helps us a lot! I will give it a try later my own |
Beta Was this translation helpful? Give feedback.
I was able to successfully build and test MOOSE on Expanse. Here was my workflow:
Self explanatory. But adding for completeness.
The following needs to be accomplished any time you wish to perform MOOSE related work.
module load gcc/10.2.0 python/3.8.12 py-virtualenv/16.7.6 py-pip/21.1.2 openmpi/4.1.3 export CC=mpicc CXX=mpicxx FC=mpif90 F90=mpif90 F77=mpif77 MOOSE_JOBS=6 METHODS=opt METHOD=optThe following packages represent what is required by the TestHarness when running…