Slow on HPC

[itgreenquist]

Hi everyone,

I'm currently trying to run several hundred relatively small simulations on INL's Sawtooth cluster, but I'm having an issue. The study I'm doing required me to modify some source code, so I did a custom install following the instructions on mooseframework.inl.gov. But the simulations are running incredibly slowly. A simulation that takes ~4 hours on my laptop had not finished after 7 days on the cluster (1 CPU in each case).

I wanted to test whether it was my installation or my PBS job script, so I timed running the tests in moose/test/ on my laptop (1 CPU) and Sawtooth (8 CPUs). My laptop took 16 minutes and Sawtooth took 40 minutes.

Does anyone have any advice/tips to get MOOSE to run faster on Sawtooth?

[GiudGiud]

Hello

I would turn on the perf_graph to see if anything obvious is being slow
Outputs/perf_graph = true.
The output of the perf_graph will be shown at the end of the simulation, so I would recommend doing a short simulation rather than the 7 day one to get some insights.

Then if there is a lot of file IO involved, it s worth running all the simulation on /scratch instead of the regular home.
Every user has a large amount of storage on scratch. The only caveat is that it s not backed up so please make sure to save your results to home every now and then.

Guillaume

[GiudGiud]

It does look like it's 3x slower on Sawtooth. Mostly driven by user objects. Are these bison-specific user objects?
@gambka for advice on Sawtooth performance

[itgreenquist]

I just picked one at random that had perf_graph already activated:
bison/examples/2D-RZ_rodlet_10pellets/2D_discrete_finiteStrain/2D_discrete_finiteStrain.i
It doesn't have a UserObject block. They must be created by some of the actions. So there probably are some BISON-specific ones.

[itgreenquist]

However, I don't think it's a problem with my BISON install specifically, because a few months ago I noticed that a MARMOT install I did was much slower than the MARMOT module.

[vanwdani]

I just picked one at random that had perf_graph already activated: bison/examples/2D-RZ_rodlet_10pellets/2D_discrete_finiteStrain/2D_discrete_finiteStrain.i It doesn't have a UserObject block. They must be created by some of the actions. So there probably are some BISON-specific ones.

It's kind of misleading since Postprocessors and VectorPostprocessors inherit from UserObjects, but there are a few true "UserObjects" that are created in that input file by the actions.

[gambka]

As @vanwdani has said we've seen significant slow down on sawtooth. I don't expect 7 days compared to 4 hours but we have seen slowdowns on the order of 3x on some cases with the same number of processors. For your situation where you are running multiple samples, is each sample running slowly or a certain one out of the few hundred? How are you spawning your jobs are they serially solved or are the several hundred submitted at once? Is your custom code also present in the example you ran?

[vanwdani]

It could just be the difference between processors used. We have long noted the difference in times when running locally on Mac, which are ARM based, versus Sawtooth, which is an Intel Xeon.

…

On Tue, May 30, 2023 at 1:15 PM Guillaume Giudicelli < ***@***.***> wrote: It does look like it's 3x slower on Sawtooth. Mostly driven by user objects. Are these bison-specific user objects? @gambka <https://github.com/gambka> for advice on Sawtooth performance — Reply to this email directly, view it on GitHub <#24534 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIAGQHG6XKAGVS2RIUNBJN3XIZBNNANCNFSM6AAAAAAYUGHRBI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[vanwdani]

@loganharbour isn't possible for @itgreenquist to run this on an ARM cluster?

[itgreenquist]

That's interesting, but I still suspect there's an issue with my installation. Is there a command to run tests or an example file using the prebuilt modules so I can compare my installation to those instead of my laptop?

[GiudGiud]

module load use.moose bison if you have access to the binaries

[vanwdani]

Sure, you can do something like ./run_tests -i examples --re 2Dtm_discrete_finiteStrain to run the aforementioned example. This will run it in parallel and test it against a common standard. You should be able to load Bison and Moose modules with the command module load use.moose moose-apps bison. As to your installation, are you building Petsc and Libmesh manually or are you using a package manager like Mamba?

[vanwdani]

I would still expect Sawtooth to take longer than your local machine. As Guillaume stated, the file IO for the HPC system is a real bottleneck for speed. As an example, you could just note the speed difference in doing a command like git status locally and remotely. HPC is aware and always trying to fix it.

Nota bene; adding more cpus to a job can make it run slower. There are tons of discussions here and on Slack for suggestions to match the job to cpus.

[itgreenquist]

Thanks for your help @GiudGiud and @vanwdani. If it's normal for Sawtooth to run 3x slower, then the tests and examples are all working perfectly well. That means the problem is with my job script. I think it might be a memory issue. I'm going to play with the script and see if I can get these to run faster.