Performance issues running MOOSE on an HPC- how much (approximate) RAM should I need to use for a problem like this?

[richmondodufisan]

I am running a problem with a very fine mesh out of necessity, the mesh is about 260 MB large, and the nonlinear system has 1,865,010 degrees of freedom. According to the documentation, 1 CPU per 20,000 DOFs is good ballpark, but I haven't gotten my simulation to run.

Initially, I did 1 node, 20 CPUs/node, 4G of memory/CPU. The simulation progressed passed "setting up", and output the initial residual (where everything is 0.000), and then failed with an OOM error.

I increased the memory to 8G/CPU, but it still failed with an OOM error. I even changed the solve type from LU:

  #petsc_options_iname = '-pc_type'
  #petsc_options_value = 'lu'

to an iterative solver:

  petsc_options_iname = '-ksp_type -pc_type -pc_hypre_type -pc_factor_levels'
  petsc_options_value = 'gmres hypre boomeramg 2'

But still got the OOM error.

Then, following the CPU/DOF guideline, I requested 5 nodes, 20 CPUs/node, 4G Memory/CPU. This time, it also made it past "setting up" but never got to the initial residual evaluation:

Setting Up........................................................................................................................................................................................................
  Setting Up Undisplaced Mesh
    Preparing Mesh.....................
      Updating Mesh...                                                                   [^[[33m 23.89 s^[[39m] [^[[33m 1576 MB^[[39m]
    Finished Preparing Mesh                                                              [^[[33m135.73 s^[[39m] [^[[33m 1576 MB^[[39m]
  Finished Setting Up Undisplaced Mesh                                                   [^[[33m135.73 s^[[39m] [^[[33m 1576 MB^[[39m]
  Initializing
    Updating Because Mesh Changed
      Updating Mesh.....                                                                 [^[[33m 31.64 s^[[39m] [^[[33m 1766 MB^[[39m]
    Finished Updating Because Mesh Changed                                               [^[[33m 31.94 s^[[39m] [^[[33m 1778 MB^[[39m]
    Currently Initializing Equation Systems............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

After which the job was cancelled for exceeding the time limit (ran it on a short partition, so 4 hours/job). I also increased the number of nodes and kept running into the same problem.

I wanted to try increasing the time now, but was wondering if there is anything I'm missing that could help performance. Also, I ran a problem with a slightly less refined mesh (160MB, so I'm guessing it has ~half the DOFs) with the initial setup, i.e 1 node, 20 CPUs/node, 4G/CPU.

P.S. The mesh changing is a bit confusing to me since this is a thermal simulation. Yes I used a custom kernel to write the heat equation in the frequency domain but it is not a simulation where the mesh should be "displaced" like the preamble says.

[GiudGiud]

Hello

The switch to an iterative solver is a good idea, LU wont scale to 2M DOFs

It seems the initializationn is either very slow or hanging. Can you use a debugger after it has been running for a while to get the backtrace on a few of the processes? This will tell us what it is doing

[richmondodufisan]

I submitted it as a batch job where I load the modules and then do mpiexec. Will the loaded modules not be loaded on each node when submitted in a batch script?

[GiudGiud]

Will the loaded modules not be loaded on each node when submitted in a batch script?

if they are loaded by the batch script then it's fine

[richmondodufisan]

Do you have any idea what else could be causing this ORTE Daemon Failure message when I try to use more than 1 node?

As for running in debug mode, I got this error:

*** ERROR ***
/projects/p32089/moose_projects/purple/FDTR_Simulation_Fourier/Steady_State_Formulation/FDTR_input_GibbsExcess_StepFunction_BesselRing_Fourier_Steady_theta_0_freq_1e6_x0_0.i:309.3:
The following error occurred in the BoundaryCondition 'heat_source_term_real' of type RingGaussianPumpBessel.

This boundary condition is being executed outside the domain of definition of its variable, on subdomain: 1


*** ERROR ***
/projects/p32089/moose_projects/purple/FDTR_Simulation_Fourier/Steady_State_Formulation/FDTR_input_GibbsExcess_StepFunction_BesselRing_Fourier_Steady_theta_0_freq_1e6_x0_0.i:309.3:
The following error occurred in the BoundaryCondition 'heat_source_term_real' of type RingGaussianPumpBessel.

This boundary condition is being executed outside the domain of definition of its variable, on subdomain: 1

Which is strange because this same code/BC object ran with a coarser mesh (but the coarser mesh was block-separated before using in MOOSE). There are block restricted variables in this code too, could it be that the partitioning in the mesh with SubdomainBoundingBoxGenerator accidentally put an element in two blocks or something like that?

Running in debug mode like this also created an output file: tmp_jit_zMznko.cc with the contents:

#define _USE_MATH_DEFINES
#include <cmath>
extern "C" void f_93591b04b2664d70_dbg(double * ret, const double *params, const double *immed, const double eps) {
double s[5];
s[0] = immed[0];
s[1] = params[0];
s[0] *= s[1];
s[1] = immed[1];
s[2] = params[1];
s[2] *= s[2];
s[3] = params[2];
s[3] *= s[3];
s[2] += s[3];
s[1] *= s[2];
s[1] = std::exp(s[1]);
s[0] *= s[1];
*ret = s[0]; }

[richmondodufisan]

Confirmed that the errors I was seeing previously were a mesh issue... I created another mesh with even higher refinement, but structured it in a way that made it easier for SubdomainBoundingBoxGenerator and the OOM errors disappeared. The ORTE Daemon Failure errors stop showing up today (I didnt do anything differently) so I assume it must've just been a system issue. Thanks!

[GiudGiud]

*** ERROR ***
/projects/p32089/moose_projects/purple/FDTR_Simulation_Fourier/Steady_State_Formulation/FDTR_input_GibbsExcess_StepFunction_BesselRing_Fourier_Steady_theta_0_freq_1e6_x0_0.i:309.3:
The following error occurred in the BoundaryCondition 'heat_source_term_real' of type RingGaussianPumpBessel.

This boundary condition is being executed outside the domain of definition of its variable, on subdomain: 1

This is from the integrity check that checks that variables are defined near the boundaries that they hold a boundary condition onto.
You can turn this off but then you need to add skip_execution_outside_variable_domain to any integrated BC.

Much better to fix your mesh!

Guillaume