Performance issues on HPC

[selarem]

Dear moose developpers,
I have installed moose on a National HPC. When Testing, I noticed that On my old Dell station moose run faster than on the HPC Cluster ! The cluster has the most new and performant nodes possible I could have. The engineers told me that moose was not compiled with good optimization options to profit from the cluster capabilities. Here my compilation (offline) slurm script:

`module purge
source /opt/cray/pe/cpe/23.02/restore_lmod_system_defaults.sh
module load PrgEnv-gnu
module load craype-x86-genoa
module load cray-python
module load craype-network-ofi
module load CPE-23.02-gcc-12.2.0-CPU-softs
module load cray-hdf5/1.12.2.1
module list

 make -j 40`

to run a job I use this:

`

#!/bin/bash
#SBATCH --account=cin4054
#SBATCH --job-name=<miehe>
#SBATCH --constraint=GENOA
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=20
#SBATCH --cpus-per-task=1
#SBATCH --threads-per-core=1
#SBATCH --ntasks=20
#SBATCH --time=23:59:00
module purge
source /opt/cray/pe/cpe/23.02/restore_lmod_system_defaults.sh
module load PrgEnv-gnu
module load craype-x86-genoa
module load cray-python
module load craype-network-ofi
module load CPE-23.02-gcc-12.2.0-CPU-softs
module load cray-hdf5/1.12.2.1
module list
export MOOSE_MPI_COMMAND="srun"
source ~/.bashrc                           
srun  -- combined-opt   -i CoupledTri_960.i  > CoupledTri_960

`

Thank you for any suggestion.
Regards,
Saber

[GiudGiud]

Hello

Which installation path did you follow?

Was the MPI distribution used compiled by the cluster administrators?

Guillaume

[selarem]

I have followed this :
https://mooseframework.inl.gov/moose/getting_started/installation/offline_installation.html
The cluster is 👍
https://dci.dci-gitlab.cines.fr/webextranet/user_support/index.html#connecting
and MPI are 👍

module available
---- /opt/cray/pe/lmod/modulefiles/comnet/crayclang/14.0/ofi/1.0 ----
cray-mpich/8.1.20 (L,D)    cray-mpich/8.1.21

Where:
 L:  Module is loaded
 D:  Default Module

[milljm]

ok I dont have any big ideas on this. You can pass additional flags to compile with higher optimization levels, for example -o3 (we default to o2) and -march=native if you want to try optimizing more.

The performance is so poor that it is far less that what I could obtain in an old Dell station !
Something is wrong I guess, since the cluster I want to use is the most "new" and "performant" of the country !

Were there any differences? Can you provide a profiling report (your_app-opt -t) of both (one built with the optimizations @GiudGiud mentioned, and one without)?

[selarem]

I have tried the combined module (combined-opt) on my old Dell station and on the super cluster. And found that The old station is many times faster. I have compiled with the (default) options on both. This looks very strange to me.

[selarem]

I could not find in which script I should change the optimizer options, to specify o3 or -march=native.
regards,

[milljm]

The engineers told me that moose was not compiled with good optimization options to profit from the cluster capabilities

Did the engineers tell you what these optimization switches should be? I took some time to read up about the GENCI cluster. It looks like there might be different queues for different sort of work. One for AI specialized work (Instinct Accelerators (which probably wouldn't work well with MOOSE... I think!?), and another for general work (AMD EPYC 7003 series). Just stating this, in case we happen to be using a partition of this cluster where MOOSE is not suited to run on.

Usually, the following machine optimizations suffice:

-march=native

Here is a list I found containing a myriad of machine options for all known CPU types (you don't have to worry about these, but I found it interesting): https://gcc.gnu.org/onlinedocs/gcc/Submodel-Options.html. Most likely this cluster falls in line with the x86 machine archetype. However, native might fail, if the compiler being used is older than the CPU archetype. If that is the case, you then have to choose one from the list I linked. But more importantly, the cluster engineers should have told you what switches to use.

Once you know what switches to use:

The compiler flags are gathered from environment variables (normally). So when you have all the appropriate modules loaded, you should see CXXFLAGS populated. Like on my Intel Mac workstation using the Conda compilers for example:

❯ conda activate moose
❯ echo $CXXFLAGS
-march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong \
-O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 \
-isystem /Users/jason/conda3/envs/moose/include -std=c++17

As you can see, Conda sets-march=core2 which is optimized for the very old Intel Core2 archetype :) (off topic: Conda chooses to optimize with core2, so that the binaries they make available, operate on a wide number of CPUs).

[milljm]

@selarem
Can you comment on any of the above? It might be that the cluster engineers may know what optimizations MOOSE needs on their cluster. We should ask them: -march=what?? optimizations.

[selarem]

[GiudGiud]

How many elements? Can you please paste th whole header?
If for example the default partitioning is terrible, this will lower performance

[selarem]

Example of header :

"
Framework Information:
MOOSE Version: git commit 04afdb8 on 2023-09-25
LibMesh Version: ceff84386bdc83d87e9eb98ac3a9b745711e415a
PETSc Version: 3.19.3
SLEPc Version: 3.19.1
Current Time: Wed Feb 21 10:46:40 2024
Executable Timestamp: Wed Feb 21 10:43:08 2024

Parallelism:
Num Processors: 20
Num Threads: 1

Mesh:
Parallel Type: replicated
Mesh Dimension: 2
Spatial Dimension: 2
Nodes:
Total: 231601
Local: 11746
Min/Max/Avg: 11337/11755/11580
Elems:
Total: 460800
Local: 23030
Min/Max/Avg: 22885/23257/23040
Num Subdomains: 1
Num Partitions: 20
Partitioner: metis

Nonlinear System:
Num DOFs: 694803
Num Local DOFs: 35238
Variables: { "disp_x" "disp_y" "c" }
Finite Element Types: "LAGRANGE"
Approximation Orders: "FIRST"

Auxiliary System:
Num DOFs: 5768407
Num Local DOFs: 289492
Variables: { "vel_x" "accel_x" "vel_y" "accel_y" } { "strain_yy" "stress_yy" "kinetic_energy"
} { "resid_x" "resid_y" "bounds_dummy" } { "stress_xx" "stress_xy" "Von_Mises_stress"
"total_strain_yy" "total_strain_xx" "total_strain_xy" }
Finite Element Types: "LAGRANGE" "MONOMIAL" "LAGRANGE" "MONOMIAL"
Approximation Orders: "FIRST" "CONSTANT" "FIRST" "CONSTANT"

Execution Information:
Executioner: Transient
TimeStepper: ConstantDT
TimeIntegrator: ImplicitEuler
Solver Mode: NEWTON
MOOSE Preconditioner: SMP

Currently Executing
Performing Initial Setup
Finished Performing Initial Setup [^[[33m 8.34 s^[[39m] [^[[33m 757 MB^[[39m]

Time Step 0, time = 0

Postprocessor Values:
+----------------+----------------+----------------+----------------+----------------+----------------------+
| time | resid_x | resid_y | top_x | top_y | total_kinetic_energy |
+----------------+----------------+----------------+----------------+----------------+----------------------+
| 0.000000e+00 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 | 0.000000e+00 |
+----------------+----------------+----------------+----------------+----------------+----------------------+

Time Step 1, time = 3e-08, dt = 3e-08
"

[milljm]

While you're at it, can you also print the profile? Just add a -t to your arguments.

[selarem]

Performance Graph:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|                                Section                               | Calls |   Self(s)  |   Avg(s)   |    %   | Mem(MB) |  Total(s)  |   Avg(s)   |    %   | Mem(MB) |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| CombinedTestApp (main)                                               |     1 |      0.031 |      0.031 |   0.04 |       3 |     74.307 |     74.307 | 100.00 |     771 |
|   Action::SetupMeshAction::Mesh::SetupMeshAction::act::setup_mesh    |     1 |      0.002 |      0.002 |   0.00 |       0 |      0.002 |      0.002 |   0.00 |       0 |
|   Action::SetupMeshAction::Mesh::SetupMeshAction::act::set_mesh_base |     2 |      0.094 |      0.047 |   0.13 |       0 |      0.095 |      0.047 |   0.13 |       0 |
|   MeshGeneratorSystem::createMeshGeneratorOrder                      |     1 |      0.002 |      0.002 |   0.00 |       0 |      0.002 |      0.002 |   0.00 |       0 |
|   FEProblem::computeUserObjects                                      |     2 |      0.000 |      0.000 |   0.00 |       0 |      0.000 |      0.000 |   0.00 |       0 |
|   FEProblem::computeUserObjects                                      |     3 |      0.000 |      0.000 |   0.00 |       0 |      0.000 |      0.000 |   0.00 |       0 |
|   FEProblem::outputStep                                              |     3 |      0.004 |      0.001 |   0.01 |       0 |      4.499 |      1.500 |   6.06 |      73 |
|   Transient::PicardSolve                                             |     2 |      0.031 |      0.015 |   0.04 |       0 |     50.788 |     25.394 |  68.35 |      41 |
|     FEProblem::computeUserObjects                                    |    16 |      1.155 |      0.072 |   1.55 |       0 |      1.155 |      0.072 |   1.55 |       0 |
|     FEProblem::outputStep                                            |     6 |      0.009 |      0.002 |   0.01 |       0 |      0.009 |      0.002 |   0.01 |       0 |
|     FEProblem::solve                                                 |     2 |     21.598 |     10.799 |  29.07 |      37 |     48.223 |     24.111 |  64.90 |      41 |
|       FEProblem::computeResidualInternal                             |     2 |      0.000 |      0.000 |   0.00 |       0 |      2.684 |      1.342 |   3.61 |       0 |
|         FEProblem::computeUserObjects                                |     4 |      0.000 |      0.000 |   0.00 |       0 |      0.000 |      0.000 |   0.00 |       0 |
|       FEProblem::computeResidualInternal                             |     6 |      0.002 |      0.000 |   0.00 |       0 |     17.688 |      2.948 |  23.80 |       1 |
|         FEProblem::computeUserObjects                                |    12 |      0.008 |      0.001 |   0.01 |       0 |      0.008 |      0.001 |   0.01 |       0 |
|       FEProblem::computeJacobianInternal                             |     1 |      0.000 |      0.000 |   0.00 |       0 |      0.757 |      0.757 |   1.02 |       1 |
|         FEProblem::computeUserObjects                                |     2 |      0.000 |      0.000 |   0.00 |       0 |      0.000 |      0.000 |   0.00 |       0 |
|       FEProblem::computeJacobianInternal                             |     4 |      0.000 |      0.000 |   0.00 |       0 |      4.220 |      1.055 |   5.68 |       1 |
|         FEProblem::computeUserObjects                                |     8 |      0.000 |      0.000 |   0.00 |       0 |      0.000 |      0.000 |   0.00 |       0 |
|       FEProblem::computeBounds                                       |     2 |      0.005 |      0.002 |   0.01 |       0 |      1.245 |      0.622 |   1.68 |       0 |
|   Transient::final                                                   |     1 |      0.011 |      0.011 |   0.02 |       0 |      0.021 |      0.021 |   0.03 |       0 |
|     FEProblem::computeUserObjects                                    |     2 |      0.000 |      0.000 |   0.00 |       0 |      0.000 |      0.000 |   0.00 |       0 |
|     FEProblem::outputStep                                            |     1 |      0.002 |      0.002 |   0.00 |       0 |      0.009 |      0.009 |   0.01 |       0 |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Finished Executing

[GiudGiud]

this all seems reasonable. 35k dofs per core is fine
what preconditioning are you using? If it s one that does not scale I could see the performance loss there

[selarem]

ok I dont have any big ideas on this. You can pass additional flags to compile with higher optimization levels, for example -o3 (we default to o2) and -march=native if you want to try optimizing more.

@friedmud or @permcody may want to pitch in

I could not find the script where to specify the optimizer options.

[selarem]

SMP REGARDS Le jeu. 22 févr. 2024, 15:39, Guillaume Giudicelli ***@***.***> a écrit :

…

this all seems reasonable. 10k dofs per core is a little under what we prescribe but it s not a hard rule. what preconditioning are you using? If it s one that does not scale I could see the performance loss there — Reply to this email directly, view it on GitHub <#26585 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBUGDEMHIZGJOOCL3JJMRLYU5KB7AVCNFSM6AAAAABCAB6JKOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DKNJXG4ZTA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[GiudGiud]

what petsc options though?

[selarem]

petsc_options_iname = '-pc_type -pc_factor_mat_solver_package -snes_type'
petsc_options_value = 'lu superlu_dist vinewtonrsls'
solve_type = NEWTON # PJFNK

I do not think this will change a lot the speed, I have been using all of the possibilities as per petsc documentation. For nonlinear stiff problems, we need to be patient.

[GiudGiud]

so you have constraints on the domain? I see the vinewtonrsls. Are they from Bounds objects?
LU is not a preconditioning that scales well with larger problems. If you plot the cost of the solve at various number of dofs, you should see a nonlinear increase in the cost. If you are already high up in that nonlinear increase with 700k dofs, that can be the problem

[selarem]

Yes I am using Bounds.
Is there any preconditioner that scales better?

[GiudGiud]

The preconditioning is problem dependent. A lot of the preconditioners scale well but they wont work for your problem.
What did you try before resorting to using a direct solve (LU)? Do you know what the other codes are using / what is the state of the art for your physics?