Performance comparison MOOSE vs. Abaqus: Bottleneck in Jacobian assembly

[TheMagnificentM]

Dear MOOSE community,

As I have recently started working with MOOSE, my initial goal was to set up a simple baseline calculation using a linear elastic material but including contact, to compare it with Abaqus. The setup investigates calculation times using different element types (C3D8, C3D20, C3D20R) coupled with a surface-to-surface penalty contact formulation.

During this process, I noticed a significant difference in computational time. While both solvers yield identical physical results and require a similar number of time steps and nonlinear iterations to converge, the total computational time in MOOSE is significantly higher. Based on the performance logs, the bottleneck appears to be the assembly of the Jacobian matrix. Furthermore, I have already tested the setup without Automatic Differentiation (AD), but it did not lead to a significant performance improvement.

I have set up a public repository containing the specific calculation cases used for this benchmark, including all necessary input files and relevant results. A direct evaluation and comparison of the performance metrics can be found in the short PDF presentation included in the repository.
Link to the repository: https://github.com/TheMagnificentM/moose-vs-abaqus_first-benchmark

Since I am new to the framework, are there any recommended MOOSE-specific performance optimizations, AD-flags (Automatic Differentiation), or alternative settings to accelerate the Jacobian evaluation for this specific setup?

Thank you for your time and feedback.

[lindsayad]

Hi @TheMagnificentM, thanks for taking the time to build such a clear presentation and reporting it to us! Would you be interested in providing some more atomic profiling, details given at https://mooseframework.inl.gov/application_development/profiling.html? If not, we may be able to do some investigation ourselves, but we're generally pretty busy. @loganharbour this could potentially be a performance data exploration opportunity even though this is outside of NEAMS. This kind of report is typical within NEAMS so I think we could still very defensibly register profiling this as NEAMS work

[GiudGiud]

We have had reports of slowdown in Jacobian assembly. In the current version of MOOSE, we have a patch that improves performance using Problem/hash_table_matrix_assembly=true.

This could be a way to improve performance.
The slowdown is mainly affected by the size of the sidesets used for contact. The more restricted they are to the area that is actually in contact the better the performance in both Jacobian assembly and the solve.

[TheMagnificentM]

Hi @lindsayad and @GiudGiud,

Thank you both for your replies and the suggestions!

@lindsayad: I tried to generate the atomic profiling data using gperftools and the oprof build as requested. I ran into quite a few environment errors during the setup before it finally worked, so I am not 100% sure if the profiling output is exactly what you wanted. However, I have uploaded the generated .prof files for the C3D20R penalty contact setup to my GitHub repository. You can find them in this new folder: 1_MOOSE_ElementsC3D20R_Performance.

@GiudGiud: Thank you for the hint regarding the patch! I tested it using the following block in my input file:

[Problem]
  use_hash_table_matrix_assembly = true
[]

While I noticed that it does save some time during the very first few times the Jacobian matrix is assembled, over the course of the entire calculation, it ends up taking the same amount of time. So overall, I didn't achieve a net performance improvement with it for this specific setup.

I agree with your assessment, that the contact is most likely the main reason for the performance difference. I have already restricted the sidesets in my mesh strictly to the exact surfaces that actually come into contact, so unfortunately, I cannot reduce them any further

Thanks again to both of you for looking into this!

[GiudGiud]

Thanks for investigating these options.
I think we will look into this soon for another project at INL and I'll keep you updated if we end up fixing the performance.

[lindsayad]

There are some further performance improvements in hash table matrix assembly that will appear once we've pulled https://gitlab.com/petsc/petsc/-/merge_requests/9230 into our petsc submodule. That should hopefully happen in the next couple weeks

[lindsayad]

@TheMagnificentM I won't be able to properly read your .prof files because it requires access to your binaries. Have you done, e.g., pprof profiling_data0.prof? Once in the pprof prompt, you can do something like svg > profiling_data0.svg. It would be nice to do this for at least two different mpi ranks, and then if you could share those svg files that would be very helpful