Memory issues with GrandPotentialPlanar example

[nwinner]

Hello all, I'm new to the MOOSE framework and I'm experimenting with examples for the phase field module to get a feel for things.

I took the GrandPotentialPlanar.i example, which runs fine, and made some modifications to try and match the parameters provided by in this paper. by Biswas et al.

I modify the input file to start to work towards the system provided in the paper. (I have questions about that too but I'll save that for another thread unless it comes up in the discussion.). When I modify the initial conditions and the material constants, I am able to get some cases where I get EXIT CODE 9, signalling out-of-memory. This is very strange to me (1) that it is dependent on the input parameters I use, when the mesh doesn't change, and also (2) because the VectorMemoryUsage doesn't show very high memory usage:

hardware_id,node_utilization,page_faults,physical_mem,total_ram,virtual_mem
0,0.004812122606905,700,324,2163051,1088

I am running this through docker moose:latest on a compute node with 720GB of RAM and 2TB of disk space. So there shouldn't be any issue unless I'm misundestanding something.

Input file:
GrandPotentialPlanar.txt

Output log:
output.log

Could I get some assistsance with understanding the problem?

[GiudGiud]

Hello

This runs fine on my machine.
SIG9 just means SIGKILL. It's not necessarily OOM though that is by far the most likely.

I dont run out of memory on a much smaller machine. It seems you use 32 processes. Can you try less? Maybe 8 like I did

Guillaume

[nwinner]

When I run the input file using 8 processes, it works. Doesn't this point to some sort of OOM error since we are reducing the memory required for each rank, or does SIG9 have some other meaning related to "over parallelizing"?

[GiudGiud]

seems fine to me.
if you run with htop, do you see the memory spike with more cores?

[nwinner]

I don't see any obvious spikes in memory with htop. I can dump the memory per core using VectorMemoryUsage and it looks like ths snippet in my original post.

I can attach a monitor with psrecord to the first mpi rank using psrecord $(pgrep moose-opt | tail -1) --interval 0.001 --log usage_log.txt and the top of the log looks like this:

# Elapsed time   CPU (%)     Real (MB)   Virtual (MB)
       0.000        0.000      304.234     1025.750
       0.001        0.000      304.234     1025.750
       0.003        0.000      304.234     1025.750
       0.004        0.000      304.418     1026.004
       0.005        0.000      304.418     1026.258
       0.006        0.000      304.418     1026.258
       0.007        0.000      304.418     1026.258
       0.009        0.000      304.418     1026.258
       0.010      807.300      304.418     1026.258

[GiudGiud]

thanks that definitely rules out memory.

At what number of MPI processes do you get the error first? Is 8 the largest you can use?

[nwinner]

Increasing the number of processes and keepting n-threads=1, I can run 24 processes. It seems to fail at similar step each time for a given number of processes:

Processes, terminates on
25,68
26,58
28,21
32,14

The time step it terminates on is not 100% consistent, but it does seem to get further as number of processes decreases. Whether it matters I don't know, but for n=26, I would sometimes get exit code 135 instead of 9:

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 32469 RUNNING AT daa54fcfb23a
=   EXIT CODE: 135
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

[GiudGiud]

hi @nwinner

I cannot reproduce this issue locally up to >100 cores.
If you want to investigate this further, I will need you to debug this in parallel using these instructions
https://mooseframework.inl.gov/application_development/debugging.html