device memory usage, nek_userchk

[tonyzahtila]

Hi,

I do most of my on-the-fly post-processing of Nek5000 quantities in the subroutine userchk. I am a bit confused on how arrays I declare in this subroutine impact the memory budget of NekRS.

I tend to define regular grids in my pipe, and then use interp_nfld to map my solution onto to a regular grid, facilitating post-processing.

Now, my confusion is that when I add these arrays, I am able to run nrspre:

nrspre turbPipe 4

This runs successfully.

When I run my case:

nrsmpi turbPipe 4

It give me an out of memory error.

I can of course then try with more ranks,

nrsmpi turbPipe 8

but the device memory usage will be quite low, such as:

device memory usage: 1.04241 GB

Any suggestions for what is happening?

[stgeke]

This would suggest that you run out of memory on the host. Note, running nrspre just compiles nek (but doesn't load it) + kernels.

[tonyzahtila]

Thanks @stgeke , that's correct, when I request more memory on the host, my case runs.

Just a follow up question, I know that during the nrspre stage, I need to specify the number of ranks, such as,

nrspre turbPipe 8

I know that in Nek5000, in the makenek file we can include a line such as mcmodel=medium, if we want to assume that there is more or less memory available to each rank, is there a similar setting in nekRS? If we are targetting GPU cards with 12GB of memory or 16 GB memory, surely the calculation for the minimum number of ranks in nrspre changes?

Kind regards,

Tony

[stgeke]

nrspre is used to precomile

• nek5000 (used for I/O handling + legacy support)
• kernels

The number of target MPI ranks is only relevant for nek5000 because it is using a static memory design. Special compiler flags like "-mcmodel=medium" are automatically applied.