Memory leak issue running GCHP14.3.1 on AWS

[YanshunLi-washu]

Your name

Yanshun Li

Your affiliation

WashU

Please provide a clear and concise description of your question or discussion topic.

Hi Team,

I'm running GCHP 14.3.1 C180 simulation on AWS and got a memory leak problem. The memory usage fill up to abortion within 24 hours' model run:

AGCM Date: 2020/12/31 Time: 23:10:00 Throughput(days/day)[Avg Tot Run]: 0.7 0.7 23.7 TimeRemaining(Est) 135:06:43 73.9% : 64.3% Mem Comm:Used
Mem/Swap Used (MB) at MAPL_Cap:TimeLoop= 1.248E+05 0.000E+00
...
AGCM Date: 2021/01/01 Time: 23:00:00 Throughput(days/day)[Avg Tot Run]: 40.6 83.2 137.2 TimeRemaining(Est) 001:46:22 108.3% : 98.1% Mem Comm:Used
Mem/Swap Used (MB) at MAPL_Cap:TimeLoop= 1.848E+05 0.000E+00

I was using hourly anthropogenic emission inventories and take the option to output check points. When using only monthly emission inventories and opt out for writing check points, the simulation can last for one month.

My intuition is that the memory leak is related to netcdf reading and writing, but I got no such issue when running on NASA pleiades or WashU compute1. I attached the log of building gchp executable so that more info about the linux environment can be provided.

Appreciate if there could be any suggestions and solutions.
ecbuild.log

Yanshun

[lizziel]

Hi @YanshunLi-washu, have you checked in with other GCHP users at WashU who use AWS? I think @BettyCroft has been doing runs with at least that version. If she has not seen this memory leak then I wonder if it has something to do with your settings. You can try to narrow down the issue by systematically reverting your changes back to default, such as turning off mid-run checkpoints, and then turning off hourly emissions.

[YanshunLi-washu]

Hi @lizziel I've identified that using hourly emission and write mid-run checkpoints are both factors. Optioning out for these two can make the C180 simulation last for one month.

[lizziel]

Have you been able to verify that the memory leak is only on AWS for both of these sources of memory leak?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the issue from closing this issue.

[github-actions]

Closing due to inactivity

[lizziel]

I am running into a small memory leak in GCHP when running on the NASA discover cluster at c180. There is a much larger memory leak when running on GEOSgcm. I am currently investigating. This used GEOS-Chem 14.5.1.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the issue from closing this issue.