GCHP Memory Leakage when Using C720 Derived Winds on AWS

[laura-hyesung-yang]

Your name

Laura Yang

Your affiliation

WashU in St. Louis

What happened? What did you expect to happen?

I'm running C180 with a stretch factor of 4 over the CONUS on AWS. I am using C720-derived winds. Memory usage increases rapidly and eventually blows up on the 3rd day of simulation. Currently testing if using C90 with a stretch factor of 2 using GEOS-FP 0.25 met fields encounters a similar issue.

I'd greatly appreciate any advice. For now, should I try to output mid-checkpoint every day and rerun when memory leaks? This might be related to #466 and #516 raised by Yanshun and @yuanjianz.

What are the steps to reproduce the bug?

Run C180 stretch factor of 4 on AWS.
Run directory setting:

• Choose simulation type: 1. Full chemistry
• Choose additional simulation option: 3. Complex SOA
• Choose complex SOA option: 1. Complex SOA
• Choose meteorology source: 2. GEOS-FP
• Choose meteorology file type: 1. 0.25x0.3125 processed files from the GEOS-Chem data archive
• Choose meteorology for advection: C720 1-hourly derived winds (Recommended for stretched grid)

Please attach any relevant configuration and log files.

• gchp.run.txt
• gchp.20211209_0000z.log
• setCommonRunSettings.sh
• ExtData.rc.txt

What GCHP version were you using?

GCHP v14.6.3 with convection scheme fix and chemistry loading fix incorporated to codes in GeosCore directory.

What environment were you running GCHP on?

AWS

What compiler and version were you using?

• gcc@=11.5.0

What MPI library and version were you using?

• mpirun (Open MPI) 4.1.7

Will you be addressing this bug yourself?

Yes

[laura-hyesung-yang]

Using the same source code, I ran C90 with a stretch factor of 2 using GEOS-FP 0.25 met fields. It runs for 25 days and crashes with segmentation fault error. gchp.log file begins with 76.4% : 59.3% Mem Comm:Used and ends with 98.1% : 81.1% Mem Comm:Used.

• allPEs.log
• gchp.20211209_0000z.log
• logfile.000000.out.txt
• slurm-1218.out.txt
• warnings_and_errors.log

[yuanjianz]

Hi @laura-hyesung-yang, thanks for opening this github issue. The memory leak is a longstanding issue in current version of MAPL. I vaguely remember it has something to do with the diagnostic you output (the less you output, the less memory leak, but I am not sure). Hopefully the MAPL v2.55 in future version of GCHP can fix that. Maybe @lizziel can provide some insights.

As for current remedy, we might need to rely on EC2 instance that features more memory:core ratio. For my current workload, 4GB:1core is enough. We can try 8GB:1core to see if it fit your needs.

[laura-hyesung-yang]

Hi @yuanjianz, thank you for your reply! I'm outputting 58 species in SpeciesConc 23 StateMet variables at daily scale HISTORY.rc.txt.

Previously when I used GCHP v14.1.1 at C90 with a stretch factor of 2, memory leakage occurred at a slower rate (though I used coarser met field (50km) at that time) so that I was able to run 40 days without crashing. I outputted at hourly scale before.

Does GCHP v14.6.3 have more memory leak than the older versions of GCHP (i.e., do they use different version of MAPL)? Or could it be that 2 fixes I incorporated (I rebased convection fix to v14.6.3 and merged load balance) lead to more memory leakage?

[laura-hyesung-yang]

Oops, I accidentally closed the issue. It should be re-opened but I don't have permission to re-open. @lizziel Would you help to reopen this?

[yuanjianz]

Does GCHP v14.6.3 have more memory leak than the older versions of GCHP?

I believe so, I think it was introduced to MAPL since a certain version and hopefully it can be fixed in MAPL v2.55.

Or could it be that 2 fixes I incorporated (I rebased convection fix to v14.6.3 and merged load balance) lead to more memory leakage?

The memory leakage should be mostly related to MAPL history component. But the load balancing definitely requires memory since it is designed to use memory to trade for speed.

[laura-hyesung-yang]

Yuanjian helped me try the AWS EC2 instance with a higher memory-to-core ratio (8GB/core; m7i-96). I previously used EC2 instance with 4GB/core (c6in-64). Now the C180 with a stretch factor of 4 over the CONUS using C720-derived winds and 25km met field can run without crashing for a month.

Yuanjian also ran GCHP v14.6.0 with two code fixes I incorporated (load balance and wet deposition) and did not encounter this severe memory leakage. He used met. field at 50km, and I'm using 25km. He also ran C180, and I'm running stretch grid. It may be that using a higher resolution met. field (I will try to use 50km met field to confirm this) and/or stretching capability of GCHP leads to more severe memory leakage.

The current recommendation is to use a higher memory-to-core ratio instance if you encounter similar memory leakage with GCHP.

[lizziel]

Thanks @laura-hyesung-yang for reporting this, and @yuanjianz for responding and helping. In my latest tests with GCHP dev/14.7.0 using MAPL 2.59 I see a memory leak in ExtData (input data handling), even more so than in diagnostics. I am working on putting a newer ExtData into GCHP (ExtData2G) and MAPL3 into GCHP (using ExtData3G and a new History component). Since both of these are in progress, and GMAO is no longer maintaining the original ExtData, we are not prioritizing fixing the memory leak in prior versions. If the memory leak persists in ExtData2G then we will get assistance from GMAO to fix it.

I will keep this issue open to guide testing for ExtData2G which will go into GCHP version 14.8.0.

[laura-hyesung-yang]

@lizziel Thank you for your explanation, Lizzie!