Long runtimes and error messages in some gfs_gempak_f jobs on WCOSS2

[KateFriedman-NOAA]

What is wrong?

Several gfs_gempak_f* jobs in the extended CI test case on WCOSS2 are consistently hitting their 30min walltimes. The automated 2nd attempt will also hit the walltime. A third attempt hours later always completes successfully. First thought was that the issue was a machine issue but other CI testing since the first instance are continuing to have the same issues in the same jobs.

Viewing the logs shows the following message repeated many times for those jobs: Error in message send = 22. The resulting job logs are about 15-20GB in size (!) with those error messages printed, whereas the gempak logs are usually MBs and not GBs.

The affected jobs are gfs_gempak_f123-f144 and gfs_gempak_f147-f168. Do not see the error messages in any of the other gfs_gempak_f* jobs in the same CI tests.

Snippet from log where the error first appears:

+ cpfs[13]cpdstfile=/lfs/h2/emc/ptmp/emc.global/PR/PR_3626/RUNTESTS/COMROOT/C96_atm3DVar_extended_3626/gfs.20211221/06//products/atmos/gempak/35km_pac/gfs_35km_pac_2021122
106f168
Error in message send = 22
itype, ichan, nwords,2,22216705,2
Error in message send = 22
itype, ichan, nwords,2,22216705,2
Error in message send = 22
itype, ichan, nwords,2,22216705,2
...

See saved logs on Cactus: /lfs/h2/emc/global/noscrub/emc.global/ci/SAVE_LOGS_3626

What should have happened?

No error message and job completes on time.

What machines are impacted?

WCOSS2

What global-workflow hash are you using?

develop and recent PR hashes

Steps to reproduce

Run develop extended CI test case on WCOSS2 with DO_GEMPAK=YES.

Additional information

No response

Do you have a proposed solution?

No response

[KateFriedman-NOAA]

@ChristopherHill-NOAA when you have time could you look at this issue?

[ChristopherHill-NOAA]

@KateFriedman-NOAA Yes - I can investigate this issue. At an initial glance this issue seems very similar to #3300, which had been addressed with an updated GEMPAK module reference in the WCOSS environment file through #3415, but this module update may not have been a sufficient solution. Other potential solutions were listed in #3300, and will be among those to be considered here.

[KateFriedman-NOAA]

Great, thanks @ChristopherHill-NOAA !

[JessicaMeixner-NOAA]

@ChristopherHill-NOAA - I was wondering if you have had found anything yet for this issue.

[JessicaMeixner-NOAA]

@ChristopherHill-NOAA - FYI I am asking because this has been an issue on several of the last PRs when running CI tests on WCOSS2.

@KateFriedman-NOAA @aerorahul - If a fix is not coming quickly, should we temporarily increase the time on these jobs so that we do not see these time-outs? We have this issue to track and fix the issue.

[KateFriedman-NOAA]

If a fix is not coming quickly, should we temporarily increase the time on these jobs so that we do not see these time-outs? We have this issue to track and fix the issue.

Unfortunately it's not a walltime issue. The jobs would hit the new walltime due to the nature of the problem. Often the jobs run fine with a rerun hours later but that's more manual intervention. Temporarily disabling the gempak jobs in the C96_atm3DVar_extended CI test case would help for now. Thoughts @aerorahul?

[aerorahul]

I don't think we want to disable the test. How about reducing the number of forecast hours to process in that gempak job?
Presumably processing hour 6 and 60 and beyond gives less and less added value to the test, unless there is something special happening at or after the later hours.

[ChristopherHill-NOAA]

The non-fatal GEMPAK error seems to occur on some recent test runs when processing output within the forecast hour range of F123 to F168. From #3300, the error occurred within the range of F207 to F219.

I hypothesize that the NAGRIB (GEMPAK) process executed by exgfs_atmos_nawips.sh for each forecast hour is not always closed by the corresponding "gpend" command before the next NAGRIB process is initiated, resulting in an accumulation of processes that is alerted through the voluminous error messaging.

I noticed that the number of forecast hours being grouped within each gfs_gempak_fFFF-fFFF ecFlow job was recently increased from 5 (per results referenced in #3300) to 8, which is likely causing the error to affect processing for an earlier forecast time range and may be causing the error to occur (more frequently) across multiple test runs.

Preliminarily, I suggest reverting the gfs_gempak_fFFF-fFFF job grouping to 5 forecast hours. Otherwise - or additionally - there may be a need to add an error check statement that ensures "gpend" has successfully closed a NAGRIB process before initiating the next process.

@JessicaMeixner-NOAA If possible, please direct me to error logs from your test runs featuring the dead gfs_gempak_fFFF-fFFF jobs (per #3670).