Long initialization times for GFS forecasts

[JessicaMeixner-NOAA]

In GFS, we have 3 forecasts: enkfgdas (C384mx025 S2S), gdas (C1152mx025 S2SW), gfs (C1152mx025 S2SW). For enkfgdas and gdas forecasts we can hit our targeted resource/time - although more improvements are always welcomed! However, for GFS we are still working towards reducing the runtime. In particular we have noticed that the initialization time seems long.

For GFS forecasts:

Depending on the various PET counts, for example I have had initialization times range from 325 to 556s (5.4min - 9.2min). (Note various PET counts for various components changed between these runs likely contributing to the variation).

For GDAS forecasts:

This is the same set-up as GFS forecast although the PET counts are different and the desired forecast length is different.

GDAS initialization in recent runs vary from 251 to 282s (over 4 minutes).

For enkfgdas forecasts:

Initialization times were seen from 133-235s. This has lower atm model resolution and no waves.

Data Locations

The original locations of these runs are WCOSS2 and can be found at: /lfs/h2/emc/ptmp/Jessica.Meixner/comroot

I have copied the ESMF profiles, forecast logs and a few of the configuration files to Ursa here: /scratch3/NCEPDEV/climate/Jessica.Meixner/scalingoutput_20250728

Some Notes:

• The wave initialization speed ups to read points from a file and changing where w3init is called is in the code used.
• version of g-w is: https://github.com/JessicaMeixner-NOAA/global-workflow/tree/retrotest12 (has updates on top of what's currently in develop)
• The version of the UFS is develop as of July 17 + a few additional updates to include some bug-fixes, but nothing substantial beyond what's already in develop.
• While more than 300 mediator PETs is helpful, at some point having a lot of MED Pets seems to increase some timings too (need to look more carefully, needs more systematic testing).
• This is using EMSFv8.8.0

Next steps for me:

• Be able to replicate @GeorgeVandenberghe-NOAA gfs forecast settings from the workflow
• Make runs with @DusanJovic-NOAA branch with tracing to see if perhaps there are large chunks of initialization for different components happening asynchronously.

[DeniseWorthen]

Related: Issue #2755

[JessicaMeixner-NOAA]

I was able to replicate @GeorgeVandenberghe-NOAA set-up for the gfs forecast:
/lfs/h2/emc/ptmp/Jessica.Meixner/comroot/time22/gfs.20241115/12/model/atmos/history/ESMF_Profile.summary
/lfs/h2/emc/ptmp/Jessica.Meixner/comroot/time22/logs/2024111512/gfs_fcst_seg0.log

All of my numbers of tasks match, except for the meditor pets, which I put 1200 instead of 1600 due to a mistake in copying things on my end. I don't get George's speed for the full forecast, or initialization time. George had a 425s iniit time mine was 510s. I have another run going so we can see if that's runtime variability or perhaps associated to the two different runs.

Some differences between our runs:
My run is a "continue" type run, w/IAU, George's is a init free forecast without IAU.
I'm looking into environment variables, but I suspect the runtime are not related to that.

[junwang-noaa]

@JessicaMeixner-NOAA For us to investigate the issue, would you please provide a GFS forecast run directory and model version? Thanks

[JessicaMeixner-NOAA]

The global-workflow is the best place to get run directories. Unfortunately it cannot update the UFS in its develop branch right now, so I"m working off of branches of both g-w and UFS (to include bug fixes we want in the retrotest while they make their way through the qeuue). It's my understanding that you want something from develop branches, so we'll have to wait a while for that.

If someone is okay with branches, I'm running this g-w branch: https://github.com/JessicaMeixner-NOAA/global-workflow/tree/retrotest12 with the yaml file: https://github.com/JessicaMeixner-NOAA/global-workflow/blob/retrotest12/dev/ci/cases/gfsv17/C1152mx025_S2SW.yaml if anyone wants to recreate things.

I hopefully will have some updates with Dusan's trace output here soon.

[JessicaMeixner-NOAA]

Enkf forecast directory: /lfs/h2/emc/stmp/jessica.meixner/RUNDIRS/trace03/es.2024111506/enkfgdasefcs001.2024111506/fcst.141668

GDAS forecast directory: /lfs/h2/emc/stmp/jessica.meixner/RUNDIRS/trace03/gdas.2024111506/gdasfcst.2024111506/fcst.3289122

Both have trace outputs using @DusanJovic-NOAA's branch of UFS

ENKF trace output:

GDAS trace output:

2 min of the 14 is waiting for MOM6 to initialize for enkf. Not sure if this is something that could be improved on @jiandewang @sanAkel It is also long in the GDAS forecast, but only ~35 seconds where the other components are waiting for it.

The mediator has about 30 seconds where it's intializing data at the very end. My guess is that can't be moved up in any way. There's also some time in between the data realize of the model components and the "Data Initialize" routine which I"m not sure what exactly is going on because it seems empty.

There's also lots of time at the end of the forecast waiting for the write grid component to finish.

In the middle of the runs, you see where we are waiting for restarts to be written as well for all of the components.

I will hopefully have this for the GFS forecast soon as well.

[JessicaMeixner-NOAA]

Looking back at some of my other ESMF profile summaries, for example here: /scratch3/NCEPDEV/climate/Jessica.Meixner/scalingoutput_20250728/time05/gdas.20241115/12/model/atmos/history/ESMF_Profile.summary it looks like increasing the amount of nodes for MOM6 could help decrease the initialization time. That being said, I don't think MOM6 needs for for the run time, but if it could help with init time, that would be good.

[sanAkel]

I'm afraid there's not much that can be done given the way things are set up:

• fixed PEs throughout the run, which includes initialization.
• if somehow one can different core counts at different times of the runs, then please go ahead and try that.
• single restart for the ocean instead of split up per node.

[JessicaMeixner-NOAA]

I'm seeing mixed results with the ocean initialization - @sanAkel what have you seen to reduce the ocean initialization - fewer or more PET counts?

How do we toggle the restart splitting? We can try to see if that impacts the model performance.

[JessicaMeixner-NOAA]

The reality on the restart is that the wave takes longer to write restarts, so that would only help for enkf jobs. The work it would take to improve model restart performance on the wave model is not realistic for v17 so that's not currently being pursued.

[sanAkel]

Suggest increasing cores

[JessicaMeixner-NOAA]

Any node counts in mind? Doubling from 100 ->200 had no impact on initialization time in my recent runs.

[GeorgeVandenberghe-NOAA]

The optimal number of cores of OCN is between 300 and 360 with not much runtime change between those numbers. The optimal number for ICE is about 240. The optimal number for WAVE is between 3500 and 4000 single thread ranks or 7000 double thread ranks. We have not looked at core counts beyond about 60,000 for ATM but are likely not at the pure ATM scalability limits if history file and restart time can be eliminated.

…

On Wed, Jul 30, 2025 at 4:55 PM Santha Akella ***@***.***> wrote: *sanAkel* left a comment (ufs-community/ufs-weather-model#2831) <#2831 (comment)> Suggest increasing cores — Reply to this email directly, view it on GitHub <#2831 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FRXQSYQS544PM4WMS33LEWL5AVCNFSM6AAAAACCSA7NCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMZXG44TCMRTGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[JessicaMeixner-NOAA]

The optimal number of cores of OCN is between 300 and 360 with not much
runtime change between those numbers. The optimal number for ICE is about
240. The optimal number for WAVE is between 3500 and 4000 single thread
ranks or 7000 double thread ranks. We have not looked at core counts
beyond about 60,000 for ATM but are likely not at the pure ATM scalability
limits if history file and restart time can be eliminated.

@GeorgeVandenberghe-NOAA - Those are optimal numbers for the GFS forecast, correct?

We have 3 different forecasts: gfs, gdas, and enkfgdas (which doesn't even have waves).

[GeorgeVandenberghe-NOAA]

These are optimal numbers for the C1152 GFS forecast

…

On Wed, Jul 30, 2025 at 5:02 PM Jessica Meixner ***@***.***> wrote: *JessicaMeixner-NOAA* left a comment (ufs-community/ufs-weather-model#2831) <#2831 (comment)> The optimal number of cores of OCN is between 300 and 360 with not much runtime change between those numbers. The optimal number for ICE is about 240. The optimal number for WAVE is between 3500 and 4000 single thread ranks or 7000 double thread ranks. We have not looked at core counts beyond about 60,000 for ATM but are likely not at the pure ATM scalability limits if history file and restart time can be eliminated. @GeorgeVandenberghe-NOAA <https://github.com/GeorgeVandenberghe-NOAA> - Those are optimal numbers for the GFS forecast, correct? We have 3 different forecasts: gfs, gdas, and enkfgdas (which doesn't even have waves). — Reply to this email directly, view it on GitHub <#2831 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FT7LJJ3MHNIQ4ZSJ2T3LEXEXAVCNFSM6AAAAACCSA7NCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMZXHAYDQMZUGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[JessicaMeixner-NOAA]

So I've looked into my logs a bit more and I need to do a clean test, but it might simply be the latest update in MOM6 tripling the initialization time of MOM6. If that's the case, then hopefully that's something that can be tracked down and improved. FYI: @sanAkel @jiandewang

[JessicaMeixner-NOAA]

@DusanJovic-NOAA - I was reading through #1554 and was curious if you've already tested the reproducibility and other operational requirements with these new build options? I'm assuming they should all pass, but thought I'd double check.

[JessicaMeixner-NOAA]

I guess I'm just trying to follow up on why we didn't adopt this before for all components.

[DusanJovic-NOAA]

@DusanJovic-NOAA - I was reading through #1554 and was curious if you've already tested the reproducibility and other operational requirements with these new build options? I'm assuming they should all pass, but thought I'd double check.

I checked the reproducibility while I was testing the gfs fcst configuration. Output files were identical between successive runs. I also ran a full regression test on Ursa and all test passed against newly created baselines. New baselines were needed for all Intel non-debug tests.

[SamuelTrahanNOAA]

Long ago, in the days of yore, when -DFASTER=ON was new, it was foretold to be the default compilation option. Perhaps it has proven its mettle, and time has come to update the defaults?

We could add -DSLOWER=ON for those who desire the old options.

[GeorgeVandenberghe-NOAA]

*Do I need to add -DFASTER=ON to the cmake below?* cmake .. -DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8_ugwpv1,FV3_GFS_v17_coupled_p8_ugwpv1,FV3_global_nest_v1 -DPDLIB=ON -DCMAKE_BUILD_TYPE=Release -DMPI=ON make -j 8 VERBOSE=1 This is how I build UFS git clone --recursive https://github.com/DusanJovic-NOAA/ufs-weather-model ufast cd ufast git checkout faster cd modulefiles ; module use `/bin/pwd` ; module load ufs_wcoss2.intel ; cd .. rm -rf build ; mkdir build ; cd build cmake .. -DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8_ugwpv1,FV3_GFS_v17_coupled_p8_ugwpv1,FV3_global_nest_v1 -DPDLIB=ON -DCMAKE_BUILD_TYPE=Release -DMPI=ON make -j 8 VERBOSE=1

…

On Fri, Oct 10, 2025 at 9:54 AM Dusan Jovic ***@***.***> wrote: *DusanJovic-NOAA* left a comment (ufs-community/ufs-weather-model#2831) <#2831 (comment)> @DusanJovic-NOAA <https://github.com/DusanJovic-NOAA> - I was reading through #1554 <#1554> and was curious if you've already tested the reproducibility and other operational requirements with these new build options? I'm assuming they should all pass, but thought I'd double check. I checked the reproducibility while I was testing the gfs fcst configuration. Output files were identical between successive runs. I also ran a full regression test on Ursa and all test passed against newly created baselines. New baselines were needed for all Intel non-debug tests. — Reply to this email directly, view it on GitHub <#2831 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FXLM3CRRKPYXTU3YXD3W63CNAVCNFSM6AAAAACCSA7NCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGOJQGI4DGNBWG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[dpsarmie]

Do I need to add -DFASTER=ON to the cmake below?

Yes, just add that to what you already have in the cmake command.

[GeorgeVandenberghe-NOAA]

cmake .. -DFASTER=ON -DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8_ugwpv1,FV3_GFS_v17_coupled_p8_ugwpv1,FV3_global_nest_v1 -DPDLIB=ON -DCMAKE_BUILD_TYPE=Release -DMPI=ON So I do this in red?

…

On Fri, Oct 10, 2025 at 10:42 AM Daniel Sarmiento ***@***.***> wrote: *dpsarmie* left a comment (ufs-community/ufs-weather-model#2831) <#2831 (comment)> *Do I need to add -DFASTER=ON to the cmake below?* Yes, just add that to what you already have in the cmake command. — Reply to this email directly, view it on GitHub <#2831 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FSNBWSVA7WDXXECDOD3W7AWXAVCNFSM6AAAAACCSA7NCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGOJQGU2DINBTGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[dpsarmie]

cmake .. -DFASTER=ON -DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8_ugwpv1,FV3_GFS_v17_coupled_p8_ugwpv1,FV3_global_nest_v1 -DPDLIB=ON -DCMAKE_BUILD_TYPE=Release -DMPI=ON

I don't see the red marking, but that cmake command should do it.

[gspetro-NOAA]

@JessicaMeixner-NOAA I was thinking of converting this issue to a GitHub Discussion. It seems more like a general discussion and Q&A around the long initialization times rather than an issue proposing a fix. If there is a PR planned to address the long initialization times, however, I could see keeping it as an issue. What do you think?

[JessicaMeixner-NOAA]

@dpsarmie - should we close issue or convert to discussion?

[GeorgeVandenberghe-NOAA]

Keep it as an issue until we have applied all of the fixes we intend to apply

…

On Thu, Dec 18, 2025 at 5:15 PM Jessica Meixner ***@***.***> wrote: *JessicaMeixner-NOAA* left a comment (ufs-community/ufs-weather-model#2831) <#2831 (comment)> @dpsarmie <https://github.com/dpsarmie> - should we close issue or convert to discussion? — Reply to this email directly, view it on GitHub <#2831 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FXVJ4XVB22I4SP3CY34CMROTAVCNFSM6AAAAACCSA7NCWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNZSGQ3TKNZSGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[gspetro-NOAA]

@GeorgeVandenberghe-NOAA In that case it sounds like we should be converting this to a discussion, and when developers are putting in specific fixes, they should open an issue with the specific problem/solution. It's not clear from this issue description what specific fixes are proposed. It's also not clear that they would be applied directly in the ufs-weather-model repository (components and apps are both discussed here to varying degrees). Normally a GitHub Issue body should contain some combination of description/solution/alternatives/steps for reproducing. This is a great conversation, but it's not really outlining in a clear or concise manner which specific problem(s) will be addressed (and what solutions are proposed). It's a robust discussion, but a discussion nonetheless.

[gspetro-NOAA]

@JessicaMeixner-NOAA I just noticed that moving this to a discussion closed this on your GFSv17 project board. You may have to create specific issues for the planned work and add them to your board, since I'm pretty sure Discussions can't be added to a project. If that's going to be a problem, I can convert this back to an issue, but as a general practice, I think we'd like to aim for conversations (even these more technical ones) to reside under Discussions and for Issues to be proposed code changes.

[dpsarmie]

@gspetro-NOAA I'll go ahead and link PRs/issues in this discussion related to the GFS runtime so that there's a git history.

[dpsarmie]

Here are the relevant PRs and issues that were worked on in order to resolve this runtime issue:

Adding the -O3 compiler option for all subroutines

The option for "-DFASTER" was available on some but not all subroutines.
Issue: Expand support for FASTER build option to more subcomponents
PR: Add compiler flags for cmake -DFASTER build option // Add post_itag_gcafs file #2957 // Automate Additional Fields in PR Board #2949

Allow FV3 to use saved routehandles

FV3 has the capability to generate and use saved route handles, but the required option was not available in the namelist templates. The option was added and an RT was changed to use the route handle functionality for testing purposes.
Issue: Add use_saved_routehandles as a configurable variable
PR: Update inline post for GFSv17 // Add FV3 route handle opt #2939 // Fix fpe in MOM6 #2927

Change the number of PETs in mediator

Testing has shown that we can increase the number of PETs for CMEPS to ~4800. Previously there was a hard cap of 1200 (300 per thread x 4 threads). For the GFSv17 configuration on WCOSS2, 4800 cores was found to be the optimal number. Increasing the number of PETs above 4800 created a slowdown.
Testing could be done to see if this is a configuration/machine specific limitation or a more general limitation.
Changes in GW: https://github.com/NOAA-EMC/global-workflow/pull/4138

Remove unused fields in UFSATM module_cplfields.F90

A speedup in initialization was also seen when removing unused fields in cpl/module_cplfields.F90 in UFSATM (similar to what was done in GFSv16). This is a very specific change to UFSATM that should only live on the production branch.
Production branch PR: Remove unused fields -- CHANGE FOR PROD ONLY

CMEPS Route handles

CMEPS also has the capability to generate and use saved route handles, similar to FV3. However, testing showed that there was a slowdown in overall runtime even though there was a reduction in CMEPS initialization. This is currently being worked on by folks at EMC and ESMF.
Issue: (I need to create an issue; I'll edit this once it's done)