Skip to content

Update scripts to use the GW-RT global namelist#3017

Merged
DavidHuber-NOAA merged 53 commits intoNOAA-EMC:developfrom
dpsarmie:feature/GW-RT_GlobalNML
May 5, 2025
Merged

Update scripts to use the GW-RT global namelist#3017
DavidHuber-NOAA merged 53 commits intoNOAA-EMC:developfrom
dpsarmie:feature/GW-RT_GlobalNML

Conversation

@dpsarmie
Copy link
Copy Markdown
Contributor

@dpsarmie dpsarmie commented Oct 18, 2024

Description


The following is a Google Doc with new variables that were added to the nml template with this PR: Google Sheets link


This PR changes the ways the input.nml file is created. Instead of creating a namelist, the variables that are set locally will now use the UFS global_control.nml.IN template and the atparse function to generate the namelist.

This should allow the GW branch to more easily test changes in the UFS regression test framework. It will also allow GW to more easily introduce UFS changes into GW.

The functionality in parsing_namelists_FV3.sh is mostly unchanged. Since variables are still saved as local, the parsing of the template will occur in this script. Sections in the namelist that were previously not generated (stoch physics options for example) will be set and parsed since the regression test template has these placeholders active. As options are turned off and on, ! will be used in the namelist to comment out sections that are not used.

The MOM6 variables have been moved here since the template place holders are active in global_control.nml.IN.

Since the stoch physics options are active, the SKEB definitions were also modified in the config.efcs files. If the first value in SKEB is not -999., then the stoch physics subroutine will turn SKEB on (do_skeb set to 1).

The FHZERO variable was also renamed to reflect what was in the namelist template.

Previous PR #2733 was closed since my old branch was stale and a new branch was created.

Unresolved issues and input needed from GW group before final code review

There are some variables that were not set in the original scripts that have been added. These were set to the UFS defaults.
The tests I've conducted were to generate the namelists but more tests should be done to make sure that outputs have not changed.

Resolves #2731
Resolves ufs-community/ufs-weather-model#1664

Outstanding dependencies:
ufs-community/ufs-weather-model#2425

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

How has this been tested?

Tested namelist generation with C48ATM and C48S2S configurations. Further testing will need to be done to ensure intended funcitonality.

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • I have made corresponding changes to the system documentation if necessary

@WalterKolczynski-NOAA
Copy link
Copy Markdown
Contributor

Not a fan of the hiding, at least not how it is implemented here. All the variables are still set by the parsing script, it makes the template more difficult to read, and doesn't even remove it from the rendered namelist, just comments the lines out. Given that, I'd rather just leave all the settings in the namelist with the values that turn them off.

@dpsarmie
Copy link
Copy Markdown
Contributor Author

Hey Walter,
Agreed, this wasn't my first choice but I didn't want to completely overhaul the scripts on the first go.
We could think about also integrating the default_vars.sh script into this workflow, but I didn't know how you all would feel about saving the environmental variables globally vs locally. If it is allowed, then you could in theory run the default_vars.sh script to set any variables that aren't defined by the user's previous scripts before populating the namelist. That should cut down on the amount of bulk in the parsing_fv3 script.

Splitting up the sections into different scripts could also work, but you run into the issue of not being able to easily pass variables between scripts if they are saved as locals (correct me if I'm wrong here).

Also open to your suggestions on how to best make this fit within the GW system.

@dpsarmie
Copy link
Copy Markdown
Contributor Author

Here is a doc with the differences between variable values for GW and RT for a C48 atm only case: GW-RT differences

We're leaning towards just changing the RT baselines to match what GW has, but we can discuss this if needed.

@dpsarmie dpsarmie reopened this Dec 20, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA marked this pull request as ready for review December 20, 2024 19:54
Copy link
Copy Markdown
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Structurally it looks fine. Didn't verify that the default for every individual setting is unchanged. Just take care of the FHZER thing and I'm happy with this.

@WalterKolczynski-NOAA
Copy link
Copy Markdown
Contributor

@dpsarmie Merge in develop again and I think this is ready for CI

@dpsarmie
Copy link
Copy Markdown
Contributor Author

I was still running tests on my end as a final check but should be done before lunch. We should still talk about the final pathing to the template (just added a review for visibility).

Added a missing period
@DavidHuber-NOAA
Copy link
Copy Markdown
Contributor

Offline tests of C96C48_hybatmaerosnowDA were successful on WCOSS2. Marking ready for CI testing on Hercules.

@DavidHuber-NOAA DavidHuber-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Wcoss2-Failed CI testing on WCOSS for this PR has failed CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed labels May 1, 2025
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels May 1, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented May 1, 2025

Experiment C96mx100_S2S FAILED on Hercules in Build# 2 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3017/RUNTESTS/COMROOT/C96mx100_S2S_aae694dc/logs/1994050100/sfs_ice_prod_mem000_f024.log

Follow link here to view the contents of the above file(s): (sfs_ice_prod_mem000_f024.log)

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels May 1, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented May 1, 2025

Experiment C96mx100_S2S FAILED on Hercules in Build# 2 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3017/RUNTESTS/EXPDIR/C96mx100_S2S_aae694dc

@emcbot emcbot removed the CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed label May 2, 2025
@emcbot
Copy link
Copy Markdown

emcbot commented May 2, 2025

CI Tests set up to run in /lfs/h2/emc/ptmp/emc.global/PR/PR_3017/RUNTESTS on WCOSS

Copy link
Copy Markdown
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good. Just noting that I own the changes to parm/config/*/config.ufs.

@emcbot
Copy link
Copy Markdown

emcbot commented May 2, 2025

Experiment C96C48mx500_S2SW_cyc_gfs FAILED on Hera in Build# 4 with error logs:

/scratch1/NCEPDEV/global/glopara/CI/3017/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_1881a891/logs/2021122018/enkfgfs_earc_tars_01.log

Follow link here to view the contents of the above file(s): (enkfgfs_earc_tars_01.log)

@emcbot
Copy link
Copy Markdown

emcbot commented May 2, 2025

Experiment C96C48mx500_S2SW_cyc_gfs FAILED on Hera in Build# 4 in
/scratch1/NCEPDEV/global/glopara/CI/3017/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_1881a891

@emcbot
Copy link
Copy Markdown

emcbot commented May 2, 2025

Experiment C96C48mx500_S2SW_cyc_gfs FAILED on Gaeac6 in Build# 3 with error logs:

/gpfs/f6/drsa-precip3/world-shared/global/CI/3017/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_1881a891/logs/2021122018/enkfgfs_earc_tars_01.log

Follow link here to view the contents of the above file(s): (enkfgfs_earc_tars_01.log)

@emcbot
Copy link
Copy Markdown

emcbot commented May 2, 2025

Experiment C96C48mx500_S2SW_cyc_gfs FAILED on Gaeac6 in Build# 3 in
/gpfs/f6/drsa-precip3/world-shared/global/CI/3017/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_1881a891

@KateFriedman-NOAA
Copy link
Copy Markdown
Contributor

Giving CI status on WCOSS2 before signing off for the weekend: testing on WCOSS2 is nearing completion with all tests completed successfully minus the extended test which is still running (with no issues so far).

Fri May  2 18:52:39 UTC 2025
******** C48_ATM_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 02 2025 15:01:37    May 02 2025 15:56:35

******** C48mx500_3DVarAOWCDA_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 02 2025 15:01:39    May 02 2025 15:16:31
202103250000        Done    May 02 2025 15:01:39    May 02 2025 16:36:22

******** C48mx500_hybAOWCDA_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 02 2025 15:01:41    May 02 2025 15:16:35
202103250000        Done    May 02 2025 15:01:41    May 02 2025 16:16:39

******** C48_S2SW_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 02 2025 15:01:44    May 02 2025 16:02:12

******** C48_S2SWA_gefs_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 02 2025 15:01:46    May 02 2025 16:12:41

******** C96_atm3DVar_extended_3017 ********
202112210000           gfs_fcst_seg1                   194749422             RUNNING                   -         0           0.0
202112210000    gfs_atmos_prod_f339-f360                   194755670             RUNNING                   -         0           0.0
202112210000        gfs_goesupp_f369                   194756453              QUEUED                   -         0           0.0
202112210000        gfs_goesupp_f372                   194756454              QUEUED                   -         0           0.0
202112210000        gfs_goesupp_f375                   194756456              QUEUED                   -         0           0.0
202112210000        gfs_goesupp_f378                   194756457              QUEUED                   -         0           0.0
202112210000    gfs_gempak_f315-f336                   194755675             RUNNING                   -         0           0.0
202112210600           gfs_fcst_seg1                   194753082             RUNNING                   -         0           0.0
202112210600    gfs_atmos_prod_f243-f264                   194756458              QUEUED                   -         0           0.0
202112210600        gfs_goesupp_f261                   194756459              QUEUED                   -         0           0.0
202112210600        gfs_goesupp_f264                   194756461              QUEUED                   -         0           0.0
202112210600        gfs_goesupp_f267                   194756462              QUEUED                   -         0           0.0
202112210600        gfs_goesupp_f270                   194756463              QUEUED                   -         0           0.0
202112210600    gfs_gempak_f147-f168                   194755680             RUNNING                   -         1           0.0
202112210600    gfs_gempak_f195-f216                   194755196             RUNNING                   -         0           0.0
202112210600    gfs_gempak_f219-f240                   194756464              QUEUED                   -         0           0.0
202112211200           gfs_fcst_seg0                   194749770             RUNNING                   -         0           0.0
202112211200    gfs_atmos_prod_f147-f168                   194756466              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f162                   194756467              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f165                   194756468              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f168                   194756469              QUEUED                   -         0           0.0
202112211200    gfs_gempak_f105-f112                   194754780             RUNNING                   -         0           0.0
202112211200    gfs_gempak_f113-f120                   194755202             RUNNING                   -         0           0.0
202112211200    gfs_gempak_f123-f144                   194756470              QUEUED                   -         0           0.0

******** C96C48_hybatmaerosnowDA_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 02 2025 15:01:51    May 02 2025 15:21:47
202112201800        Done    May 02 2025 15:01:51    May 02 2025 17:26:48
202112210000        Done    May 02 2025 15:01:51    May 02 2025 17:16:44

******** C96C48_hybatmDA_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    May 02 2025 15:01:53    May 02 2025 15:21:50
202112210000        Done    May 02 2025 15:01:53    May 02 2025 17:26:51
202112210600        Done    May 02 2025 15:01:53    May 02 2025 17:06:47

******** C96C48mx500_S2SW_cyc_gfs_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 02 2025 15:01:56    May 02 2025 15:21:53
202112201800        Done    May 02 2025 15:01:56    May 02 2025 17:26:55
202112210000        Done    May 02 2025 15:01:56    May 02 2025 18:21:48
202112211800        Done    May 02 2025 15:26:54    May 02 2025 18:21:48

******** C96C48_ufs_hybatmDA_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    May 02 2025 15:01:58    May 02 2025 15:21:57
202402240000        Done    May 02 2025 15:01:58    May 02 2025 17:57:09
202402240600        Done    May 02 2025 15:01:58    May 02 2025 17:42:07

******** C96mx100_S2S_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    May 02 2025 15:02:00    May 02 2025 16:37:05

@emcbot
Copy link
Copy Markdown

emcbot commented May 2, 2025

Experiment C96C48_hybatmaerosnowDA FAILED on Gaeac6 in Build# 3 with error logs:

/gpfs/f6/drsa-precip3/world-shared/global/CI/3017/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_1881a891/logs/2021122100/gdas_arch_tars.log

Follow link here to view the contents of the above file(s): (gdas_arch_tars.log)

@emcbot
Copy link
Copy Markdown

emcbot commented May 2, 2025

Experiment C96C48_hybatmaerosnowDA FAILED on Gaeac6 in Build# 3 in
/gpfs/f6/drsa-precip3/world-shared/global/CI/3017/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_1881a891

@KateFriedman-NOAA
Copy link
Copy Markdown
Contributor

WCOSS2 CI completed minus a few gempak jobs that hit their walltimes. Still investigating cause but am setting this as passed.

Mon May  5 13:04:05 UTC 2025
******** C48_ATM_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 02 2025 15:01:37    May 02 2025 15:56:35

******** C48mx500_3DVarAOWCDA_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 02 2025 15:01:39    May 02 2025 15:16:31
202103250000        Done    May 02 2025 15:01:39    May 02 2025 16:36:22

******** C48mx500_hybAOWCDA_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 02 2025 15:01:41    May 02 2025 15:16:35
202103250000        Done    May 02 2025 15:01:41    May 02 2025 16:16:39

******** C48_S2SW_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 02 2025 15:01:44    May 02 2025 16:02:12

******** C48_S2SWA_gefs_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 02 2025 15:01:46    May 02 2025 16:12:41

******** C96_atm3DVar_extended_3017 ********
202112210600    gfs_gempak_f147-f168                   194755680                DEAD                 -29         2        1853.0
202112211200    gfs_gempak_f123-f144                   194760980                DEAD                 -29         2        1848.0
202112211200    gfs_gempak_f147-f168                   194761997                DEAD                 -29         2        1829.0

******** C96C48_hybatmaerosnowDA_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 02 2025 15:01:51    May 02 2025 15:21:47
202112201800        Done    May 02 2025 15:01:51    May 02 2025 17:26:48
202112210000        Done    May 02 2025 15:01:51    May 02 2025 17:16:44

******** C96C48_hybatmDA_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    May 02 2025 15:01:53    May 02 2025 15:21:50
202112210000        Done    May 02 2025 15:01:53    May 02 2025 17:26:51
202112210600        Done    May 02 2025 15:01:53    May 02 2025 17:06:47

******** C96C48mx500_S2SW_cyc_gfs_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 02 2025 15:01:56    May 02 2025 15:21:53
202112201800        Done    May 02 2025 15:01:56    May 02 2025 17:26:55
202112210000        Done    May 02 2025 15:01:56    May 02 2025 18:21:48
202112211800        Done    May 02 2025 15:26:54    May 02 2025 18:21:48

******** C96C48_ufs_hybatmDA_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    May 02 2025 15:01:58    May 02 2025 15:21:57
202402240000        Done    May 02 2025 15:01:58    May 02 2025 17:57:09
202402240600        Done    May 02 2025 15:01:58    May 02 2025 17:42:07

******** C96mx100_S2S_3017 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    May 02 2025 15:02:00    May 02 2025 16:37:05

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully

Projects

None yet

6 participants