Update high resolution tests and fix minor wave issues by JessicaMeixner-NOAA · Pull Request #3289 · NOAA-EMC/global-workflow

JessicaMeixner-NOAA · 2025-01-30T17:36:13Z

Description

This PR updates the wave fix which was unintentionally left out of PR #3190
Additionally, this updates what ICs the highres ci tests are pointing to (IC update issues: #3287 #3288)

-- Fixes #3260
-- Refs #3326 - small changes in comments

Additionally, some smaller bug fixes and changes were made:

Removes logic that tries to copy files to fix directory in ush/wave_grid_interp_sbs.sh
-- Fixes logic issue in ush/wave_grid_interp_sbs.sh #3162
Adds MPI variables that were found needed when investigating the HR4 hang HRv4 hangs on orion and hercules ufs-community/ufs-weather-model#2486
-- Fixes Add MPI environment variables for orion/hercules #3259
Changes the log file output directory corresponding to an update requested by NCO in v16.3.22 in scripts/exgfs_wave_post_pnt.sh
Through debugging HR5 (Include fixes from bugs found when launching HR5 #3260) it was noticed the wave ICs were not used. While there is a code change required (Add reading of binary restart with PIO for runtype=initial WW3#1359), this PR fixes the scripting level issues & updates UFS to fix WW3 code issues.
Removes COMOUT_WAVE_PREP from wave post job that did not need it, plus ensured temporary files were not copied back to COM and then over-written (NCO bug)
--- Fixes [NCO Bug] Wave post should not copy files to COM whose names might collide #297
Updates ufs-weather-model hash. Details in table below

Notes on ufs-weather-model updates:

Commit date	Commit hash/ PR	Notes for g-w changes	Baseline Changes
Jan 7, 2025	ufs-community/ufs-weather-model@`5324d64` ufs-community/ufs-weather-model#2531	n/a	changes in regional tests, no global changes
Jan 13, 2025	ufs-community/ufs-weather-model@`c7412fa`	n/a	changes in some global coupled tests
Jan 17, 2025	ufs-community/ufs-weather-model@`3a5e52e`	n/a	no changes
Jan 21, 2025	ufs-community/ufs-weather-model@`33cde4b`	extra variable needed for waves	no changes
Jan 22, 2025	ufs-community/ufs-weather-model@`b9d2bec`	n/a	no changes
Jan 28, 2025	ufs-community/ufs-weather-model@`0682f90`	n/a
Feb 3, 2025	ufs-community/ufs-weather-model@`a498735`	add fv3atm_output_dir to model_configure	answers change
Feb 5, 2025	ufs-community/ufs-weather-model@`ee3d52d`	changes in this PR for initialization for wave fixes	if using WW3 binary restart, answers will change

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

While the full fix for the wave IC issue does require code changes, these changes are still needed and can be included before the WW3 update is there.

How has this been tested?

Ran a C48 S2SW with the WW3 code update: /scratch1/NCEPDEV/climate/Jessica.Meixner/UpdateFixandHiresIC/c48t05 (Also earlier ran ci/cases/hires/C768_S2SW.yaml which ran out of time in the forecast but otherwise was succeeding: /scratch1/NCEPDEV/climate/Jessica.Meixner/UpdateFixandHiresIC/c768t01 )
hires/C768_S2SW.yaml on orion is in progress: /work2/noaa/marine/jmeixner/UpdateFixandHiresIC/t01c768
hires/C1152_S2SW.yaml on hercules is in progress: /work2/noaa/marine/jmeixner/hercules/UpdateFixandHiresIC/t01c1152

Checklist

fix broken logic so that we do not try to copy somethign to COM

and subsequent scripts

for addressing hang issues

jobs/JGLOBAL_WAVE_POST_SBS

WalterKolczynski-NOAA

LGTM

WalterKolczynski-NOAA · 2025-02-03T15:49:01Z

Hold CI until the new ICs gets staged

sbanihash

changes in ush/forecast_postdet.sh look good to me. I did a test only with this change and the restart reading is working as expected. I have not tested changes within the post job.

aerorahul

lgtm

WalterKolczynski-NOAA · 2025-02-05T15:50:23Z

@JessicaMeixner-NOAA would you please merge in develop before we start CI? There were a couple issues that were corrected yesterday that I want to capture.

JessicaMeixner-NOAA · 2025-02-05T15:58:28Z

@WalterKolczynski-NOAA - Merged develop.

emcbot · 2025-02-18T21:26:31Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 1 with error logs:

/work2/noaa/global/CI/HERCULES/3289/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d0923663/logs/2021032418/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-02-18T21:26:39Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 1 in
/work2/noaa/global/CI/HERCULES/3289/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_d0923663

emcbot · 2025-02-19T02:07:25Z

CI Failed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3289


Experiment C48mx500_3DVarAOWCDA_d0923663 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Tue Feb 18 15:26:13 CST 2025
Experiment C48mx500_3DVarAOWCDA_d0923663 Terminated: *FAIL*
Error logs:
/work2/noaa/global/CI/HERCULES/3289/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d0923663/logs/2021032418/gdas_fcst_seg0.log
Experiment C48_ATM_d0923663 Completed 1 Cycles: *SUCCESS* at Tue Feb 18 16:20:49 CST 2025
Experiment C48mx500_hybAOWCDA_d0923663 Completed 2 Cycles: *SUCCESS* at Tue Feb 18 17:15:23 CST 2025
Experiment C96_S2SWA_gefs_replay_ics_d0923663 Completed 1 Cycles: *SUCCESS* at Tue Feb 18 17:46:12 CST 2025
Experiment C96_atm3DVar_d0923663 Completed 3 Cycles: *SUCCESS* at Tue Feb 18 18:34:09 CST 2025
Experiment C96C48_hybatmDA_d0923663 Completed 3 Cycles: *SUCCESS* at Tue Feb 18 18:34:30 CST 2025
Experiment C48_S2SW_d0923663 Completed 1 Cycles: *SUCCESS* at Tue Feb 18 18:40:20 CST 2025
Experiment C48_S2SWA_gefs_d0923663 Completed 1 Cycles: *SUCCESS* at Tue Feb 18 20:07:12 CST 2025

JessicaMeixner-NOAA · 2025-02-19T13:18:01Z

Not sure why these tests were launched last night. I haven't pushed the changes for the IC date yet because I hadn't had a chance to test and confirm things are working.

JessicaMeixner-NOAA · 2025-02-19T16:35:53Z

This PR is now ready for retesting through the CI system. @WalterKolczynski-NOAA @aerorahul as I have finally had a successful test on hera of C48mx500_3DVarAOWCDA (minus the archiving because I forgot to change it to an area I had access to).

Please let me know if there's anything else needed from me for this PR.

DavidHuber-NOAA · 2025-02-19T18:49:39Z

Manual CI has started on Hera. The clone is located in /scratch1/NCEPDEV/global/David.Huber/GW/pr_3289 and tests will be run in /scratch1/NCEPDEV/global/David.Huber/pr/3289.

emcbot · 2025-02-19T22:01:18Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 3 in
/work2/noaa/global/CI/HERCULES/3289/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_32f24687

emcbot · 2025-02-19T22:26:50Z

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 3 in
/work2/noaa/global/CI/HERCULES/3289/RUNTESTS/EXPDIR/C48_S2SWA_gefs_32f24687

emcbot · 2025-02-19T22:32:57Z

CI Failed on Hercules in Build# 3
Built and ran in directory /work2/noaa/global/CI/HERCULES/3289


Experiment C48_ATM_32f24687 Completed 1 Cycles: *SUCCESS* at Wed Feb 19 13:35:59 CST 2025
Experiment C48mx500_hybAOWCDA_32f24687 Completed 2 Cycles: *SUCCESS* at Wed Feb 19 14:18:48 CST 2025
Experiment C96_S2SWA_gefs_replay_ics_32f24687 Completed 1 Cycles: *SUCCESS* at Wed Feb 19 14:43:12 CST 2025
Experiment C96_atm3DVar_32f24687 Completed 3 Cycles: *SUCCESS* at Wed Feb 19 15:13:40 CST 2025
Experiment C96C48_hybatmDA_32f24687 Completed 3 Cycles: *SUCCESS* at Wed Feb 19 15:13:45 CST 2025
Experiment C48mx500_3DVarAOWCDA_32f24687 Completed 2 Cycles: *SUCCESS* at Wed Feb 19 15:44:22 CST 2025
Experiment C48_S2SWA_gefs_32f24687 Completed 1 Cycles: *SUCCESS* at Wed Feb 19 16:20:54 CST 2025
Experiment C48_S2SW_32f24687 Completed 1 Cycles: *SUCCESS* at Wed Feb 19 16:32:39 CST 2025

WalterKolczynski-NOAA · 2025-02-20T01:41:09Z

CI on Hercules actually seems to have passed, but some Jenkins error resulted in an incorrect status after the fact.

DavidHuber-NOAA · 2025-02-20T12:44:41Z

The C96C48_hybatmaerosnowDA test failed on Hera on cycle 2021122018 for job gfs_metpg2o1. The log can be found here: /scratch1/NCEPDEV/global/David.Huber/pr/3289/COMROOT/C96C48_hybatmaerosnowDA_3289/logs/2021122018/gfs_metpg2o1.log.0.

The error stems from being unable to read the file /scratch1/NCEPDEV/global/Mallory.Row/prepbufr/gdas/prepbufr.gdas.2021122018. This is odd for 2 reasons: 1) I was able to read that exact same file when running the C96C48_hybatmaerosnowDA test for PR #3238 just one hour earlier and 2) that file does exist. I am rebooting this job now to see if it works this morning.

DavidHuber-NOAA · 2025-02-20T12:56:29Z

The gfs_metpg2o1 job passed on the third attempt. The test is now complete and all tests passed on Hera. Setting CI-Hera-Passed label.

JessicaMeixner-NOAA added 11 commits January 22, 2025 18:10

update fix for wave

d52977d

do not copy temp files to COM as they are not needed

26765f6

fix broken logic so that we do not try to copy somethign to COM

remove un-used COMOUT_WAVE_PREP in JGLOBAL_WAVE_POST_SBS

ac141b6

and subsequent scripts

change name of restart variables

9ac3313

update ic date

02ca6ae

Merge remote-tracking branch 'EMC/develop' into fixhighrestest

5b97753

Merge remote-tracking branch 'EMC/develop' into fixhighrestest

754f0b0

changes requested by NCO for v16 adding to develop

3342450

Merge remote-tracking branch 'EMC/develop' into fixhighrestest

e54d216

add environment variable for fcst job for hercules/orion

a06de0f

for addressing hang issues

update WW3 IC error handling

a9d8f7e

JessicaMeixner-NOAA requested review from WalterKolczynski-NOAA, aerorahul and sbanihash as code owners January 30, 2025 17:36

github-advanced-security bot found potential problems Jan 30, 2025

View reviewed changes

jobs/JGLOBAL_WAVE_POST_SBS Fixed Show fixed Hide fixed

This was referenced Jan 30, 2025

Add new ICs for new wave grid + Move to HR5 atm ICS for C1152mx025 #3287

Closed

Add new ICs for new wave grid + Move to HR5 atm ICS for C768mx025 #3288

Closed

remove loop in JGLOBAL_WAVE_POST_SBS

ee352fe

WalterKolczynski-NOAA previously approved these changes Feb 3, 2025

View reviewed changes

sbanihash reviewed Feb 3, 2025

View reviewed changes

sbanihash mentioned this pull request Feb 3, 2025

Add reading of binary restart with PIO for runtype=initial NOAA-EMC/WW3#1359

Merged

4 tasks

aerorahul previously approved these changes Feb 4, 2025

View reviewed changes

Merge branch 'NOAA-EMC:develop' into fixhighrestest

583e2fb

WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Feb 5, 2025

emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Feb 5, 2025

JessicaMeixner-NOAA mentioned this pull request Feb 13, 2025

Gaea: Issues running C768 S2SW #3324

Closed

JessicaMeixner-NOAA and others added 4 commits February 13, 2025 13:15

Merge branch 'NOAA-EMC:develop' into fixhighrestest

7207ad9

update comments

c707bff

Merge branch 'NOAA-EMC:develop' into fixhighrestest

7de44e0

Merge branch 'NOAA-EMC:develop' into fixhighrestest

d092366

WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Feb 18, 2025

emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Feb 18, 2025

emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Feb 19, 2025

update IC date of ci/cases/pr/C48mx500_3DVarAOWCDA.yaml

32f2468

WalterKolczynski-NOAA approved these changes Feb 20, 2025

View reviewed changes

Conversation

JessicaMeixner-NOAA commented Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

Uh oh!

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

WalterKolczynski-NOAA commented Feb 3, 2025

Uh oh!

sbanihash left a comment

Choose a reason for hiding this comment

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

WalterKolczynski-NOAA commented Feb 5, 2025

Uh oh!

JessicaMeixner-NOAA commented Feb 5, 2025

Uh oh!

emcbot commented Feb 18, 2025

Uh oh!

emcbot commented Feb 18, 2025

Uh oh!

emcbot commented Feb 19, 2025

Uh oh!

JessicaMeixner-NOAA commented Feb 19, 2025

Uh oh!

JessicaMeixner-NOAA commented Feb 19, 2025

Uh oh!

DavidHuber-NOAA commented Feb 19, 2025

Uh oh!

emcbot commented Feb 19, 2025

Uh oh!

emcbot commented Feb 19, 2025

Uh oh!

emcbot commented Feb 19, 2025

Uh oh!

WalterKolczynski-NOAA commented Feb 20, 2025

Uh oh!

DavidHuber-NOAA commented Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHuber-NOAA commented Feb 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

JessicaMeixner-NOAA commented Jan 30, 2025 •

edited

Loading

DavidHuber-NOAA commented Feb 20, 2025 •

edited

Loading