Skip to content

Update eobs ATMGES variable in JJOB for develop#4691

Merged
DavidHuber-NOAA merged 3 commits intoNOAA-EMC:developfrom
CatherineThomas-NOAA:dev_eobs
Mar 20, 2026
Merged

Update eobs ATMGES variable in JJOB for develop#4691
DavidHuber-NOAA merged 3 commits intoNOAA-EMC:developfrom
CatherineThomas-NOAA:dev_eobs

Conversation

@CatherineThomas-NOAA
Copy link
Contributor

@CatherineThomas-NOAA CatherineThomas-NOAA commented Mar 20, 2026

Description

@lgannoaa discovered a silent failure in the v17 real time in the eobs JJOB JGLOBAL_ENKF_SELECT_OBS:

/lfs/h2/emc/gfstemp/emc.global/git/global-workflow/retrov17_01/dev/jobs/JGLOBAL_ENKF_SELECT_OBS: line 62: ATMGES: unbound variable

JGLOBAL_ENKF_SELECT_OBS[62]LEVS=
JGLOBAL_ENKF_SELECT_OBS[63]export err=1
JGLOBAL_ENKF_SELECT_OBS[63]err=1

The variable ATMGES is not defined in this JJOB. This PR replaces this variable with ATMGES_ENSMEAN, eliminating the error and properly defining LEVS.

This PR brings this fix into develop branch. A separate PR was opened to bring this fix into dev/gfs.v17 #4663

Resolves #4615

Type of change

  • Bug fix (fixes something broken)

Change characteristics

  • Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? NO
  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

generate_workflow -G -E -S -C on Gaea C6 and WCOSS2, all Gaea tests passed (will post below), WCOSS2 passed except for gempak jobs that seem unrelated to this PR. Compared results with develop on WCOSS2 for lowres v17 tests and results were identical. Testing done at 28f907f

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@CatherineThomas-NOAA
Copy link
Contributor Author

Gaea-C6 Tests - All Pass

/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C48_ATM_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Mar 16 2026 21:55:07    Mar 16 2026 22:45:24
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C48_gsienkf_atmDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Mar 16 2026 21:55:07    Mar 16 2026 22:15:25
202402240000        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:05:20
202402240600        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:50:16
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C48mx500_3DVarAOWCDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:00:25
202103250000        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:30:17
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C48mx500_hybAOWCDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Mar 16 2026 21:55:07    Mar 16 2026 22:15:31
202103250000        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:05:27
 
SKIP C48_S2SWA_gefs_RT on gaeac6
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C48_S2SWA_gefs_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Mar 16 2026 21:55:09    Mar 16 2026 23:10:24
 
SKIP C48_S2SW_extended on gaeac6
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C48_S2SW_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Mar 16 2026 21:55:09    Mar 16 2026 22:50:10
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C48_ufsenkf_atmDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Mar 16 2026 21:55:07    Mar 16 2026 22:15:25
202402240000        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:15:11
202402240600        Done    Mar 16 2026 21:55:07    Mar 17 2026 00:05:09
 
SKIP C96_atm3DVar_extended on gaeac6
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C96_atm3DVar_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Mar 16 2026 21:55:06    Mar 16 2026 22:15:23
202112210000        Done    Mar 16 2026 21:55:06    Mar 16 2026 23:45:13
202112210600        Done    Mar 16 2026 21:55:06    Mar 16 2026 23:55:21
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C96C48_hybatmDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Mar 16 2026 21:55:07    Mar 16 2026 22:15:25
202112210000        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:40:21
202112210600        Done    Mar 16 2026 21:55:07    Mar 17 2026 00:05:17
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C96C48_hybatmsnowDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Mar 16 2026 21:55:07    Mar 16 2026 22:15:24
202112201800        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:40:16
202112210000        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:50:16
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C96C48_hybatmsoilDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202205150600        Done    Mar 16 2026 21:55:15    Mar 16 2026 22:20:23
202205151200        Done    Mar 16 2026 21:55:15    Mar 16 2026 23:45:18
202205151800        Done    Mar 16 2026 21:55:15    Mar 17 2026 00:05:14
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C96C48mx500_S2SW_cyc_gfs_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Mar 16 2026 21:55:08    Mar 16 2026 23:20:11
202112201800        Done    Mar 16 2026 21:55:08    Mar 17 2026 00:25:25
202112210000        Done    Mar 16 2026 21:55:08    Mar 17 2026 00:50:14
 
SKIP C96C48_ufsgsi_hybatmDA on gaeac6
 
SKIP C96C48_ufs_hybatmDA on gaeac6
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C96_gcafs_cycled_noDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Mar 16 2026 21:55:07    Mar 16 2026 22:15:23
202112201800        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:35:24
202112210000        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:10:24
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C96_gcafs_cycled_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Mar 16 2026 21:55:07    Mar 16 2026 22:15:25
202112201800        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:50:16
202112210000        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:40:16
 
/gpfs/f6/gfs-cpu/world-shared/Catherine.Thomas/tmp/RUNTESTS_dev_eobs3/EXPDIR/C96mx100_S2S_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Mar 16 2026 21:55:07    Mar 16 2026 23:00:33

@CatherineThomas-NOAA
Copy link
Contributor Author

WCOSS2 Tests

/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C48_ATM_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Mar 16 2026 22:00:10    Mar 16 2026 23:00:12
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C48_gsienkf_atmDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Mar 16 2026 22:00:29    Mar 16 2026 22:20:37
202402240000        Done    Mar 16 2026 22:00:29    Mar 16 2026 23:20:37
202402240600        Done    Mar 16 2026 22:00:29    Mar 17 2026 00:15:29
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C48mx500_3DVarAOWCDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Mar 16 2026 22:00:31    Mar 16 2026 23:20:40
202103250000        Done    Mar 16 2026 22:00:31    Mar 16 2026 23:45:37
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C48mx500_hybAOWCDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Mar 16 2026 22:00:14    Mar 16 2026 22:25:39
202103250000        Done    Mar 16 2026 22:00:14    Mar 16 2026 23:35:14
 
SKIP C48_S2SWA_gefs_RT on wcoss2
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C48_S2SWA_gefs_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Mar 16 2026 22:00:17    Mar 16 2026 23:30:11
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C48_S2SW_extended_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200      Active    Mar 16 2026 22:00:35             -          
202103231800        Done    Mar 16 2026 22:00:35    Mar 17 2026 00:25:38
 
SKIP C48_S2SW on wcoss2
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C48_ufsenkf_atmDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Mar 16 2026 22:00:32    Mar 16 2026 22:25:37
202402240000        Done    Mar 16 2026 22:00:32    Mar 16 2026 23:25:36
202402240600        Done    Mar 16 2026 22:00:32    Mar 17 2026 00:25:16
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C96_atm3DVar_extended_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Mar 16 2026 22:00:19    Mar 17 2026 00:25:39
202112210000      Active    Mar 16 2026 22:00:19             -          
202112210600      Active    Mar 16 2026 22:00:19             -          
202112211200      Active    Mar 17 2026 00:30:33             -          
202112211800    Inactive             -                       -          
 
SKIP C96_atm3DVar on wcoss2
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C96C48_hybatmDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Mar 16 2026 22:00:15    Mar 16 2026 22:25:14
202112210000        Done    Mar 16 2026 22:00:15    Mar 17 2026 00:15:13
202112210600        Done    Mar 16 2026 22:00:15    Mar 17 2026 00:15:13
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C96C48_hybatmsnowDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Mar 16 2026 22:00:36    Mar 16 2026 22:25:40
202112201800        Done    Mar 16 2026 22:00:36    Mar 17 2026 00:25:29
202112210000        Done    Mar 16 2026 22:00:36    Mar 17 2026 00:20:36
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C96C48_hybatmsoilDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202205150600        Done    Mar 16 2026 22:00:29    Mar 16 2026 22:25:18
202205151200        Done    Mar 16 2026 22:00:29    Mar 17 2026 00:25:38
202205151800        Done    Mar 16 2026 22:00:29    Mar 17 2026 00:30:28
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C96C48mx500_S2SW_cyc_gfs_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Mar 16 2026 22:00:39    Mar 16 2026 23:55:21
202112201800        Done    Mar 16 2026 22:00:39    Mar 17 2026 01:00:36
202112210000        Done    Mar 16 2026 22:00:39    Mar 17 2026 01:20:32
202112211800        Done    Mar 17 2026 00:00:43    Mar 17 2026 01:31:04
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C96C48_ufsgsi_hybatmDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Mar 16 2026 22:00:31    Mar 16 2026 22:25:18
202402240000        Done    Mar 16 2026 22:00:31    Mar 17 2026 00:35:36
202402240600        Done    Mar 16 2026 22:00:31    Mar 17 2026 00:35:36
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C96C48_ufs_hybatmDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Mar 16 2026 22:00:12    Mar 16 2026 22:25:16
202402240000        Done    Mar 16 2026 22:00:12    Mar 17 2026 00:35:27
202402240600        Done    Mar 16 2026 22:00:12    Mar 17 2026 00:10:16
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C96_gcafs_cycled_noDA_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Mar 16 2026 22:00:30    Mar 16 2026 22:25:56
202112201800        Done    Mar 16 2026 22:00:30    Mar 17 2026 00:35:56
202112210000        Done    Mar 16 2026 22:00:30    Mar 16 2026 23:25:41
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C96_gcafs_cycled_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Mar 16 2026 22:00:11    Mar 16 2026 22:25:13
202112201800        Done    Mar 16 2026 22:00:11    Mar 17 2026 00:40:18
202112210000        Done    Mar 16 2026 22:00:11    Mar 17 2026 00:00:14
 
/lfs/h2/emc/ptmp/catherine.thomas/RUNTESTS_dev_eobs3/EXPDIR/C96mx100_S2S_dev_eobs3
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Mar 16 2026 22:00:29    Mar 16 2026 23:45:29

@CatherineThomas-NOAA
Copy link
Contributor Author

Failed WCOSS2 Tests

C96_atm3DVar_extended_dev_eobs3

C96_atm3DVar_extended_dev_eobs3 active tasks:
202112210000          gfs_gempakmeta                   254673314                DEAD                 271         2          46.0
202112210000     gdas_gempakmetancdc                   254654043                DEAD                 271         2          37.0
202112210600          gfs_gempakmeta                   254678616                DEAD                 271         2          38.0
202112210600     gdas_gempakmetancdc                   254659667                DEAD                 271         2          39.0
202112211200          gfs_gempakmeta                   254684856                DEAD                 271         2          43.0
202112211200     gdas_gempakmetancdc                   254668573                DEAD                 271         2          40.0

C48_S2SW_extended_dev_eobs3

C48_S2SW_extended_dev_eobs3 active tasks:
202103231200          gfs_gempakmeta                   254659294                DEAD                 271         2          38.0

This errors do not seem to be related to this PR. Opening for review.

@CatherineThomas-NOAA CatherineThomas-NOAA marked this pull request as ready for review March 20, 2026 19:21
Copy link
Contributor

@TravisElless-NOAA TravisElless-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change here is identical to the change test and approved for v17 in #4663. The observed failures in testing of this PR were likely corrected by #4662, which was not included in the hash that ran the test. Approving this change for develop

@DavidHuber-NOAA
Copy link
Contributor

Merging.

Also, I happen to be running a baseline test on WCOSS2 from develop. If there are errors in the gempakmeta jobs, I'll open a follow-up issue.

@DavidHuber-NOAA DavidHuber-NOAA merged commit c0cab10 into NOAA-EMC:develop Mar 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ENKF_SELECT_OBS job silent failure

3 participants