Forecast jobs sometimes failing on reading ocean_geometry file

[shlyaeva]

What is wrong?

I, Laura Slivinski and @JohnSteffen-NOAA have noticed that sometimes forecast jobs fail with:

204: FATAL from PE 0: Permission denied: netcdf_file_open:MOM6_OUTPUT/ocean_geometry.nc

Here is a specific example of my run where it happened. Experiment yaml is based on dev/ci/cases/gfsv17/C384mx025_hybAOWCDA.yaml.

• job enkfgdas_fcst_mem028 failed on the first attempt due to running out of walltime
• second attempt: job fails with the Permission denied: netcdf_file_open:MOM6_OUTPUT/ocean_geometry.nc message.
• manual reboot of the job: job fails with the Permission denied: netcdf_file_open:MOM6_OUTPUT/ocean_geometry.nc message.

Laura reported that removing the offending RUNDIR and rebooting the job after that ocean_geometry failure solved the issue for her.

I checked that RUNDIR indeed contains unreadable ocean_geometry.nc file in RUNDIRS/<expname>/enkfgdas.2025101000/enkfgdasefcs028.2025101000/output/MOM6_OUTPUT/ocean_geometry.nc and manually removed it. The job succeeded when rebooting after that.

What should have happened?

I expect the job to rerun successfully without the need to manually remove files from run directories

What machines are impacted?

All or N/A

What global-workflow hash are you using?

debf5c9

Steps to reproduce

See above.

Additional information

No response

Do you have a proposed solution?

Perhaps some cleanup needs to happen when the forecast executable fails, I am not sure.

[shlyaeva]

This appears to be duplicate of #4418

[DavidHuber-NOAA]

@shlyaeva since this issue shows that this can occur in the GFS as well, I will close #4418 as a duplicate of this issue as it appears to be a larger-scale issue.

[JessicaMeixner-NOAA]

@jiandewang @sanAkel - Any thoughts on this issue?

[NeilBarton-NOAA]

FYI, @YangxingZheng-NOAA We repeatedly see this in sfs. Does in only occur during running ensembles?

[NeilBarton-NOAA]

MOM6 write out this file. For ensembles (at least in SFS), all members write out to the same location. I'm curious if this is the issue.... It's hard to debug as it does not consistently error out for me

[YangxingZheng-NOAA]

Hi, I may have resolved the issue with the help of Jiande. I have tested on Ursa and the issue did not appear any more. To check and handle the ocean_geometry.nc, I add four lines in the MOM6_predet() of forecast_predet.sh: ......................................... MOM6_predet() { echo "SUB ${FUNCNAME[0]}: MOM6 before run type determination" if [[ ! -d "${COMOUT_OCEAN_HISTORY}" ]]; then mkdir -p "${COMOUT_OCEAN_HISTORY}"; fi if [[ ! -d "${COMOUT_OCEAN_RESTART}" ]]; then mkdir -p "${COMOUT_OCEAN_RESTART}"; fi if [[ ! -d "${COMIN_OCEAN_INPUT}" ]]; then mkdir -p "${COMIN_OCEAN_INPUT}"; fi if [[ ! -d "${DATAoutput}/MOM6_OUTPUT" ]]; then mkdir -p "${DATAoutput}/MOM6_OUTPUT"; fi if [[ ! -d "${DATArestart}/MOM6_RESTART" ]]; then mkdir -p "${DATArestart}/MOM6_RESTART"; fi # check if ocean_geometry.nc in DATAoutput/MOM6_OUTPUT exists and its size, if size is zero, then delete it manually if [[ -f "${DATAoutput}/MOM6_OUTPUT/ocean_geometry.nc" && ! -s "${DATAoutput}/MOM6_OUTPUT/ocean_geometry.nc" ]]; then rm -f "${DATAoutput}/MOM6_OUTPUT/ocean_geometry.nc" fi # Link the output and restart directories to the DATA directory ${NLN} "${DATAoutput}/MOM6_OUTPUT" "${DATA}/MOM6_OUTPUT" ${NLN} "${DATArestart}/MOM6_RESTART" "${DATA}/MOM6_RESTART" ......................... Yangxing Zheng Physical Scientist Contractor with Lynker in support of NOAA/NWS/NCEP/EMC 5830 University Research CT, Cubicle 2019 College Park, MD 20740 Tel: (301) 615-0768 ***@***.***

…

On Thu, Jan 29, 2026 at 9:21 AM Neil Barton ***@***.***> wrote: *NeilBarton-NOAA* left a comment (NOAA-EMC/global-workflow#4490) <#4490 (comment)> MOM6 write out this file. For ensembles (at least in SFS), all members write out to the same location. I'm curious if this is the issue.... It's hard to debug as it does not consistently error out for me — Reply to this email directly, view it on GitHub <#4490 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BBQ5GU6RLUZFK4AOA43KA2D4JIJM7AVCNFSM6AAAAACTIBME4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTQMJXHE4TIMJSGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[sanAkel]

Set

Write_geometry=0 of False

[JessicaMeixner-NOAA]

Set

Write_geometry=0 of False

If the file isn't needed... does anyone know if this is used other places?

[jiandewang]

geometry file is not being used in any place. It is kind of "fixed" file which can always be generated in one shot if we really needed that in case.

[jiandewang]

MOM6 write out this file. For ensembles (at least in SFS), all members write out to the same location. I'm curious if this is the issue.... It's hard to debug as it does not consistently error out for me

this is very likely the reason. Multiple members trying to write to the same address at the same time which lead to a zero sized file. This file is being generated in the very beginning of the model run before model start the integration.

[jiandewang]

quick and clean soultion is set WRITE_GEOMETRY=0 (as @sanAkel said).

[NeilBarton-NOAA]

agreed, if we don't use the fine, why write it