Model instability

[benjamin-cash]

I have begun testing C192mx025 on my local cluster, but the initial set of runs are either hanging right after MOM initialization completes or blowing up with the following:

FATAL from PE 62: NaN in input field of mpp_reproducing_sum(_2d), this indicates numerical instability

Image PC Routine Line Source
ufs_model.x 00000000042DFD47 mpp_mod_mp_mpp_er 80 mpp_util_mpi.inc
ufs_model.x 000000000458DFE6 mpp_efp_mod_mp_mp 195 mpp_efp.F90
ufs_model.x 00000000044E5B79 mpp_domains_mod_m 143 mpp_global_sum.fh
ufs_model.x 0000000001C946F0 fv_grid_utils_mod 3077 fv_grid_utils.F90
ufs_model.x 0000000001CCD959 fv_mapz_mod_mp_la 794 fv_mapz.F90
libiomp5.so 000014B36F167D93 __kmp_invoke_micr Unknown Unknown

It looks like this is probably related to this issue ufs-community/ufs-weather-model#2423. @NeilBarton-NOAA , there is also some discussion of a CICE related issue here: ufs-community/ufs-weather-model#2320

I think for the moment I will try reducing the timestep and resubmitting my runs and see if I get past the instability.

[benjamin-cash]

Update - going all the way down to DELTIM=150 did not solve the problem. Another issue ufs-community/ufs-weather-model#1042 with this same instability was addressed by changing cdmbgwd, so I will try that next.

[benjamin-cash]

FYI, there is a discussion about the instabilities going here ufs-community/ufs-weather-model#2447, including what are potentially some pretty big changes to the gravity wave drag settings.