Enable triggering restart writes at any forecast time ("restart_fh") via UFS configuration as for other components

[NickSzapiro-NOAA]

For detailed information about submitting Pull Requests (PRs) to the CICE-Consortium,
please refer to: https://github.com/CICE-Consortium/About-Us/wiki/Resource-Index#information-for-developers

PR checklist

• Short (1 sentence) summary of your PR:
  Enable writing forecast hour defined restarts ("restart_fh") as delimited floating point forecast hours in UFS model_configure component attributes
• Developer(s):
  Nick Szapiro, Denise Worthen, & Jun Wang
• Suggest PR reviewers from list in the column to the right.
• Please copy the PR test results link or provide a summary of testing completed below.
  Testing in UFS with cpld_control_gfsv17 confirms that restarts are bit-for-bit when (1) sharing common times with RESTART_N and (2) instead of RESTART_N. This feature is exercised in ufs-weather-model regression testing (see Flexible restart write times (restart_fh) for med, ocn, ice, wav ufs-community/ufs-weather-model#2419)
• How much do the PR code changes differ from the unmodified code?
  • bit for bit
  • different at roundoff level
  • more substantial
• Does this PR create or have dependencies on Icepack or any other models?
  • Yes
  • No
• Does this PR update the Icepack submodule? If so, the Icepack submodule must point to a hash on Icepack's main branch.
  • Yes
  • No
• Does this PR add any new test cases?
  • Yes
  • No
• Is the documentation being updated? ("Documentation" includes information on the wiki or in the .rst files from doc/source/, which are used to create the online technical docs at https://readthedocs.org/projects/cice-consortium-cice/. A test build of the technical docs will be performed as part of the PR testing.)
  • Yes
  • No, does the documentation need to be updated at a later time?
    • Yes
    • No Docs for caps remain in caps
• Please document the changes in detail, including why the changes are made. This will become part of the PR commit log.

This PR enables writing forecast hour defined restarts ("restart_fh") in the same way as space-delimited floating point forecast hours in model_configure attributes, like restart_fh: 0.25 2.5 6 17 24. The implementation builds on current restart_fh in MOM6 for CMEPS, MOM6, CICE, and WW3 to have option of forecast hour restarts in addition to existing functionality. Restart writes are triggered when input restart_fh forecast hours are evenly divisible by a component's timestep (internally compared in units of integer seconds) and skipped otherwise. These restarts may be particularly helpful for data assimilation.

Enabling the functionality depends on UFS shared code (hence the #ifndef blocks), but this only depends on ESMF and could be adapted and shared with CESM as well, if CESM would like:
https://github.com/ufs-community/ufs-weather-model/blob/develop/CDEPS-interface/ufs/cdeps_share/shr_is_restart_fh_mod.F90

[apcraig]

@dabail10, @anton-seaice, please have a quick look and provide feedback/approve as needed.

[anton-seaice]

No concerns from me.

Where is shr_is_restart_fh_mod defined ? Is it more adaptable to add an #ifdef UFSCOUPLED instead of relying on #ifndef CESMCOUPLED having this meaning?

[NickSzapiro-NOAA]

Thanks @anton-seaice. Maybe the main goal is not to have many of these #ifdefs...certainly open to being more adaptable if it helps.

shr_is_restart_fh_mod is in UFS so that components don't each need to maintain separate copies (https://github.com/ufs-community/ufs-weather-model/blob/develop/CICE-interface/cice_files.cmake#L173)

[dabail10]

This looks fine, but I think this is a bit of an issue where we have all of these different methods of reading / writing restarts depending on the coupled system.

[DeniseWorthen]

Two comments here, just my .02....

The ifdef CESMCOUPLED is the ifdef we've tried to use across all component models which CESM and UFS share (WW3, CMEPS etc), so I think it makes more sense than creating a new UFSCOUPLED ifdef.

The issue of restart writing frequency is probably just going to be fundamentally different in a weather-forecasting setting than in a climate/research setting. The more flexible solution is to have this capability defined by ESMF (through config), but for now that capability doesn't exist.