allow MOM6 to write non-interval restarts

[DeniseWorthen]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full RT suite (compared to current baselines) on either Hera/Derecho/Hercules AND have committed the log to my PR branch.
• Add list of all failed regression tests in "Regression Tests" section.

PR Information

Description

Allows a non-interval restart for MOM6 by setting a configuration attribute with the list of additional restart hours required. For example, when intialized from 2021-03-22-06, the following will produce additional restarts at 2021-03-22-09, 2021-03 -22-15 and 2021-03-22-21

# OCN #
OCN_model:                      mom6
  (snip)
  restart_fh = 3,9,15
::

Restarts at intervals are not impacted.

Commit Message

Add non-interval restart capability to MOM6

Priority

• Critical Bugfix (This PR contains a critical bug fix and should be prioritized.)
• High (This PR contains a feature or fix needed for a time-sensitive project (eg, retrospectives, implementations))
• Normal.

Blocking Dependencies

• closes Add non-interval restart capability NOAA-EMC/MOM6#126

Git Issues Fixed By This PR

• fixes Enable controlling MOM6 ocean restart files with a more flexible approach #1976

Changes

Subcomponent (with links)

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Input data

• No changes are expected to input data.
• Changes are expected to input data:
  • New input data.
  • Updated input data.

Regression Tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

FAILED REGRESSION TESTS

Libraries

• Not Needed
• Needed
  • Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
  • Add issue link from JCSDA/spack-stack following this item

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[DeniseWorthen]

Combine with PR #2085. Closing.