Add Restart Capability: jrtofs_global_hycom_var, jrtofs_global_forecast_grib2_post, jrtofs_global_ncoda_qc (bugzilla #1637)

[DanIredell-NOAA]

http://www2.spa.ncep.noaa.gov/bugzilla/show_bug.cgi?id=1637

Standards require restart capability for jobs running longer than 15 minutes.

At present, RTOFS jobs running longer than 15 minutes and lacking restart capability include:

• jrtofs_global_hycom_var
• jrtofs_global_forecast_grib2_post
• jrtofs_global_ncoda_qc

DA and post jobs that process files can keep a log of processed files from which a restart may be resumed.

If NCODA is to be replaced with JEDI, then a restart capability will need to be included for that implementation.

---

jrtofs_global_hycom_var should be omitted from the list of jobs requiring a restart capability (Implementation Standards v1.0 pg 13: "Data assimilation jobs are exempt from [the restart capability] requirement, but steps should be taken to
minimize runtimes and enhance re-runnability of these processes.")

[DanIredell-NOAA]

Job jrtofs_global_ncoda_qc should also be exempt under the same reason as jrtofs_global_hycom_var (it is a data assimilation job).

Here is some info:

qc - this job is currently taking approx. 16.2 minutes. jpss and runsurf qc jobs take about 11 minutes. Add in the cp data in, alarm, and cp data out gets the total to 16 minutes.

hycom_var - total time is 47 minutes. Breakdown of internal steps:
cp data to data- 2 minutes
setup - 3 minutes
prep - 18 minutes
ncoda3d - 12 minutes
post - 8 minutes
cp data to comout - 2 minutes

forecast_grib2_post (d01 through d04) - these jobs range from 29 to 37 minutes.