Jwade ngwpc 10211 wcoss

[jswade-rtx]

Refactor forecast manager to remove ensemble workflow and add no_valid mode for WCOSS., enabling the fcst-mgr to run real-time forecast configurations using the default/regionalization run types instead of only the existing calibration/validation/forecast workflow.

Changes
Lagged Ensemble Removal
The lagged ensemble workflow has been removed from the fcst-mgr to make it compatible with WCOSS parallel execution and to allow checkpoint restarting. Lagged ensembles are now set up by the msw-mgr directly, with each individual ensemble member submitted as a standalone forecast run using the run_forecast workflow.

no_valid Flag
Added a no_valid flag to the fcst-mgr to support two execution modes:

• no_valid=False (default): Existing behavior, the fcst-mgr runsf rom an existing validation run, with the validation configuration file supplied via the optional valid_yaml argument
• no_valid=True: New WCOSS compliant mode, allowing the fcst-mgr to execute a forecast run for a default/regionalization run type where no validation yaml eixts.

[mxkpp]

This should be reviewed in conjunction with these PRs of other repositories:

https://github.com/NGWPC/nwm-rte/pull/84
NGWPC/nwm-msw-mgr#84
NGWPC/ngen-forcing#170

[zhengtaocui]

It can run hindcast and forecast. How about the default run? Can it run with state saving and warm restart?

[mxkpp]

It can run hindcast and forecast. How about the default run? Can it run with state saving and warm restart?

Please see: https://github.com/NGWPC/nwm-rte/pull/85

[zhengtaocui]

It looks good to me.

[mxkpp]

The SIGTERM handler was simple enough to add to ForecastExecutionManager since the close() method was already fault-tolerant for running multiple times. It's there now. I tested it by having nwm-rte send a SIGTERM to itself after starting a realization.