Add async ngen execution via new context manager class ForecastExecutionManager

[mxkpp]

This PR adds a new context manager class ForecastExecutionManager with primary methods preprocess, execute, and postprocess, which allows asynchronous subprocess calls to the ngen binary, external polling of the ngen subprocess, and external stopping of the ngen subprocess.

On exit, the context manager closes the log file and terminates ngen if it is still running. A wait parameter is provided on the execute method for cases where external polling is not desired, in which case the ForecastExecutionManager performs its own polling.

Function run_fcst still exists, and it uses the new class by calling execute(wait=True) such that its user interface is effectively unchanged.

To stop ngen execution, the user calls method schedule_ngen_stoppage().

See here for example of async user call (wait=False) where the user intentionally quits execution after certain messages are found in the ngen log file: https://github.com/NGWPC/nwm-rte/blob/64f79860170c04983916be898ebddd5cac02247d/bin_mounted/example_workflow.py#L245-L250

[mxkpp]

Note that the new class changes the type of exception raised during ngen execution. Previously it would raise a generic CalledProcessError. Now it will raise a NgenCalledProcessError, or a NgenIntentionallyStoppedError situationally.

[jswade-rtx]

Changes look good.

[mxkpp]

@peterkronenberg-rtx, these changes should not cause a change in default/existing behavior: when a forecast is executed via call to run_fcst().

If CERF leverages that function, could you please confirm that behavior is unchanged from CERF usage perspective?

[peterkronenberg-rtx]

Successfully ran a forecast from ngenCerf. No server changes were needed. Everything worked fine

[mxkpp]

The 2 commits I added earlier today were to support a testing environment, which needed:

1. To be interruptible via ctrl+C (KeyboardInterrupt)
2. A new log file per call, since it scans the log file for errors.