NGWPC NWM Regionalization, Hindcast, and Coastal Tools 2026-06-09

[cmaynard-ngwpc]

Implemented major forecasting workflow enhancements, including support for hindcasting, lagged ensemble forecasting, state saving and warm-start execution, intermediate analysis runs, separate cold-start jobs, and MPI-based parallel forecast processing. Refactored the forecast manager architecture to improve workflow orchestration, direct integration with msw-mgr, configuration caching, forecast execution management, and handling of multiple forecast runs. Added robust asynchronous forecast execution through a new ForecastExecutionManager, improving process lifecycle management, interrupt handling, status tracking, logging, and subprocess control. Enhanced operational reliability through improved state loading, forecast partitioning support, validation of hindcast intervals, restructuring of run directories, and better handling of warm-start and lagged-ensemble workflows. Integrated EWTS logging, standardized log naming and storage, improved documentation and READMEs, and updated CI/CD, Docker, and customer delivery workflows. Numerous bug fixes, logging improvements, and code quality updates were also included to improve stability and maintainability across forecasting operations.

Additions

• Added NHF, hydrofabric, geopackage, VPU, and crosswalk workflow capabilities, including utility scripts, oCONUS handling, and domain/gage data integration.
• Integrated EWTS logging support across affected components, including module-specific logging updates, optional logging behavior, Docker/build integration, and unit-test coverage where applicable.
• Added and expanded forcing workflow support, including updated configuration templates, forecast product handling, lagged-ensemble inputs, regional forcing behavior, and package-delivered forcing resources.
• Added state saving, state restoration, reset-time, restart, and hindcast workflow capabilities, including related time-management updates and model restart handling.
• Expanded verification and evaluation capabilities with additional metric calculation behavior, observation retrieval support, plotting updates, gage-source integration, and forecast-to-observation alignment.
• Expanded automated testing coverage, expected-result data, unit-test support, tolerance-based comparisons, serialization checks, and CI-exercised validation paths.
• Added additional release updates covering additions for optional arg partition_file to run_forecast, additions for support for MPI parallel processing for forecasts, additions for back forecast subparser, and Support alternate ngen log path by persisting OS env var NGEN_RESULTS_DIR if already set.

Removals

• Removed unused, duplicate, obsolete, and deprecated code paths, including stale helper functions, duplicated definitions, older EWTS references, and no-longer-needed implementation artifacts.

Changes

• Updated CI/CD, security scanning, downstream pipeline triggers, and customer-delivery workflow configuration.
• Improved validation, warning, exception, traceback, permission, and user-facing error-reporting behavior.
• Updated Docker, build, packaging, dependency, and installed-resource configuration.
• Refined configuration, realization, template, sample-config, and package-resource handling across supported workflows and deployment environments.
• Updated hydrologic model outputs and model-specific runtime behavior, including formulation outputs, GIUH/channel-inflow values, ponded-depth variables, and model-specific state/runtime handling.
• Refactored and cleaned up implementation details, including naming consistency, type hints, imports, property handling, and internal helper structure.
• Updated cache, dataset, temporary-file, scratch-directory, and data-access behavior, including lazy loading, explicit dataset closure, retry handling, and local/S3 access refinements.
• Corrected unit conversions and size/type reporting across model calculations, serialized buffers, time-step handling, and BMI-facing data exchanges.
• Updated additional release updates covering fixes for state loading for warm state runs, Minor script update, Perfix ngen stdout stderr log name with the job id, updates to script, and Revert changes to cold_start_state argument.