Forecast validation multiple calls

.gitignore

@@ -1,2 +1,94 @@
+# Compiled source #
+###################
+*.com
+*.class
+*.dll
+*.exe
+*.o
+*.so
+_site/
+
+# Packages #
+############
+# it's better to unpack these files and commit the raw source
+# git has its own built in compression methods
+*.7z
+*.dmg
+*.gz
+*.iso
+*.jar
+*.rar
+*.tar
+*.zip
+
+# Logs and databases #
+######################
+*.log
+*.sql
+*.sqlite
+
+# OS generated files #
+######################
+.DS_Store
+.DS_Store?
+.Spotlight-V100
+.Trashes
+Icon?
+ehthumbs.db
+Thumbs.db
+
+# Vim swap files #
+##################
+*.swp
+
+# Python #
+#################
+*.pyc
+*.egg-info/
+__pycache__/
+*.py[cod]
+.env
+.python-version
+venv
+*.pytest_cache
+build
+
+# pyenv #
+#########
+.python-version
+
+# Django #
+#################
+*.egg-info
+.installed.cfg
+
+# Unit test / coverage reports
+#################
+htmlcov/
+.tox/
+.coverage
+.cache
+nosetests.xml
+coverage.xml
+
+# Front-End #
+#############
+node_modules/
+bower_components/
+.grunt/
+src/vendor/
+dist/
+
+# Temporary Directories #
+*tmp/
+*temp/
+
+# Other
 .idea
 /git_info.json
+
+# Ngen FIles #
+#############
+cat*
+nex*
+troute*

Dockerfile

@@ -1,27 +1,39 @@
+# syntax=docker/dockerfile:1.4
 ARG  NGEN_IMAGE_TAG=latest
 FROM ghcr.io/ngwpc/ngen:${NGEN_IMAGE_TAG}
 
-RUN set -eux; \
-    dnf install -y \
-        jq; \
-    dnf clean all
+# Uncomment when building ngen locally or if ngen-int image is available locally
+# modify to use image tag for local ngen image if needed
+#FROM ngen
 
-COPY requirements.txt .
-RUN set -eux; \
-	\
-    pip3 install -r requirements.txt ; \
-    pip3 cache purge ; \
-    rm --force requirements.txt ;
+# Activate the existing virtual environment
+ENV PATH="/ngen-app/ngen-python/bin:${PATH}"
 
+RUN set -eux; \
+    dnf install -y jq; \
+    dnf clean all
 
 COPY . /ngen-app/ngen-fcst/
 COPY ./docker/run-ngen-fcst.sh /ngen-app/bin/
+
 RUN set -eux; \
-	\
     chmod +x /ngen-app/bin/run-ngen-fcst.sh
 
 WORKDIR /ngen-app/ngen-fcst
 
+# Install missing dependencies that aren't in base image
+RUN --mount=type=cache,target=/root/.cache/pip,id=pip-cache \
+    set -eux; \
+    pip3 install \
+        "matplotlib~=3.10.6"; \
+        #"geopandas~=1.1.1"; \
+    pip3 cache purge
+
+# Install into the existing virtual environment without upgrading base packages
+RUN set -eux; \
+    pip3 install --no-deps . || pip3 install .; \
+    pip3 cache purge;
+
 ARG CI_COMMIT_REF_NAME
 
 RUN set -eux; \

README.md

@@ -1,18 +1,13 @@
-# ngen-fcst
-
-
-
-## Name
-ngen Forecast
+# nwm-fcst-mgr
 
 ## Description
-A program to run ngen given the forecast forcing provided via a .nc file and a configuration file from validation with ngen-cal
+A program to execute ngen cold start and forecast runs provided a configuration file from validation with nwm-cal-mgr and realization files for the cold start and forecast periods
 
 ## Installation
 
-### Clone ngen-fcst
+### Clone nwm-fcst-mgr
 
-git clone -b development --recurse-submodules https://gitlab.sh.nextgenwaterprediction.com/NGWPC/nwm-ngen/ngen-fcst.git
+git clone -b development --recurse-submodules https://github.com/NGWPC/nwm-fcst-mgr.git
 
 ### Build the environment
 
@@ -38,62 +33,48 @@ where [VENV_ROOT] and [NGEN_ROOT] refer to the directory to install the python v
 Follow the following steps to test the program:
 
 1. source [VENV_ROOT]/env.ngen/bin/activate
-2. cd [NGEN-FCST_ROOT]/ngen-fcst
-3. python python/run_ngen_fcst.py test_data/forcing.nc test_data/valid_config.yaml fcst_run1
+2. cd [NWM-FCST-MGR_ROOT]/nwm-fcst-mgr
+3. pip install .
+
+where [NWM-FCST-MGR_ROOT] is where nwm-fcst-mgr is installed.
+
+The program takes three arguments:
+1) Path to the config yaml file for a validation run (from nwm-cal-mgr)
+2) Path to a realization file for the forecast run (generated by nwm-msw-mgr)
+3) Optional Path to a realization file for the cold start run in provided by user (generated by nwm-msw-mgr)
 
-where [NGEN-FCST_ROOT] is where ngen-fcst is installed
+Nwm-fcst-mgr can be run from the CLI or from Python code directly. 
 
-The program takes three command line arguments:
-1) Path to the NetCDF forcing file or folder containing csv forcing files for all catchments in the basin
-2) Path to the config yaml file for a validation run (from ngen-cal)
-3) Path to the folder to be created for storing inputs/outputs from running ngen, relative to the Output directory of the calibration run as indicated in the config yaml file. For example, if "fcst_run1" is the 3rd argument, and "yaml_file" in the "general" section of the config file is '/home/yuqiong.liu/work/Gitlab/run/kge_DDS/noah_cfes/01123000/Output/Validation_Run/01123000_config_valid_best.yaml', then the new output directory to be created for the ngen-fcst run would be:
+### Python
+1. from nwm_fcst_mgr.forecast import run_fcst
+2. valid_yaml = '~/ngwpc/run_ngen/kge_dds/noah_cfes/01123000/Output/Validation_Run/01123000_config_valid_best.yaml'
+3. real_path = '~/ngwpc/run_ngen/kge_dds/noah_cfes/01123000/Output/Forecast_Run/fcst_run1/01123000_realization_config_bmi_fcst.json'
+4. run_fcst(valid_yaml=valid_yaml, real_path=real_path)
+
+If the user wishes to run a cold_start_period, then real_path would be replaced by the path to the cold start realization file.
+
+### CLI
+python -m nwm_fcst_mgr.forecast valid_yaml real_path
+
+where the arguments are replaced by the paths above.
 
-/home/yuqiong.liu/work/Gitlab/run/kge_DDS/noah_cfes/01123000/Output/Forecast_Run/fcst_run1
 
 ## Docker container
 
 ### Requirements
 
-To build and run ngen-fcst, you will need the following software installed and running on your system:
+To build and run nwm-fcst-mgr, you will need the following software installed and running on your system:
 - Docker Engine
 
-You will also need the following data:
-- a forcing file in NetCDF format
-- a YAML-formatted configuration file
-
 ### Build
 
-To build the ngen-fcst container, execute the following command:
+To build the nwm-fcst-mgr container, execute the following command:
 ```
-docker build --tag=ngen-fcst .
+docker build --tag=nwm-fcst-mgr .
 ```
 
 ### Running
 
-To run the ngen-fcst applicaton, execute the following command:
-```
-docker run ngen-fcst
-```
-
-This will print a usage statement for the container:
-```
-Usage: run-ngen-fcst.sh <forcing_file> <config_file> <output_path> [log_file] [venv_path]
-
-FORCING_FILE: Path to the NetCDF forcing file or a folder containing csv forcing files for all catchments in the basin.
-CONFIG_FILE: Path to the config yaml file for a validation run (from ngen-cal).
-OUTPUT_PATH: Path to the folder to be created for storing inputs/outputs from running ngen.
-LOG_FILE (optional): Path to the output file where the script's output will be saved.  Used when running in LOCAL or DOCKER environment
-VENV_PATH (optional): Path to the Python virtual environment.  Used when running in the LOCAL environment.
-
-Examples:
-  run-ngen-fcst.sh test_data/forcing.nc test_data/valid_config.yaml fcst_run1
-  run-ngen-fcst.sh test_data/forcing.nc test_data/valid_config.yaml fcst_run1 /path/to/output /path/to/venv
-```
-
-The path provided for any files should match the path within the container, as well as the paths insider your configuration file. So if `forcing.nc` is located at `~/ngencerf/data/ngen-cal-data/forcing/forcing.nc` and `valid_config.yaml` is located at `~/ngencerf/data/ngen-cal-data/configs/valid_config.yaml`, you should run the command:
-```
-docker run -v ~/ngencerf/data/ngen-cal-data/forcing/:/ngencerf/data/forcing/ -v  ~/ngencerf/data/ngen-cal-data/configs/:/ngencerf/data/configs/ ngen-fcst /ngencerf/data/forcing/forcing.nc /ngencerf/data/configs/valid_config.yaml fcst_run1
-```
 
 ## Contributing
 State if you are open to contributions and what your requirements are for accepting them.