docs / cleanup

README.md

@@ -3,43 +3,56 @@ Building Hydrofabric & Processing Ancillary Data
 
 <img style="display: block; margin-left: auto; margin-right: auto;" src="docs/img/hydrofabric.png" alt="hydrofabric" width="40%" height="40%"/>
 
-
-#### Proposed Schema
+# About the Data
+## Schema
 
 The following schema is the proposed data model for NGWPC hydrofabric datasets produced by this repo.
 
+TODO: Update
+
 <img style="display: block; margin-left: auto; margin-right: auto;" src="docs/img/nhf_v1.1.2_schema.png" alt="nhf_v1.1.2_schema.png" width="100%" height="100%"/>
 
-##### Flowpaths FACT Table
+## Flowpaths FACT Table
 
 The central table (or FACT Table) is `Flowpaths`. Each `flowpath` has a downstream, and upstream `nexus` point, allowing for traversal of a river network through a single table. Additionally, there is a 1:1 relationship between `flowpath` and `divide`.
 
-##### NGEN Tables
+## NGEN Tables
 
 The tables highlighted in green are the infomation needed for lumped modeling to take place. Lumped models require attributes, the shape of the `divide` that is being modeled, and a `nexus` point for flow to be aggregated to.
 
-##### Routing Tables
+## Routing Tables
 
 The tables highlighted in blue contain the information needed for routing at a high resolution. T-Route is expected to run at a fine-scale (~300m segments) with many `virtual_flowpaths`. Each virtual flowpath is delineated based on the reference fabric, and there should be a many -> one relationship between `virtual_flowpaths` and `flowpaths`, with some `virtual flowpaths` not being represented in the `flowpaths` table. These non-represented `flowpaths` have the parameter of `routing_segment` set to False, and will have flow estimated through flow-scaling.
 
-##### Reference Crosswalks
+The `reservoir_da` table encodes crosswalks between lakes and gages with an assigned data assimilation code. The `lakes_polygons` layer mirrors the traditional `lakes` point layer, but includes the polygon representation. This polygon representation is used to derive the flowpaths associated with lakes for routing. The `lake_vfp_crosswalk` table contains the intersection of lake polygons and virtual flowpaths so that T-route treats all lake flowpaths as lakes rather than channels.
+
+## Reference Crosswalks
 
-The NGWPC Hydrofabric is build using many reference materials:
+The NGWPC Hydrofabric is built using many reference materials:
 - Reference Flowpaths
 - Reference Reservoirs
-- USGS/ENVCA/CADWR/TXDOT Streamflow Gages
+- Reference Waterbodies
+- NWM v3 Lakes
+- National Inventory of Dams
+- USGS/ENVCA/CADWR/TXDOT/RFC/USBR/USACE Streamflow Gages
 - NHD+
 
 To ensure `flowpaths` can be mapped to back to the materials that created them, each of the reference materials is mapped to `flowpaths`, `hydrolocations`, and `virtual flowpaths`. The following IDs pairings are used:
 
 - Reference Flowpaths -> `ref_fp_id`
 - Reference Reservoirs -> `dam_id`
-- USGS/ENVCA/CADWR/TXDOT Streamflow Gages -> `site_no`
+- Reference Reservoirs -> `ref_fab_wb` is `lake_id` / NHD `COMID`
+- Streamflow Gages -> `site_no`
 - NHD+ -> `nhd_feature_id`
 
-##### Visual Diagram
+## Validation
+The `validate_hf` task in the pipeline produces a JSON report called `nhf_{version}_validation.json`. This report details various metrics from the built product, such as: number of null divide attributes, number of attributes out of defined minimum and maxium range, and assertions that necessary lakes and gages are present and assigned to flowpaths.
+
+
+## Visual Diagram
 <img style="display: block; margin-left: auto; margin-right: auto;" src="docs/img/nhf_diagram.png" alt="NHF Diagram" width="100%" height="100%"/>
 
+
 # Development Commands
 
 Run these commands from the repository root.
@@ -53,56 +66,8 @@ The following command installs the project's base dependencies, the `docs` optio
 uv sync --all-extras --all-groups
 ```
 
-Python 3.12 or newer is required.
-
-## Sync input data using the `justfile`
-
-`just` calls series of commands called "recipes" similar to a `make` file. Install on linux with `apt get just` or follow linked readme for other platforms. After installing `just`, you can use the following commands to set up the data sources for `nhf-builds`. You can also use `just` to build hydrofabrics for each domain or specify a config.
-
-Provide AWS credentials in the current shell:
-
-```bash
-export AWS_DEFAULT_REGION="us-east-1"
-export AWS_ACCESS_KEY_ID="..."
-export AWS_SECRET_ACCESS_KEY="..."
-export AWS_SESSION_TOKEN="..."  # Required for temporary credentials
-```
-
-Verify that AWS recognizes the credentials:
-
-```bash
-aws sts get-caller-identity
-```
-
-Then sync the input data for the desired domain:
-
-```bash
-just sync       # CONUS
-just sync-ak    # Alaska
-just sync-hi    # Hawaii
-just sync-prvi  # Puerto Rico and the US Virgin Islands
-```
-
-To select a different OCONUS reference-fabric version, pass the `oconus-version` variable:
-
-```bash
-just oconus-version=0.1.8 sync-ak
-```
-
-> **Warning:** The sync recipes overwrite the corresponding input datasets under `data/`.
-
-### AWS credential handling
-
-Exporting AWS credentials in the shell is functionally sufficient because `just` and its child processes inherit those environment variables. The `justfile` also automatically loads variables from a repository-root `.env` file.
-
-Avoid committing credentials or entering long-lived secrets directly into commands that may be saved in shell history. When available, prefer an AWS SSO or named-profile workflow for data synchronization:
-
-```bash
-aws sso login --profile ngwpc-test
-AWS_PROFILE=ngwpc-test just sync
-```
-
-An AWS profile is sufficient for the `aws s3` commands used by the sync recipes. Some hydrofabric build paths access S3 credentials directly through `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`, so a profile alone may not be sufficient for every build configuration.
+## Unpack Data
+Extract the `hydrofabric_builds_data.tar` archive to the `data` folder. This archive includes all data for running the canonical NHF for all domains.
 
 ## Run the hydrofabric build
 
@@ -112,7 +77,9 @@ Run the main build script directly with the CONUS example configuration:
 uv run python scripts/hf_runner.py --config configs/example_config.yaml
 ```
 
-Alternatively, use a domain-specific `just` recipe:
+Alternatively, use a domain-specific `just` recipe.
+
+`just` calls series of commands called "recipes" similar to a `make` file. Install on linux with `apt get just` or follow linked readme for other platforms. After installing `just`, you can use the following commands to build the hydrofabric.
 
 ```bash
 just build-conus
@@ -127,6 +94,21 @@ Run the build with a custom configuration:
 just build "configs/my_custom_config.yaml"
 ```
 
+## Documentation
+This repository has documentation that can be served via [mkdocs](https://www.mkdocs.org/).
+Ensure that dependencies are installed:
+
+```bash
+uv sync --extra docs
+```
+To serve docs locally, run:
+
+```bash
+mkdocs serve -a localhost:8080
+```
+Navigate to `localhost:8080/` in your browser.
+
+
 ## Run tests
 
 ### Run the complete test suite

docs/builds/POI_gages_builder.md

@@ -1,4 +1,5 @@
 ## POI/Gages Builder — Integration Guide
+TODO: Update
 
 This document explains how to assemble a single, canonical gages layer by merging USGS (active + discontinued) and partner gage sources (TXDOT, CADWR, ENVCA, NWM calibration sets, AK/HI/PR supplements). You’ll download the source files from the S3 bucket (see below), place them in your local user directory.
 ### Download source files:

docs/builds/divide_attributes.md

@@ -1,5 +1,7 @@
 # Divide Attributes Build
 
+TODO: Update
+
 The divide attributes task calculates zonal statistics for divides from a number of rasters.
 
 To run:

docs/builds/flowpath_attributes.md

@@ -1,5 +1,7 @@
 # Flowpath Attributes Build
 
+TODO: Update
+
 The flowpath attributes task calculates metrics per flowpath linestring from multiple sources: DEM, WRF defaults, and RiverML outputs.
 
 ## To run:

docs/builds/lakes.md

@@ -1,5 +1,7 @@
 # Lakes
 
+TODO: Update
+
 The NHF lakes layer integrates lakes and reservoir data from multiple sources. Hydraulic parameters for t-route are calculated.
 
 ## Data sources

docs/builds/quickstart.md

@@ -1,5 +1,7 @@
 ### Quickstart
 
+TODO: Replace s3 paths with unpacking file
+
 ## CONUS
 Below are the data files needed for running a full build of the NHF dataset and their locations. Each of the `aws` commands should be run from the NGWPC Test account
 

docs/builds/reservoir_attrs.md

@@ -1,5 +1,6 @@
 ## Reservoirs Description:
 
+TODO: Update
 
 Below is a summary of what the Reservoir Python script is doing and the dependencies it assumes. Potential future action items for each data source has been descussed here for record.
 This issue is mainly about capturing and solidifying the current workflow so we can (a) understand the data dependencies, and (b) reproduce the same processing in our Python-based toolchain.

docs/development/development.md

@@ -13,54 +13,8 @@ uv sync --all-extras --all-groups
 
 Python 3.12 or newer is required.
 
-## Sync input data using the `justfile`
-
-`just` calls series of commands called "recipes" similar to a `make` file. Install on linux with `apt get just` or follow linked readme for other platforms. After installing `just`, you can use the following commands to set up the data sources for `nhf-builds`. You can also use `just` to build hydrofabrics for each domain or specify a config.
-
-Provide AWS credentials in the current shell:
-
-```bash
-export AWS_DEFAULT_REGION="us-east-1"
-export AWS_ACCESS_KEY_ID="..."
-export AWS_SECRET_ACCESS_KEY="..."
-export AWS_SESSION_TOKEN="..."  # Required for temporary credentials
-```
-
-Verify that AWS recognizes the credentials:
-
-```bash
-aws sts get-caller-identity
-```
-
-Then sync the input data for the desired domain:
-
-```bash
-just sync       # CONUS
-just sync-ak    # Alaska
-just sync-hi    # Hawaii
-just sync-prvi  # Puerto Rico and the US Virgin Islands
-```
-
-To select a different OCONUS reference-fabric version, pass the `oconus-version` variable:
-
-```bash
-just oconus-version=0.1.8 sync-ak
-```
-
-> **Warning:** The sync recipes overwrite the corresponding input datasets under `data/`.
-
-### AWS credential handling
-
-Exporting AWS credentials in the shell is functionally sufficient because `just` and its child processes inherit those environment variables. The `justfile` also automatically loads variables from a repository-root `.env` file.
-
-Avoid committing credentials or entering long-lived secrets directly into commands that may be saved in shell history. When available, prefer an AWS SSO or named-profile workflow for data synchronization:
-
-```bash
-aws sso login --profile ngwpc-test
-AWS_PROFILE=ngwpc-test just sync
-```
-
-An AWS profile is sufficient for the `aws s3` commands used by the sync recipes. Some hydrofabric build paths access S3 credentials directly through `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`, so a profile alone may not be sufficient for every build configuration.
+## Unpack Data
+Extract the `hydrofabric_builds_data.tar` archive to the `data` folder. This archive includes all data for running the canonical NHF for all domains.
 
 ## Run the hydrofabric build
 
@@ -70,7 +24,9 @@ Run the main build script directly with the CONUS example configuration:
 uv run python scripts/hf_runner.py --config configs/example_config.yaml
 ```
 
-Alternatively, use a domain-specific `just` recipe:
+Alternatively, use a domain-specific `just` recipe.
+
+`just` calls series of commands called "recipes" similar to a `make` file. Install on linux with `apt get just` or follow linked readme for other platforms. After installing `just`, you can use the following commands to build the hydrofabric.
 
 ```bash
 just build-conus

docs/schemas/constants/groundwater_projection_ak.json

@@ -0,0 +1,11 @@
+{
+  "class_name": "GroundWaterProjectionAK",
+  "values": {
+    "PROJ4": "+proj=stere +lat_0=90 +lat_ts=60 +lon_0=-135 +x_0=0 +y_0=0 +R=6370000 +units=m +no_defs",
+    "X_ORIGIN": -1130764.7202253528,
+    "Y_ORIGIN": -3163389.53353531,
+    "WIDTH": 3516,
+    "HEIGHT": 1816,
+    "DX": 250
+  }
+}

docs/schemas/constants/groundwater_projection_conus.json

@@ -0,0 +1,11 @@
+{
+  "class_name": "GroundWaterProjectionCONUS",
+  "values": {
+    "PROJ4": "+proj=lcc +lat_1=30 +lat_2=60 +lat_0=40.0000076293945 +lon_0=-97 +x_0=0 +y_0=0 +a=6370000 +b=6370000 +units=m +no_defs",
+    "X_ORIGIN": -2303874.17655,
+    "Y_ORIGIN": -1919874.66329,
+    "WIDTH": 18432,
+    "HEIGHT": 15360,
+    "DX": 250
+  }
+}

docs/schemas/constants/groundwater_projection_hi.json

@@ -0,0 +1,11 @@
+{
+  "class_name": "GroundWaterProjectionHI",
+  "values": {
+    "PROJ4": "+proj=lcc +units=m +a=6370000.0 +b=6370000.0 +lat_1=10.0 +lat_2=30.0 +lat_0=20.6 +lon_0=-157.42 +x_0=0 +y_0=0 +k_0=1.0 +nadgrids=@null +wktext +no_defs",
+    "X_ORIGIN": -294950.07097397465,
+    "Y_ORIGIN": -194949.36969098,
+    "WIDTH": 5900,
+    "HEIGHT": 3900,
+    "DX": 100
+  }
+}

docs/schemas/constants/groundwater_projection_prvi.json

@@ -0,0 +1,11 @@
+{
+  "class_name": "GroundWaterProjectionPRVI",
+  "values": {
+    "PROJ4": "+proj=lcc +units=m +a=6370000.0 +b=6370000.0 +lat_1=18.1 +lat_2=18.1 +lat_0=18.1 +lon_0=-65.91 +x_0=0 +y_0=0 +k_0=1.0 +nadgrids=@null +wktext  +no_defs",
+    "X_ORIGIN": -149949.83,
+    "Y_ORIGIN": -54948.968,
+    "WIDTH": 3000,
+    "HEIGHT": 1100,
+    "DX": 100
+  }
+}