Datastream Roadmap

[jameshalgren]

The datastream is both a software product and the actual stream of data produced by the application of that product to simulate historic or potential future hydrologic states of the CONUS and OCONUS stream networks in our domain of interest.

Simply put the CIROH community datastream is a demonstration of what operational hydrologic forecasting could be. We hope this demonstration makes it easier for life-saving advances to be implemented in ways that they can... save lives!

This issue might be better as is now a discussion... the goal is to dynamically document where the datastream should go so that it can fulfill it's purpose.

[jameshalgren]

Here's a diagram of what the datastream looks like with other data sources. There are implied interfaces in a number of areas in this diagram. Maybe someone with cooler skills can help create that.

graph TD

    %% Main Data Sources at the Top
    subgraph "Main Data Sources & Models"
        direction LR
        USGS["USGS Streamflow Observations"]
        NWM_V3["National Water Model (NWM) v3.0"]
        COMM_DS["Community Model (NGIAB) Datastream"]
    end

    %% Details for NWM
    NWM_V3 --- NWM_API[AWS & Google<br/>BigQuery API]
    NWM_V3 --- NWM_Horizons[Horizons:<br/>SR, MR, LR]

    %% Details for Community Model
    COMM_DS --- COMM_Geo[Geography:<br/>- Local<br/>- VPU/HUC<br/>- Whole Domain<br/>- OCONUS]
    COMM_DS --- COMM_Horizons[Horizons:<br/>SR, MR, LR]

    %% Funneling into the Hydrologist
    USGS --> Hydrologist
    NWM_API --> Hydrologist
    NWM_Horizons --> Hydrologist
    COMM_Geo --> Hydrologist
    COMM_Horizons --> Hydrologist

    Hydrologist((Hydrologist<br/><i>Asks questions the<br/>data can answer</i>))

    %% Applications at the Bottom
    subgraph "Targeted User Applications"
        direction LR
        App1[First Responders]
        App2[Reservoir Operators]
        App3[Emergency Managers]
        App4[Road Safety Specialists]
        App5[Recreation Users]
    end
    Hydrologist --> App1
    Hydrologist --> App2
    Hydrologist --> App3
    Hydrologist --> App4
    Hydrologist --> App5


    %% Feedback Loop
    subgraph "Model Improvement Feedback Loop"
        direction TB
        Hydrologist_Feedback(Hydrologist<br/><i>Makes model improvements</i>)
        Approval[Approval Chain]
        PullRequest["Submits Pull Request"]
        
        Hydrologist_Feedback --> Approval
        Approval --> PullRequest
        PullRequest --> COMM_DS
    end

    %% Styling
    style Hydrologist fill:#f9f,stroke:#333,stroke-width:2px,color:#000;
    style Hydrologist_Feedback fill:#f9f,stroke:#333,stroke-width:2px,color:#000;
    linkStyle 0 stroke-width:2px,stroke:#1e90ff,fill:none;
    linkStyle 1 stroke-width:2px,stroke:#1e90ff,fill:none;
    linkStyle 2 stroke-width:2px,stroke:#1e90ff,fill:none;
    linkStyle 3 stroke-width:2px,stroke:#1e90ff,fill:none;
    linkStyle 4 stroke-width:2px,stroke:#1e90ff,fill:none;
    linkStyle 5 stroke-width:2px,stroke:#008000,fill:none;
    linkStyle 6 stroke-width:2px,stroke:#008000,fill:none;
    linkStyle 7 stroke-width:2px,stroke:#008000,fill:none;
    linkStyle 8 stroke-width:2px,stroke:#008000,fill:none;
    linkStyle 9 stroke-width:2px,stroke:#008000,fill:none;

Loading

[jameshalgren]

graph TD
    subgraph "Data Stream Inputs & Pre-processing"
        MET_FORC["Meteorological Forcings<br>(e.g., AORC, GFS, HRRR, CFS, ERA 5)"] --> PRE_PROC
        HYDRO_FABRIC["Hydro Fabric (Geopackage)"] --> PRE_PROC
        GAUGE_DATA["Gauge Observations<br>(USGS, External Sources)"] --> HIST_REANALYSIS
        REALIZ_FILE["Realization Files<br>(JSON Configuration, Model Steps, Parameters)"] --> SIM_CONFIG
        PRE_PROC["Pre-processing Tools & Data Elements"] --> SIM_CONFIG
    end

    subgraph "Data Stream Core Models & Framework"
        HIST_REANALYSIS["Historical Reanalysis Data Set<br>(Calibration Tool, ML Methods: Transformers, LSTMs)"] --> MODELS
        MODELS["Hydrologic & ML Models<br>(NOM, CF, T-Route, LSTM, Diffusion Models, Community Models)"] --> NGAB
        NGAB["NextGen in a Box (NGAB)<br>(Containerized NGEN Framework)"] --> SIM_EXECUTION
        SIM_CONFIG["Simulation Configurations<br>(Forcings, Models, Storage)"] --> SIM_EXECUTION
        SIM_EXECUTION["Automated Simulation Execution<br>(Scheduled Forecast & Retrospective Runs)"] --> DATA_STORAGE
    end

    subgraph "Data Stream Outputs & Archival"
        DATA_STORAGE["Data Storage & Serving<br>(Archiving Inputs, Hydro Fabric, Forcings, Simulation Outputs)"] --> STREAM_OUT
        DATA_STORAGE --> HYDRO_OUT
        DATA_STORAGE --> RETRO_SIM_OUTPUTS
        DATA_STORAGE --> FORECAST_OUTPUTS

        STREAM_OUT[Streamflow Outputs]
        HYDRO_OUT["Other Hydrology Outputs<br>(e.g., Soil Moisture, Snow Water Equivalent)"]
        RETRO_SIM_OUTPUTS["Retrospective Simulation Outputs<br>(40+ Year Historical Behavior)"]
        FORECAST_OUTPUTS["Forecasts<br>(Short, Medium, Long Range)"]
    end

    subgraph "Data Stream Interfaces & User Engagement"
        DS_UI["Data Stream User Interface<br>(e.g., datastream.ciroh.org, Water.NOAA.gov-like)"]
        EVAL_INT["Evaluation Interface<br>(Tools for Model Comparison & Calibration)"]

        STREAM_OUT --> DS_UI
        HYDRO_OUT --> DS_UI
        RETRO_SIM_OUTPUTS --> DS_UI
        FORECAST_OUTPUTS --> DS_UI

        DS_UI --> USERS
        EVAL_INT --> USERS

        USERS["User Categories<br>(General Public, Hydrologic Scientist, CIROH Partner, NWS RFC User, Internal Scientist, Internal Developer)"] --> TARGET_APPS
        TARGET_APPS["Targeted User Applications<br>(First Responders, Reservoir Operators, Emergency Managers, Road Safety, Recreation)"]
    end

    subgraph "Model Improvement & Feedback Loop"
        HYD_FB("Hydrologist: Makes Model Improvements") --> APPROVAL
        APPROVAL["Approval Chain"] --> PULL_REQ
        PULL_REQ["Submits Pull Request<br>(CI/CD Trigger for New Models/Configurations)"] --> MODELS
    end

    subgraph "Relationship to National Water Model (NWM)"
        NWM_OPERATIONAL["National Water Model (NWM) v3.0<br>(Operational Forecasting, Comparison Baseline)"]
        NWM_OPERATIONAL --> DS_UI
        NWM_OPERATIONAL --> EVAL_INT
    end

Loading

[JordanLaserGit]

Dropping links relevant here

Community contributions approval workflow #209
The NRDS testing and evaluation with new software updates #212, #214
Automated deployment of the NRDS in AWS #230
Baseline model selection in the NRDS #232