Add data publish/sync tooling (DVC/R2) + docs page (run_data_sync)

Author: mbaronnet

.gitignore

@@ -5,8 +5,13 @@
 **/~*
 *.dat
 *.sh
+!tools/sync.sh
 *~
 
+# Data publishing (DVC) — see docs/run/run_data_sync.md
+tools/.env
+epm/output_view/
+
 # GAMS artifacts
 *.bk
 *.g00

Publish.bat

@@ -0,0 +1,5 @@
+@echo off
+REM Double-clic : publie tes donnees (code+pointeurs sur GitHub, donnees sur le store, EPM View a jour)
+powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0tools\publish.ps1"
+echo.
+pause

Sync.bat

@@ -0,0 +1,5 @@
+@echo off
+REM Double-clic : recupere code + donnees a jour (git pull + dvc pull)
+powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0tools\sync.ps1"
+echo.
+pause

docs/run/run_data_sync.md

@@ -0,0 +1,135 @@
+# Publishing & Syncing Data (DVC)
+
+EPM **code** is public, but the **data** (input CSVs, results) is kept in a **private
+store** — not in this public repository. The repo only keeps tiny **pointers**; the real
+data lives in the store and is fetched on demand.
+
+This page explains, from scratch, how to **publish** your data and how to **get it back**
+on another machine or on the server. No prior knowledge of DVC is assumed.
+
+!!! note "Prototype"
+    The store is currently a Cloudflare R2 bucket (test data). For confidential data,
+    the production store will be the World Bank S3 — same workflow, different endpoint.
+
+---
+
+## The idea in one picture
+
+```
+YOU (laptop) — you edit inputs/outputs in EPM
+        │
+        │  one action (double-click Publish):
+        │   ├─ git push  → GitHub : code + .dvc pointers   (no data)
+        │   ├─ dvc push  → STORE  : the data               (for runs)
+        │   └─ publish   → STORE  : a readable copy         (for EPM View)
+        ▼
+   ┌──────────────── PRIVATE STORE (R2 / S3) ────────────────┐
+   └───────────────┬─────────────────────────┬──────────────┘
+                   │ dvc pull                 │ gated read
+            SERVER (run the model)        EPM View (display)
+```
+
+**What is DVC?** Think *"git for data"*. Git stays light because the data is replaced by a
+6-line pointer file (`*.dvc` = a hash + a path, **no values**). The real data lives in the
+store, moved with `dvc push` / `dvc pull`.
+
+The store keeps **two layouts** of the same data: a hash-addressed copy for DVC (runs), and
+a readable copy by path for EPM View (the browser can't speak DVC). Both are written by the
+same "publish" action — you don't manage this by hand.
+
+---
+
+## One-time setup (per machine)
+
+1. Install the dependencies (DVC is included):
+   ```bash
+   pip install -r requirements.txt
+   ```
+2. Add your store keys: copy `tools/.env.example` to `tools/.env` and paste the 4 values
+   (ask the store admin).
+   ```
+   tools/.env        ← your keys   (git-ignored — NEVER commit real keys)
+   tools/.env.example ← template
+   ```
+
+That's it. The remote URL/endpoint is already in the repo (`.dvc/config`).
+
+---
+
+## Publish your data
+
+After you change inputs (or want to show results), **publish in one action**:
+
+=== "Windows"
+    Double-click **`Publish.bat`** at the repo root.
+
+=== "Command line"
+    ```bash
+    powershell -File tools/publish.ps1   # Windows
+    ```
+
+This does everything: re-hash the data (`dvc add`) → commit + push the pointer →
+`dvc push` (data to the store, for the server) → upload readable copies (inputs +
+`epm/output_view/`) for EPM View.
+
+!!! tip "Showing results in EPM View"
+    Copy the runs you want to display into
+    `epm/output_view/<run>/<scenario>/output_csv/` **before** publishing (only `.csv`
+    files are sent). `epm/output_view/` is a local staging folder (git-ignored).
+
+---
+
+## Get the data (another machine / the server)
+
+=== "Server (Linux)"
+    ```bash
+    bash tools/sync.sh
+    ```
+=== "Windows"
+    Double-click **`Sync.bat`**.
+
+This runs `git pull` (code + pointers) then `dvc pull` (data from the store) — then you can
+run the model with up-to-date data.
+
+---
+
+## Onboarding a new model (one-time per model)
+
+Moving a model's data out of git into the store (the "switch"):
+
+```bash
+dvc init                                          # once per repo
+dvc remote add -d store s3://<bucket>/dvcstore
+dvc remote modify store endpointurl <endpoint>
+# in .gitignore: remove the whitelist of the data folder, add  !epm/input/<data>.dvc
+git rm -r --cached epm/input/<data_folder>        # untrack (files stay on disk)
+dvc add epm/input/<data_folder>                   # create the pointer
+```
+Then publish (`Publish.bat`). After this, everyone just does *setup + publish/sync*.
+
+To make **EPM View** read this branch from the store, add the branch name to `R2_BRANCHES`
+in `src/utils/epmFetch.js` of the `epm-data-explorer` repo.
+
+---
+
+## Troubleshooting
+
+- **`Unable to locate credentials`** → keys not loaded. On the server, either
+  `export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=…`, or set them once with
+  `dvc remote modify --local store access_key_id …` (stays in git-ignored `config.local`).
+- **`CERTIFICATE_VERIFY_FAILED` / SSL** → a corporate TLS proxy is intercepting the
+  connection. Try off-VPN, or `pip install pip-system-certs` (makes Python trust the OS
+  certificate store). The AWS server is not affected.
+- **`dvc` command not found** (server) → use `python -m dvc …`, or add `~/.local/bin` to
+  your `PATH`.
+
+---
+
+## Files (in `tools/`)
+
+| File | Role |
+|------|------|
+| `publish.ps1` | engine behind `Publish.bat` |
+| `sync.ps1` / `sync.sh` | get code + data (Windows / server) |
+| `upload_to_r2.py`, `upload_output_view_to_r2.py` | upload the readable copies |
+| `.env.example` | keys template (copy to `.env`, git-ignored) |

mkdocs.yml

@@ -48,6 +48,7 @@ nav:
     - Run from Python: run/run_python.md
     - Run from GAMS Studio: run/run_gams_studio.md
     - Run on Remote Server: run/run_remote_server.md
+    - Publish & Sync Data (DVC): run/run_data_sync.md
     - Dispatch Mode: run/run_dispatch.md
     - Solver Options: run/options_solver.md
     - Performance & Threads: run/options_thread.md

requirements.txt

@@ -20,5 +20,8 @@ rasterio
 entsoe-py
 pycountry
 mcp
+# Data publishing/sync to private store (R2/S3) — see docs/run/run_data_sync.md
+dvc[s3]
+s3fs
 # Optional (Monte Carlo analysis only):
 # chaospy==4.3.18

tools/.env.example

@@ -0,0 +1,9 @@
+# Cles d'acces au data store (R2 maintenant ; S3 WB plus tard).
+# 1. Copie ce fichier en  tools/.env
+# 2. Colle tes cles (fournies par l'admin du store)
+# /!\ tools/.env est gitignore : NE JAMAIS committer de vraies cles.
+
+AWS_ACCESS_KEY_ID=
+AWS_SECRET_ACCESS_KEY=
+STORE_ENDPOINT=https://25d26c21c96631ccd04cb2178ed3fb6b.r2.cloudflarestorage.com
+STORE_BUCKET=epm-view-data

tools/DATA_PUBLISH.md

@@ -0,0 +1,81 @@
+# Publier / récupérer les données du modèle (store privé)
+
+> 📖 **Guide complet (doc EPM) :** `docs/run/run_data_sync.md` — *Installation & Run →
+> Publish & Sync Data (DVC)*. Ce fichier-ci est un aide-mémoire rapide à côté des scripts.
+
+Les **données** d'EPM (input/output) ne vivent plus dans ce repo public : elles sont
+dans un **store privé** (R2 aujourd'hui, S3 WB plus tard). Le repo ne garde que de
+petits **pointeurs `.dvc`**. Ce dossier fournit l'outillage pour publier et récupérer.
+
+> **Prototype.** Le store actuel est un bucket Cloudflare R2 (données de test).
+> Données confidentielles → attendre le store S3 de la Banque.
+
+---
+
+## 🔧 Mise en place — une fois par machine
+
+1. Installer les dépendances (inclut DVC) :
+   ```
+   pip install -r requirements.txt
+   ```
+2. Mettre tes clés d'accès au store :
+   - copie `tools/.env.example` → `tools/.env`
+   - colle les 4 valeurs (fournies par l'admin du store)
+   - ⚠️ `tools/.env` est **gitignoré** : ne jamais committer de vraies clés.
+
+C'est tout. La config du remote (URL/endpoint) est déjà dans le repo (`.dvc/config`).
+
+---
+
+## ⬆️ Publier (après avoir modifié des données) — Windows
+
+**Double-clic sur `Publish.bat`** (à la racine du repo).
+
+Ça fait tout : re-hash des données (`dvc add`) → commit + push des pointeurs →
+`dvc push` (données → store, pour le serveur) → upload des copies lisibles
+(inputs + `epm/output_view/`) → store, pour **EPM View**.
+
+> Pour montrer des **résultats** dans EPM View : copie les runs voulus dans
+> `epm/output_view/<run>/<scenario>/output_csv/` avant de publier (seuls les `.csv`
+> sont envoyés). `epm/output_view/` est gitignoré (zone de staging locale).
+
+---
+
+## ⬇️ Récupérer code + données
+
+- **Windows** : double-clic sur `Sync.bat`
+- **Serveur Linux** : `bash tools/sync.sh`
+
+= `git pull` (code + pointeurs) + `dvc pull` (données depuis le store).
+
+---
+
+## Onboarder un NOUVEAU modèle (une fois par modèle)
+
+Sortir les données d'un modèle de git vers le store (la « bascule ») :
+```
+dvc init                                   # une fois par repo
+dvc remote add -d store s3://<bucket>/dvcstore
+dvc remote modify store endpointurl <endpoint>
+# retirer la whitelist du dossier dans .gitignore, garder !epm/input/<data>.dvc
+git rm -r --cached epm/input/<data_folder>
+dvc add epm/input/<data_folder>
+```
+Puis publier (`Publish.bat`). Après ça, tout le monde fait juste *mise en place + publish/sync*.
+
+---
+
+## Côté EPM View (l'app)
+
+EPM View lit les données par branche. Les branches dont les données sont dans le
+store privé sont listées dans `R2_BRANCHES` (fichier `src/utils/epmFetch.js` du repo
+`epm-data-explorer`). Pour brancher une nouvelle région : ajouter sa branche là.
+
+---
+
+## Fichiers de ce dossier (publication données)
+
+- `publish.ps1` — moteur de `Publish.bat`
+- `sync.ps1` / `sync.sh` — récupération (Windows / serveur)
+- `upload_to_r2.py`, `upload_output_view_to_r2.py` — helpers d'upload lisible
+- `.env.example` — modèle de clés (copier en `.env`, gitignoré)

tools/README.md

@@ -6,6 +6,7 @@ This directory contains utility scripts and tools for working with EPM.
 
 - **`mcp_server.py`** - MCP (Model Context Protocol) server that allows Cursor AI to run EPM directly
 - **`test_epm_simple.py`** - Simple test script to run EPM with minimal configuration
+- **Data publishing/sync** (`publish.ps1`, `sync.ps1`/`sync.sh`, `upload_*.py`) - publish model data to / pull it from the private store (R2/S3). See **[DATA_PUBLISH.md](DATA_PUBLISH.md)** / **docs/run/run_data_sync.md**, and the `Publish.bat` / `Sync.bat` launchers at the repo root.
 
 ## MCP Server
 

tools/publish.ps1

@@ -0,0 +1,64 @@
+# publish.ps1 — publie TOUT en un geste (lance par Publish.bat) :
+#   re-hash (dvc add) -> commit+push des pointeurs -> dvc push (donnees, pour le serveur)
+#   -> copies lisibles inputs + output_view (pour EPM View)
+# Auto-detecte le repo et la branche. Lit les cles dans tools/.env (gitignore).
+
+$ErrorActionPreference = "Stop"
+$TOOLS = $PSScriptRoot
+$REPO  = Split-Path $TOOLS -Parent
+
+# --- 1. charger les cles depuis tools/.env ---
+$envFile = Join-Path $TOOLS ".env"
+if (-not (Test-Path $envFile)) {
+    Write-Error "Manque $envFile  ->  copie tools/.env.example en tools/.env et mets tes cles."
+    exit 1
+}
+Get-Content $envFile | Where-Object { $_ -match '^\s*[^#].*=' } | ForEach-Object {
+    $k, $v = $_ -split '=', 2
+    Set-Item -Path "env:$($k.Trim())" -Value $v.Trim()
+}
+
+# --- 2. auto-detect repo + branche ---
+Set-Location $REPO
+$env:EPM_REPO   = $REPO
+$env:EPM_BRANCH = (git rev-parse --abbrev-ref HEAD).Trim()
+Write-Host ""
+Write-Host "Repo : $REPO" -ForegroundColor DarkGray
+Write-Host "Branche : $($env:EPM_BRANCH)" -ForegroundColor DarkGray
+
+# --- 3. pour chaque dossier de donnees suivi par DVC : re-hash + copie lisible ---
+$pointers = Get-ChildItem "epm\input" -Filter "data_*.dvc" -ErrorAction SilentlyContinue
+if (-not $pointers) { Write-Host "Aucun dossier data_*.dvc (modele pas encore migre vers DVC ?)" -ForegroundColor Yellow }
+foreach ($p in $pointers) {
+    $folder = $p.BaseName    # ex: data_blacksea
+    Write-Host ""
+    Write-Host "[data] $folder : dvc add + upload lisible ..." -ForegroundColor Cyan
+    dvc add "epm/input/$folder"
+    $env:EPM_DATA_FOLDER = $folder
+    python "$TOOLS\upload_to_r2.py"
+}
+
+# --- 4. commit + push des pointeurs (si change) ---
+Write-Host ""
+Write-Host "[git] commit + push des pointeurs (si change) ..." -ForegroundColor Cyan
+git add epm/input/*.dvc
+git diff --cached --quiet
+if ($LASTEXITCODE -ne 0) {
+    git commit -m "update data ($(Get-Date -Format 'yyyy-MM-dd HH:mm'))"
+    git push
+} else {
+    Write-Host "   (pointeurs inchanges -> rien a committer)" -ForegroundColor DarkGray
+}
+
+# --- 5. donnees -> store (DVC, pour le serveur) ---
+Write-Host ""
+Write-Host "[dvc] push des donnees vers le store (pour le serveur) ..." -ForegroundColor Cyan
+dvc push
+
+# --- 6. resultats lisibles -> store (pour EPM View) ---
+Write-Host ""
+Write-Host "[results] upload epm/output_view -> store (pour EPM View) ..." -ForegroundColor Cyan
+python "$TOOLS\upload_output_view_to_r2.py"
+
+Write-Host ""
+Write-Host "OK - publie : GitHub (pointeurs) + store (DVC + lisible pour EPM View)." -ForegroundColor Green