Docs+tools: automate model onboarding (setup_model.ps1) + tabs for setup, smaller fonts, collapsible details

Author: mbaronnet

docs/introduction/introduction_case_studies.md

@@ -6,7 +6,7 @@ As the World Bank's primary power sector planning tool, EPM has been applied in
 
 ## Coverage
 
-<div style="display:inline-block; margin:0.1rem 0 0.7rem 0; padding:4px 11px; border:1px solid #d8b24a; border-radius:8px; background:#fcf6e6; font-size:0.66rem; font-style:italic; color:#7a5d1e;">
+<div style="display:inline-block; margin:0.1rem 0 0.7rem 0; padding:3px 10px; border:1px solid #d8b24a; border-radius:8px; background:#fcf6e6; font-size:0.58rem; font-style:italic; color:#7a5d1e;">
   → Explore the data &amp; results interactively in
   <a href="https://epm-data-explorer.vercel.app" target="_blank" rel="noopener" style="font-style:normal; font-weight:600; color:#1E6DB8;">EPM&nbsp;View</a>
   &nbsp;·&nbsp;<span style="color:#a98a4a;">access required — ask the team</span>

docs/run/run_data_sync.md

@@ -1,5 +1,7 @@
 # Publishing & Syncing Data (DVC)
 
+<style>.md-typeset { font-size: 0.8rem; }</style>
+
 EPM **code** is public, but the **data** (input CSVs, results) is kept in a **private
 store** — not in this public repository. The repo only keeps tiny **pointer files**
 (`*.dvc`); the real data lives in the store and is fetched on demand.
@@ -27,23 +29,24 @@ and is moved with `dvc push` / `dvc pull`. So git stays light, and the data stay
 
 You work in **your clone of the EPM repo** — the same folder you run EPM from.
 
-**A. If you already run EPM on this machine** (Python env ready) — just add the data tools:
+=== "Already running EPM here"
 
-```bash
-# in your EPM Python environment (the one you use to run EPM)
-pip install -r requirements.txt
-```
-This installs **DVC**, `s3fs`, and (on Windows) `pip-system-certs`.
+    Your Python environment is ready — just add the data tools:
+    ```bash
+    pip install -r requirements.txt
+    ```
+    Installs **DVC**, `s3fs`, and (on Windows) `pip-system-certs`.
 
-**B. If it's a fresh machine** — set EPM up first, then the line above:
+=== "Fresh machine"
 
-```bash
-git clone https://github.com/ESMAP-World-Bank-Group/EPM.git
-cd EPM
-git checkout <your-branch>                 # e.g. blacksea_2026
-conda activate <your-epm-env>              # your EPM Python environment
-pip install -r requirements.txt
-```
+    Set EPM up first, then install:
+    ```bash
+    git clone https://github.com/ESMAP-World-Bank-Group/EPM.git
+    cd EPM
+    git checkout <your-branch>             # e.g. blacksea_2026
+    conda activate <your-epm-env>          # your EPM Python environment
+    pip install -r requirements.txt
+    ```
 
 **Then (both cases) add your store keys — once:**
 
@@ -78,25 +81,32 @@ That's it — the store URL/endpoint is already in the repo (`.dvc/config`).
 This moves a model's data **out of git** into the store. Do it **once**, by whoever sets the
 model up; afterwards everyone just uses *setup + publish/sync*.
 
+**One command** (from the repo root) — it asks for confirmation, then makes the change locally:
+
 ```bash
-# 1. Initialise DVC for the repo (once per repo)
-dvc init
+powershell -File tools/setup_model.ps1 data_<model>     # e.g. data_sapp
+```
 
-# 2. Point DVC at the store
-dvc remote add -d store s3://<bucket>/dvcstore
-dvc remote modify store endpointurl <endpoint>
+Then **review** (`git status`) and **publish** (double-click `Publish.bat`). Finally, to make
+**EPM View** show this model, add its branch name to `R2_BRANCHES` in `src/utils/epmFetch.js`
+of the `epm-data-explorer` repo.
 
-# 3. In .gitignore: remove the line that force-tracks your data folder,
-#    and add an exception for its pointer:
-#        !epm/input/data_<model>.dvc
+??? info "What `setup_model.ps1` does under the hood (the manual steps)"
+    The script automates exactly these — your files **stay on disk**, and it commits/pushes
+    nothing (you review, then publish):
 
-# 4. Stop tracking the data in git (files stay on your disk), then hand it to DVC
-git rm -r --cached epm/input/data_<model>
-dvc add epm/input/data_<model>             # creates the .dvc pointer
-```
-
-Then **publish** (step 4). Finally, to make **EPM View** read this model from the store, add
-its branch name to `R2_BRANCHES` in `src/utils/epmFetch.js` of the `epm-data-explorer` repo.
+    ```bash
+    # 1. Initialise DVC for the repo (once per repo)
+    dvc init
+    # 2. Point DVC at the store (once — already set in this repo's .dvc/config)
+    dvc remote add -d store s3://<bucket>/dvcstore
+    dvc remote modify store endpointurl <endpoint>
+    # 3. .gitignore: drop the whitelist of the data folder, keep its pointer tracked
+    #        !epm/input/data_<model>.dvc
+    # 4. Stop tracking the data in git (files stay), then hand the folder to DVC
+    git rm -r --cached epm/input/data_<model>
+    dvc add epm/input/data_<model>            # creates the .dvc pointer
+    ```
 
 ---
 
@@ -163,5 +173,6 @@ run the model with up-to-date data.
 |------|------|
 | `publish.ps1` | engine behind `Publish.bat` |
 | `sync.ps1` / `sync.sh` | get code + data (Windows / server) |
+| `setup_model.ps1` | onboard a new model's data to the store (one-time, step 3) |
 | `upload_to_r2.py`, `upload_output_view_to_r2.py` | upload the readable copies |
 | `.env.example` | keys template (copy to `.env`, git-ignored) |

tools/setup_model.ps1

@@ -0,0 +1,65 @@
+# setup_model.ps1 — onboard a model's data to the private store (one-time, per model).
+#
+#   Usage:   powershell -File tools/setup_model.ps1 data_<model>
+#   Example: powershell -File tools/setup_model.ps1 data_sapp
+#
+# Moves a data folder OUT of git into DVC (the store). Files stay on your disk.
+# This script only makes LOCAL changes — review them, then publish with Publish.bat.
+
+param([Parameter(Mandatory = $true)][string]$DataFolder)
+
+$ErrorActionPreference = "Stop"
+$TOOLS = $PSScriptRoot
+$REPO  = Split-Path $TOOLS -Parent
+Set-Location $REPO
+
+$path = "epm/input/$DataFolder"
+if (-not (Test-Path $path)) { Write-Error "Not found: $path"; exit 1 }
+
+$branch = (git rev-parse --abbrev-ref HEAD).Trim()
+Write-Host ""
+Write-Host "Repo   : $REPO"            -ForegroundColor DarkGray
+Write-Host "Branch : $branch"          -ForegroundColor DarkGray
+Write-Host "Folder : $path"            -ForegroundColor DarkGray
+Write-Host ""
+Write-Host "This moves '$path' OUT of git into DVC/the store." -ForegroundColor Yellow
+Write-Host "Files STAY on your disk; on your next push the data leaves the public repo." -ForegroundColor Yellow
+if ((Read-Host "Continue? (y/N)") -ne "y") { Write-Host "Aborted."; exit 0 }
+
+# 1. DVC init (once per repo)
+if (-not (Test-Path ".dvc")) {
+    Write-Host "[1/4] dvc init"
+    dvc init | Out-Null
+    Write-Host "      /!\ remote not configured. Run once, then re-run this script:" -ForegroundColor DarkYellow
+    Write-Host "          dvc remote add -d store s3://<bucket>/dvcstore" -ForegroundColor DarkYellow
+    Write-Host "          dvc remote modify store endpointurl <endpoint>" -ForegroundColor DarkYellow
+} else {
+    Write-Host "[1/4] dvc already initialised"
+}
+
+# 2. .gitignore: drop the whitelist of the folder, keep only its .dvc pointer tracked
+Write-Host "[2/4] update .gitignore"
+$gi  = ".gitignore"
+$esc = [regex]::Escape("epm/input/$DataFolder")
+$lines = Get-Content $gi | Where-Object { $_ -notmatch "^\!$esc(/|$)" }   # remove !epm/input/<folder>/  and  /**
+$ptr = "!epm/input/$DataFolder.dvc"
+if ($lines -notcontains $ptr) {
+    # make sure the data folder is ignored at all (covers the 'epm/input/*' pattern repos)
+    if (-not (Select-String -Path $gi -SimpleMatch "epm/input/*" -Quiet)) { $lines += "epm/input/$DataFolder/" }
+    $lines += $ptr
+}
+Set-Content $gi $lines
+
+# 3+4. stop tracking in git (files stay on disk), then hand the folder to DVC
+Write-Host "[3/4] git rm --cached (untrack, files stay on disk)"
+git rm -r --cached $path | Out-Null
+Write-Host "[4/4] dvc add (create the pointer)"
+dvc add $path
+
+Write-Host ""
+Write-Host "DONE (local changes only)." -ForegroundColor Green
+Write-Host "Next:" -ForegroundColor Green
+Write-Host "  1. review        : git status"
+Write-Host "  2. publish       : double-click Publish.bat"
+Write-Host "  3. EPM View      : add '$branch' to R2_BRANCHES in"
+Write-Host "                     epm-data-explorer/src/utils/epmFetch.js"