Initial code, set up python workflow, start script to import geojson data (#1)

Initial code, set up python workflows to import water companies and zones

Author: nicocrm

.env.example

@@ -0,0 +1 @@
+NOCODB_API_TOKEN=

.gitignore

@@ -160,4 +160,5 @@ dmypy.json
 cython_debug/
 
 # Precommit hooks: ruff cache
-.ruff_cache
+.ruff_cache
+data/

.pre-commit-config.yaml

@@ -1,12 +1,13 @@
 repos:
-  - repo: https://github.com/charliermarsh/ruff-pre-commit
+  - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.
-    rev: "v0.2.1"
+    rev: "v0.14.9"
     hooks:
       - id: ruff
         args: [--fix]
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.3.0
+    rev: v6.0.0
     hooks:
       - id: check-merge-conflict
+      - id: check-illegal-windows-names
       - id: mixed-line-ending

README.md

@@ -1,42 +1,29 @@
-# Template DataForGood
+# VCM Water Watch
 
-This file will become your README and also the index of your
-documentation.
+## Objectif
 
-# Contributing
+Créer une plateforme collaborative et scientifique pour cartographier et analyser les risques de pollution de l’eau par les CVM/VCM dans les tuyaux en PVC installés dans les années 70/80 en Europe. La plateforme permettra de visualiser les risques connus et potentiels, d’identifier les manques de données, et de stimuler la contribution citoyenne et institutionnelle à la recherche.
 
+## Contributing
 
-## Installation
+### Prerequisites
 
-- [Installation de Python](#installation-de-python)
+- Python 3.12
+- [uv](https://docs.astral.sh/uv/) for dependency management
+- [just](https://just.systems/) for running tasks
+- [pre-commit](https://pre-commit.com/)
 
-Ce projet utilise [uv](https://docs.astral.sh/uv/) pour la gestion des dépendances Python. Il est préréquis pour l'installation de ce projet.
-
-Une fois installé, il suffit de lancer la commande suivante pour installer la version de Python adéquate, créer un environnement virtuel et installer les dépendances du projet.
-
-```bash
-uv sync
-```
-
-A l'usage, si vous utilisez VSCode, l'environnement virtuel sera automatiquement activé lorsque vous ouvrirez le projet. Sinon, il suffit de l'activer manuellement avec la commande suivante :
+### Installation
 
 ```bash
-source .venv/bin/activate
+just install
 ```
 
-Ou alors, utilisez la commande `uv run ...` (au lieu de `python ...`) pour lancer un script Python. Par exemple:
-
-```bash
-uv run pipelines/run.py run build_database
-```
-
-
-## Lancer les precommit-hook localement
-
-[Installer les precommit](https://pre-commit.com/)
+This will install:
 
-    pre-commit run --all-files
+ - Python dependencies, using `uv`
+ - pre-commit hooks, using `pre-commit`
 
-## Utiliser Tox pour tester votre code
+### Running Python ETL
 
-    tox -vv
+See [Pipelines Documentation](pipelines/README.md)

d4g-utils/install_poetry.sh

@@ -0,0 +1,18 @@
+_default:
+  @just --list
+  @echo ""
+  @echo "Run 'just <recipe>' to execute a task."
+
+# install environment and pre-commit hooks
+install:
+	uv sync
+	pre-commit install
+
+# run tests for the Python pipelines
+pipelines-test:
+	uv run pytest pipelines
+
+mod extract 'pipelines/extract'
+mod transform 'pipelines/transform'
+mod load 'pipelines/load'
+mod task 'pipelines/tasks'

data/raw/countries.geojson

@@ -0,0 +1 @@
+.prefect-storage/

justfile

@@ -0,0 +1,62 @@
+# Python Workflows
+
+## Python Environment
+
+Install python dependencies:
+
+```bash
+uv sync
+```
+
+`uv` will create a Python virtual environment.  IDE such as VSCode will usually find that automatically, but you can activate
+it manually with:
+
+```bash
+source .venv/bin/activate
+```
+
+or use `uv` to run commands:
+
+```bash
+uv run python
+```
+
+The justfile recipes from the top of the repo will do that automatically.
+
+## Configuration
+
+Credentials are configured in the file `.env` at the root of this repository.
+Copy from .env.example and update with actual values.
+
+## Running an import task
+
+To start a workflow for an import task, use the `just` command at the root of the repository.
+
+There are 4 categories:
+
+ - extract: download raw data and process it into the staging directory
+ - transform: additional processing of staging data
+ - load: load staging data into NocoDB
+ - tasks: additional processing on data within the database
+
+Run `just` with the corresponding category to get a list, for example `just extract`.
+Then run a task by adding the name, for example `just extract download-municipalities`.
+
+## Common Tasks
+
+### Adding a water company
+
+Manually in NocoDB:
+
+- enter Water Company data in NocoDB: Actor 
+- create one or more DistributionZone records, linked with the Water Company.  Link the DistributionZone with corresponding
+municipalities
+- run the task to calculate the missing distribution zone geometries: `just task calculate-distribution-zone`
+
+In bulk:
+
+- alternatively, add data in a raw/WaterCompany_*.ndjson file, with 1 row per distribution zone.  Specify the covered municipalities.
+- run `just transform create-distribution-zones`
+- run `just load zones-distribution`
+- run `just load water-companies`
+- run the task to calculate the missing distribution zone geometries: `just task calculate-distribution-zone`