An end-to-end, config-driven pipeline that turns the minimum manual inputs for a new river basin — a catchment polygon, a DEM, and four soil rasters — into a complete, internally consistent LISFLOOD model, then runs it and validates the output. A separate module performs multi-objective hydrological calibration.
LISFLOOD is the distributed rainfall–runoff and channel-routing model developed by the European Commission's Joint Research Centre (the engine behind the European and Global Flood Awareness Systems, EFAS/GloFAS). Setting it up for a new basin by hand means preparing dozens of interlocking rasters, NetCDF forcing files, and an XML settings file in which a single mismatched grid or wrong flow-direction cell makes the run fail or, worse, succeed with silently wrong physics. This pipeline automates that setup and checks its own work at each step.
Scope. The pipeline produces a technically consistent model and verifies that LISFLOOD runs on it. It does not decide that the result is hydrologically correct — see Limitations. An uncalibrated hydrograph is not a final estimate of streamflow.
Standing up LISFLOOD on an ungauged or newly studied basin is mostly data engineering, not modelling: reproject everything onto one grid, derive flow direction and accumulation, pick an outlet, build the local drainage direction (LDD) map with exactly one pit, cut the country-wide meteorology to the basin window, compute the forcing variables LISFLOOD expects, translate global land-cover classes into fraction maps, convert SoilGrids into model parameters, generate LAI and ancillary maps, and finally write an XML settings file whose dates and paths all line up with the files just produced. Each of these is a place to make a quiet, run-ruining mistake.
The pipeline does all of it from a single JSON config, reuses expensive
intermediate results between runs, and refuses to proceed when an input fails an
audit. The design assumption is a shared national setup: one country-wide
ERA5-Land file and one global land-cover raster live once in a common/ folder and
are windowed per basin, so adding a new basin only requires that basin's polygon,
DEM, and soil.
run_pipeline.sh prepare runs fifteen steps in order; each writes files the next
one audits.
| # | Step | What it does |
|---|---|---|
| 0 | validate config / build grid | check required files and CRS; define the target grid and basin mask |
| 1 | crop ERA5 | window the country-wide ERA5-Land NetCDF to the basin, without loading the whole file |
| 2 | build meteo | interpolate forcing onto the grid; compute tp, ta, e0, es0, et0 (chunked over time) |
| 3 | check meteo | audit the forcing NetCDFs for range, gaps, and grid agreement |
| 4 | build + inspect hydrology | derive D8 flow direction and accumulation from the DEM; report the drainage structure |
| 5 | prepare hydro grid | align the hydrology onto the model grid |
| 6 | build LDD | select the outlet and build a local drainage direction map with a single pit, using a BFS from the outlet over the accumulation surface |
| 7 | build base maps | channel geometry, gradients, and the other static routing maps |
| 8 | build land cover | read only the basin window of the global raster; convert classes into forest / sealed / water / irrigated / other fractions |
| 9 | build soil | turn SoilGrids clay/sand/silt/bulk-density into LISFLOOD soil parameters |
| 10 | build LAI + ancillary | leaf-area-index and remaining ancillary maps |
| 11 | validate inputs | audit every generated NetCDF before the model sees it |
| 12 | build settings | write settings_PreRun.xml, settings_Smoke.xml, settings_Run.xml from the actual ERA5 date range; the reference XML is never mutated |
Then the model itself is run in three escalating stages, each validated:
run_pipeline.sh prerun # initialise state (avgdis, lzavin) via settings_PreRun.xml
run_pipeline.sh smoke # short run to catch failures fast, then 14_validate_run.py
run_pipeline.sh full # full-period run -> out/Q_model.csv, then validation
run_pipeline.sh all # prepare + prerun + smoke + full
Once the hydrology is built it is reused on later runs; set "hydrology": {"rebuild": true} in the config to force a recompute after changing the DEM or
polygon.
- Config-driven, no hard-coded paths. Every input, output, and option comes
from one
pipeline.json.pipeline_config.pyresolves paths relative to the basin folder, expands~and environment variables, and fails loudly on a missing required key. The same bundle runs any basin without edits. - Windowed reads of huge global rasters. The land-cover step reads only the polygon's window; the full global raster would need tens of GiB of RAM. ERA5 is cropped before any per-day computation.
- Self-auditing. Config, meteorology, all generated NetCDFs, and the final run each have a dedicated validation step. The pipeline is built to stop early rather than produce a plausible-looking but broken model.
- Idempotent and resumable. Expensive hydrology is cached; the escalating prerun → smoke → full sequence catches errors on a short run before committing to the full period.
- Reference XML is immutable. Settings files are generated fresh from real ERA5 dates; the template you supply is never modified in place.
run_calibration.sh (module in calibration/) tunes the model against observed
discharge using Optuna. It is deliberately separate from
setup: you calibrate only after prerun, smoke, and full succeed, and it
writes a separate settings_Run_calibrated.xml rather than touching
settings_Run.xml.
-
Parameters (configurable ranges, log-scale where appropriate):
SnowMeltCoef,UpperZoneTimeConstant,LowerZoneTimeConstant,GwPercValue,b_Xinanjiang,CalChanMan. -
Objective — a weighted multi-criteria score rather than a single metric, so the optimiser is not allowed to buy peak fit at the cost of low-flow or volume bias:
Component Default weight Guards against KGE 0.40 overall fit (correlation, bias, variability) log-KGE 0.25 poor low-flow / recession behaviour NSE 0.15 squared-error fit, peak-weighted log-NSE 0.10 low-flow fit PBIAS penalty 0.10 long-term volume bias α penalty 0.10 wrong flow variability -
Honest evaluation — a calibration period and a separate validation period are configured explicitly; the study is stored in a resumable SQLite database so a long run can be stopped and continued.
Observed discharge is supplied by the user as a CSV (DATE,Q_obs, daily m³/s) and
is not part of this repository.
The pipeline targets Linux (or WSL on Windows) because LISFLOOD, PCRaster, and the GDAL stack are most reliable there. In a conda environment:
conda create -n lisflood python=3.11 -y
conda activate lisflood
conda install -c conda-forge \
gdal pcraster netcdf4 "numpy<2" xarray dask pandas scipy \
rasterio geopandas shapely pyogrio affine matplotlib openpyxl \
optuna scikit-image -y
pip install lisflood-model
pip install elevation pysheds affine --no-depsVerify:
python -c "import lisflood; print('LISFLOOD OK')"
python -c "from osgeo import gdal; import pcraster, netCDF4, rasterio, geopandas, pysheds; print('deps OK')"Full step-by-step setup for a new basin — including how to obtain the DEM
(elevation) and SoilGrids
(ISRIC WCS) — is in
README_UNIVERSAL.md; calibration details are in
CALIBRATION_README.md.
| Input | Format | Notes |
|---|---|---|
| Catchment polygon | shapefile | any CRS; reprojected automatically. The one input the pipeline cannot infer. |
| DEM | GeoTIFF, EPSG:4326 | cover the basin with margin so border hydrology is correct |
| SoilGrids | 4 × GeoTIFF | clay / sand / silt / bulk density, top 0–5 cm layer |
| ERA5-Land | NetCDF | daily d2m, ssrd, t2m, tp, u10, v10; shared across basins |
| Land cover | GeoTIFF | global raster, read per-basin window; shared across basins |
Outlet selection is configurable: automatic (max accumulation inside the basin) or snapped to the nearest accumulation maximum around a supplied gauge coordinate.
- Setup, not truth. The pipeline guarantees a runnable, internally consistent model, not a hydrologically validated one. Channel, soil, LAI, and threshold parameters must be checked for the specific basin, and the model calibrated, before any hydrograph is used for decisions.
- Linux / WSL is assumed for the geospatial stack.
- DEM quality drives everything downstream. A poor DEM yields a poor flow network and a poor LDD; the inspection step reports the drainage structure, but the user must sanity-check it.
- ERA5-Land resolution (~9 km) sets the spatial detail; small headwater basins are only coarsely resolved.
This repository contains the method only. No basin data, observations, calibrated parameters, or model outputs are included. The reference settings file ships with neutral default parameters, not values calibrated on any specific gauge. The pipeline runs on any basin for which the user supplies the inputs above; global sources (ERA5-Land, SoilGrids, DEMs) are freely available.
Developed with AI-assisted coding. The pipeline design, the hydrological workflow, and the choice of validation steps are the author's. Some in-code comments and the two detailed guides are in Russian; this overview and the code interfaces are in English.
MIT — see LICENSE.