Version: 1.1.1
License: LGPL-3.0-or-later.
Copyright (c): 2020-2026 EDF (Electricité De France).
Copyright (c): 2023 Sorbonne Université.
Active contributor: Yann Allioux (2020-2026)
Past contributor: Nathan Doumèche (2023-2025)
Scientific advisor: Yannig Goude
FORCE provides an extract, load, and transform (ETL) pipeline to aggregate, harmonize, and validate multi-source energy data for the French power system. It addresses the challenge of heterogeneous data by unifying divergent temporal resolutions (e.g., 15-minute vs. 30-minute), spatial granularities, and physical units into a continuous time series matrix.
The package is built for researchers working on time series forecasting, grid economics, and climate-energy modeling. Relying on DuckDB as its analytical backend {cite:p}raasveldt2019duckdb, FORCE consolidates data from four primary sources:
- RTE (Transmission): A multi-layer architecture tracking data maturity (real-time, consolidated, definitive) to synthesize a single reference record {cite:p}
RTE:Eco2Mix. - Enedis (Distribution): Granular consumption profiles and decentralized generation metrics {cite:p}
Enedis:Bilan. - Météo-France (Weather): Synoptic observations transformed via asset-weighted spatial aggregation {cite:p}
MeteoFrance:Synop. - ODRÉ (Assets): A georeferenced registry of power plants used to generate spatial weights {cite:p}
ODRE:Registre.
Each component of the pipeline is documented with full algorithmic transparency:
| Component | Documentation Link | Key Algorithms |
|---|---|---|
| Calendar | CALENDAR.md | Skeleton generation, UTC-DST bridge, topological day typing {cite:p}Etalab:Feries,EducationNationale:Calendrier,Augusti:Vacances. |
| RTE | RTE.md | Multi-version pivoting, conservative upsampling, maturity-aware deduplication. |
| Enedis | ENEDIS.md | Patch and fill strategy, profile reconstruction. |
| Météo | METEO.md | Multi-barycenter aggregation, vector decomposition (for circular wind variables). |
| Radiation | RADIATION.md | PVGIS POA (Plane of Array) & ERA5 GHI (Global Horizontal Irradiance) collection. |
| Assets | ODRE.md | Two-stage geocoding {cite:p}OpenDataSoft:Georef, spatial centroids. |
| Imputation | IMPUTATION.md | Pluggable engines, recursive feature relaxation, spatial kriging (interpolation), 7-step causal DAG (Directed Acyclic Graph). |
| Hierarchies | HIERARCHIES.md | Verified column sum-trees, energy balance, calendar encodings. |
| Audit | audit/README.md | Per-dataset visualization suite, notebook-engine reproduction. |
| TAM Extraction | TAM.md | Autoregressive lag/thermal-inertia feature engineering, legacy dataset alignment. |
FORCE requires Python 3.10 or higher (Python 3.12+ recommended for the full TS-ICL imputation engine).
As a package (for end users):
pip install force-dataset
From source (for development or running the pipeline). All dependencies are managed via pyproject.toml.
python -m venv .venv
# activate: PowerShell: .\.venv\Scripts\Activate.ps1 | bash/zsh: source .venv/bin/activate
pip install -e ".[dev]"
Optional extras include dev (tests/lint), docs (Sphinx), dashboard (Jupyter), climate (CDS/CMIP6), and icl (TS-ICL imputation engine, requiring Python >= 3.12).
Execute the pipeline steps sequentially to build the dataset:
Phase A: Collection Fetches raw data from APIs. The state-aware engine detects data maturation (e.g., when real-time data is retroactively validated).
energy-collect
Phase B: Transformation Applies cleaning, upsampling, and technical repair. Structural gaps are strictly preserved as NaN.
energy-transform
Phase C: Database Audit Scans DuckDB files for temporal coverage gaps, empty tables, and schema anomalies. For diagnostic visualizations, use energy-viz-transform and energy-viz-assemble.
energy-check
Phase D: Assembly & Imputation Joins sources and applies physics-informed imputation.
energy-assemble
energy-impute # Uses TS-ICL by default; pass '--engine xgboost' for gradient boosting
The pipeline addresses specific challenges in energy data engineering:
The pipeline standardizes all data streams to a 15-minute resolution:
-
Energy Flows (Grid Data): We apply a conservative smoothing kernel that mathematically preserves the energy integral (MWh) over the block. Note: Minor numerical deviations may occur when enforcing physical bounds (e.g., clipping negative generation). Formula:
$y_{t+15} = \frac{3Y_t + Y_{t+30}}{4}$ . -
State Variables (Meteo Data): Instantaneous observations (e.g., temperature) use strict linear upsampling (
$limit=1$ ) to calculate the intermediate point ($t+15$ ) without modifying the original anchors.
A simple geographic average of weather stations introduces bias because energy assets are unevenly distributed.
-
Spatial Solution: We calculate usage-specific centroids (
$G_{load}$ ,$G_{wind}$ ,$G_{solar}$ ) for every region. Station data is aggregated using a modified inverse distance weighting (IDW) algorithm. To ensure numerical stability, this IDW applies a 1 km softening factor and assumes an oblate spheroid Earth ($R=6371$ km). -
Temporal Solution: The pipeline computes time-varying weights (
$W_{r,y}$ ). The influence of a region$r$ on the national signal evolves annually$y$ based on its actual market share. Formula:$I_{national}(t) = \sum_{r} W_{r, y(t)} \cdot I_{r}(t)$
Energy data evolves from real-time estimates to validated definitive records. For critical metrics like national load, the pipeline produces multiple time series versions:
_realtime: For simulating production forecasting._definitive: For training on ground truth._best: A synthesized golden record for general use.
We separate technical repair from statistical reconstruction to guarantee data integrity:
Targets micro-gaps (< 1h) caused by sensor dropouts.
-
Grid Data: Cyclical variables are repaired via a profile-guided bridge (
$D-7$ ) to preserve intra-hour peaks. Stochastic variables (wind/solar) use linear interpolation. - Meteo Data: Utilizes lapse-rate aware k-Nearest Neighbors (k-NN) to prioritize vertical similarity, and Time-of-Day (TOD) fallbacks to bridge gaps when spatial neighbors are unavailable.
Targets structural outages and historical backfilling for gaps > 1h.
-
TS-ICL Engine (Default): A zero-shot time-series foundation model. It handles raw temporal context directly and ignores the
training_cutoff_yearsparameter. -
XGBoost Engine (Fallback): An extreme gradient boosting regressor that uses engineered lag features, thermal inertia smoothing, and a Brownian bridge to ensure
$C_0$ continuity at gap boundaries.
The Outputs/Imputed/ directory contains the fully reconstructed datasets in two frequencies (15min, 30min).
dataset_15min_france_load.csv: Drivers for consumption forecasting (temperature, nebulosity, calendar).dataset_15min_france_wind.csv: Drivers for wind power (wind speed, air density, gusts).dataset_15min_france_solar.csv: Drivers for photovoltaics (irradiance proxies, temperature).dataset_15min_full.csv: The complete high-dimensional matrix (~400 columns).
| Feature | Unit | Description |
|---|---|---|
rte_load_france |
MW | Reference record for national load (best of definitive/consolidated/real-time). |
rte_load_france_definitive |
MW | Strictly validated historical load (isolated for deviation/error analysis). |
enedis_load_residential_total_france |
MW | Reconstructed residential load combining telemetered (Linky) and profiled data. |
meteo_temperature_celsius_france_load |
°C | National temperature proxy, dynamically weighted by regional population density. |
day_type_week_period_hour_changed |
Cat | Composite calendar key encoding day of the week, holidays, and seasonality. |
force/
├── src/force/
│ ├── collectors/ # API Connectors (State-Aware & Robust)
│ ├── transformation/ # Physics-Aware Smoothing & Multi-Version Pivoting
│ ├── check_*.py # Quality Control & Auditing Modules
│ └── imputation/ # ML Imputation Engine (Predictive, Spatial, Audit)
├── documentation/ # Detailed Methodological Papers
├── data_warehouse/ # Local DuckDB Storage (Normalized Schemas)
└── Outputs/ # Final CSV Datasets
Recognizing that reproducing the full ETL pipeline requires time, computational resources and reliable API access, we provide a static, versioned snapshot of the finalized dataset hosted on Zenodo.
- Download the 2026 Dataset Snapshot: https://doi.org/10.5281/zenodo.21109134
If you use the FORCE package or the pre-compiled dataset in your research, please cite them using their permanent archives:
To cite the FORCE Software Pipeline:
@misc{force2026package,
title={FORCE: French Open Research Catalogue of Energy (v1.1.1)},
author={Allioux, Yann and Goude, Yannig},
year={2026},
doi={10.5281/zenodo.21108937},
note={With a contribution of Nathan Doumèche during his thesis}
}To cite the FORCE Dataset Snapshot:
@dataset{force2026dataset,
title={FORCE Dataset: French Open Research Catalogue of Energy - 2026 Snapshot},
author={Allioux, Yann and Goude, Yannig},
year={2026},
publisher={Zenodo},
doi={10.5281/zenodo.21109134},
url={https://doi.org/10.5281/zenodo.21109134}
}FORCE: French Open Research Catalogue of Energy is released under the LGPL-3.0-or-later license.
- 2025-2026 Advanced imputation engine, pipeline architecture, Enedis & radiation features, DuckDB Ecosystem: Copyright © EDF, author Yann Allioux.
- 2023 Imputation engine & scientific publication: Copyright © EDF & Sorbonne Université, authors Yann Allioux, Nathan Doumèche.
- 2020-2022 Core framework & baselines: Copyright © EDF, author Yann Allioux.
Data Attribution & Licenses:
- Grid and synoptic weather data provided by RTE, Enedis, Météo-France, and ODRÉ under the Etalab Open License 2.0.
- Solar radiation baseline data provided by the European Commission PVGIS (SARAH-3).
- Reanalysis and climate projection data generated using Copernicus Climate Change Service information (ERA5, CMIP6) and subject to the Copernicus License. Users of this pipeline must register for a Copernicus account and accept their Terms of Use.