RTE.md

RTE Power Data Pipeline

Back to README

This document details the extraction and standardization algorithms applied to the RTE (Réseau de Transport d'Électricité) data. The pipeline utilizes a data processing architecture designed to unify historical consolidated records with real-time streams, ensuring a continuous and strictly typed dataset for energy forecasting.

---

Part 1: Collection Module (collect_rte.py)

This module manages the acquisition of French power system data (load, generation, and exchanges). It implements a multi-threaded, state-aware ingestion engine capable of navigating the API constraints of the Open Data Réseaux Énergies (ODRÉ) platform while handling schema evolution between legacy and modern datasets.

Data Source

• Provider: RTE (via ODRÉ - Open Data Réseaux Énergies).
• Datasets:
  • Consolidated (Definitive): eco2mix-national-cons-def, eco2mix-regional-cons-def
  • Real-time (Provisional): eco2mix-national-tr, eco2mix-regional-tr
• Protocol: REST API v2.1 (OpenDataSoft) with pagination.
• Granularity: 15-minute (real-time) to 30-minute (historical) intervals.
• License: Etalab Open License {cite:p}RTE:Eco2Mix.

Key Collection Strategies

1. Safe Pagination

The OpenDataSoft API limits standard offset pagination to 10,000 records. To extract decades of high-frequency data, the collector slices the requested time horizon into discrete 10-day chunks, querying each chunk concurrently to bypass deep-paging limitations.

2. Physical Reconstruction (Imputation)

Due to API schema changes over the years, certain aggregate variables (like total wind or total solar generation) may be missing in older records. The collector automatically reconstructs these missing totals by summing their available sub-components (e.g., Total Wind = Onshore Wind + Offshore Wind) to guarantee physical consistency.

3. Handling Data Maturity

Energy data matures over time. A record starts as a real-time estimate, becomes consolidated after several months, and is finally marked as definitive. The collector prioritizes downloading consolidated streams over real-time streams to minimize the ingestion of volatile data.

---

Part 2: Transformation Module (transform_rte.py)

The raw data downloaded from the API is structured in a "Long Format" (few columns, many rows, where the metric name is a value in a column). This module pivots the data into a "Wide Format" (where each metric becomes its own column) and handles the temporal alignment required for machine learning.

1. Temporal Normalization

• Timestamp Shifting: Raw RTE data uses an "end-of-period" convention (e.g., data labeled 00:30 represents the period from 00:00 to 00:30). The transformer applies a constant -30 minute shift to every row, regardless of the source resolution, to align with standard data science "start-of-period" conventions.
• Resolution Unification: Legacy data is reported in 30-minute intervals, while modern data uses 15-minute intervals. The transformer upsamples all 30-minute legacy data to 15-minute intervals using a conservative smoothing kernel (which preserves the total energy integral in MWh) and bridges micro-gaps (up to 1h) using plain linear interpolation for stochastic metrics (wind, solar, CO2 rate, net exchange) and profile-guided interpolation for everything else.

2. Transforming the Structure (Flattening)

The raw data mixes metrics (load, solar, wind) with spatial scopes (national vs. regional) in a single column. The transformer flattens this into a strict variable-space matrix:

• Format: {source}_{metric}_{spatial_scope}
• Example: rte_load_france, rte_solar_bretagne

3. Multi-Version Synthesis (The Golden Record)

Because energy data matures, multiple values can exist for the same timestamp (e.g., a real-time estimate and a later definitive measurement).

For critical metrics (specifically national load), the transformer preserves the distinct versions to allow for model deviation analysis:

• rte_load_france_realtime
• rte_load_france_consolidated
• rte_load_france_definitive

It then synthesizes a Golden Record (rte_load_france) by deduplicating the versions based on a strict hierarchy: Definitive > Consolidated > Real-Time.

---

Final Feature Schema

1. Asymmetric Topology

The pipeline enforces a strict hierarchical topology. Not all metrics are available (or relevant) at the regional level.

• National Scope: Contains all aggregated totals and granular sub-types (e.g., offshore vs. onshore wind, gas vs. coal).

• Regional Scope: Contains only major aggregates (load, total wind, total solar, total nuclear, total hydro, bioenergy, pumped storage, and net exchange).

• Rule: $m \in M_{aggregate}, r \in R_{regions}$

• Metric Space ($M_{aggregate}$): All metrics that have regional data except the national-only sub-types wind_onshore / wind_offshore.

2. Consistency Validation (The "National Count" Check)

The asymmetric structure allows for a physical consistency check during the validation phase. The sum of regional aggregates should converge towards the national aggregate:

$\sum_{r \in Regions} rte_wind_{r}(t) \approx rte_wind_{france}(t)$

3. Table Structure

Column Name	Scope	Description
date	Index	Primary Key (UTC). Continuous step.
rte_load_france	National	Best Available load (Definitive > Consolidated > Real-Time).
rte_load_france_definitive	National	Load strictly from the "Données définitives" stream.
rte_load_france_consolidated	National	Load strictly from the "Données consolidées" stream.
rte_load_france_realtime	National	Load strictly from the "Données temps réel" stream.
rte_wind_france	National	Best available total wind generation.
rte_wind_onshore_france	National	Sub-type: Onshore wind generation.
rte_load_bretagne	Regional	Regional load for Bretagne.
rte_solar_occitanie	Regional	Regional solar generation for Occitanie.
rte_exchange_net_france	National	Net cross-border commercial exchanges.

(Note: The actual output table contains ~100 columns encompassing all regions and available generation types).