Skip to content

CALENDAR.md

Latest commit

 

History

History
103 lines (76 loc) · 7.45 KB

CALENDAR.md

File metadata and controls

103 lines (76 loc) · 7.45 KB
Raw

Calendar and Temporal Feature Pipeline

Back to README

This document details the extraction and feature engineering algorithms applied to the pipeline's temporal dimension. The pipeline constructs a deterministic "skeleton" of time (spanning from 2012 to two years into the future) and enriches it with exogenous context, such as public holidays, school vacations, Daylight Saving Time (DST) transitions, and special periods like COVID-19 lockdowns.

This enriched calendar serves as the baseline grid for all time-series alignment in the broader extract, transform, and load (ETL) pipeline.

Part 1: Collection Module (collect_calendar.py)

This module generates the fundamental time index and retrieves official administrative calendars from French government APIs. It creates a temporal baseline that resolves timezone complexities—specifically bridging Coordinated Universal Time (UTC) and local European time (Europe/Paris) containing DST shifts.

Data Sources

  • Primary Source: Generated continuous time skeleton (via Python's pandas).
  • Holidays: data.gouv.fr dataset for national public holidays {cite:p}Etalab:Feries.
  • School Vacations: A merged dataset aggregating two sources to ensure historical continuity:
    1. Post-2018: The modern interval-format dataset from the Ministry of Education {cite:p}EducationNationale:Calendrier.
    2. Pre-2018: An archived daily-boolean dataset to backfill historical data {cite:p}Augusti:Vacances.
  • Scope: Metropolitan France (broken down into school zones A, B, and C).

Key Processing Steps

  1. Merging Historical and Modern Datasets: The French administration changed how it formats school vacation data in 2018. The collector automatically detects the schema, applies heuristic separator detection, and standardizes both into a single continuous timeline.
  2. Temporal Skeleton Generation: Creates standard high-frequency time grids (15-minute and 30-minute intervals) spanning the entire project horizon.

Part 2: Transformation Module (transform_calendar.py)

The raw calendar data is transformed into a rich set of categorical and continuous features designed specifically for machine learning models (e.g., Gradient Boosting Machines, Neural Networks).

1. Categorical Day Classification

The model explicitly flags days that exhibit abnormal energy consumption patterns:

  • Weekends and Holidays: Boolean flags for Saturdays/Sundays and official bank holidays.
  • Bridge Days (Ponts): Automatically detects if a Tuesday or Thursday is a holiday, subsequently flagging the adjacent Monday or Friday as a "bridge day" (where a significant portion of the workforce takes leave).
  • School Vacations: Boolean flags tracking whether any zone (A, B, or C) is currently on vacation, which directly impacts residential versus commercial load distribution.

2. Time Standardization: Continuous Local Time (CLT)

Energy consumption is driven by human activity, which follows local clock time, not UTC. However, standard local time includes non-linear jumps (skipping an hour in spring, repeating an hour in autumn) due to DST.

To solve this, the pipeline maps all data to Continuous Local Time (CLT). CLT is a monotonic index that aligns with the local human clock but mathematically removes the DST discontinuities, preventing models from interpreting DST shifts as sudden, massive drops or spikes in energy demand.

3. Cyclical Encoding (Time of Day / Time of Year)

To help machine learning models understand the circular nature of time (e.g., that 23:45 is only 15 minutes away from 00:00), we normalize the time steps into continuous cyclical variables:

  • Time of Day (tod): Normalized from 0 to 1 based on the daily frequency limit.
    • Formula: $tod = \frac{\text{minute_of_day}}{\text{steps_per_day}}$
  • Time of Year (toy): Normalized from 0 to 1 based on the exact progress through the current year (accounting for leap years).
    • Formula: $toy = \frac{\text{day_of_year} - 1}{\text{days_in_year}} + \frac{tod}{\text{days_in_year}}$

(Note: In downstream imputation models, these tod and toy features are converted into sine and cosine pairs to provide a strictly continuous, circular input.)

4. Composite Categorical Keys

Instead of feeding the model multiple separate dummy variables, the pipeline creates composite features that capture cross-interactions. For example, day_type_week_period_hour_changed merges the day of the week with the current DST season (Summer/Winter), allowing tree-based models to split on seasonal weekly routines in a single operation.

5. Validity Masking (Outlier Exclusion)

Certain periods contain highly irregular data that can corrupt historical training sets. The module defines a DayValidity flag, setting it to 0 (invalid) for:

  • DST Switch Days: The exact 24-hour periods during the spring and autumn clock changes.
  • COVID-19 Lockdowns: The three major French lockdown periods (Spring 2020, Autumn 2020, and Spring 2021). These periods are masked out so forecasting models do not learn pandemic-era demand patterns as standard behavior.

Final Feature Schema

The transformed database (transformed_calendar_15min and _30min) produces the following structure:

Column Name Type Description
date TIMESTAMPTZ Standard UTC timestamp (Primary Index).
Date INTEGER Local date reference (Format: YYYYMMDD).
month INTEGER Month of the year (1-12).
year INTEGER 4-digit year.
tod DOUBLE Time of day (normalized 0.0 to 1.0).
toy DOUBLE Time of year (normalized 0.0 to 1.0).
week_number INTEGER ISO week number (Format: yyyyww).
DayValidity INTEGER Validity Flag. 0 if DST switch or COVID-19 period, else 1.
Day Types
day_type_week INTEGER Day of week (0=Monday ... 6=Sunday).
day_type_jf INTEGER Is a bank holiday (1/0).
day_type_vjf INTEGER Is the day before a holiday (1/0).
day_type_ljf INTEGER Is the day after a holiday (1/0).
day_type_week_jf INTEGER Is a weekend OR a holiday (1/0).
day_type_hc INTEGER Is a day containing a UTC DST switch (1/0).
dst_repair_target INTEGER Is a day containing a Local CLT DST switch (1/0). Used for gap imputation.
Periods
period_hour_changed INTEGER DST Status (1=Summer Time, 0=Winter Time).
period_holiday INTEGER Is any school vacation active (1/0).
period_holiday_zone_a INTEGER School vacation active in Zone A (1/0).
period_holiday_zone_b INTEGER School vacation active in Zone B (1/0).
period_holiday_zone_c INTEGER School vacation active in Zone C (1/0).
period_christmas INTEGER Is the Christmas holiday period (1/0).
period_summer INTEGER Is the Summer holiday period (1/0).
Composite Keys
day_type_week_period_hour_changed INTEGER Composite: Day of Week + Summer/Winter Season.
day_type_week_jf_period_holiday INTEGER Composite: Weekend/Holiday + Active School Vacation.

Output Integration

The outputs of this module are saved to data_warehouse/transform_calendar.duckdb. They are never used in isolation; rather, the calendar_15min and calendar_30min tables are utilized as the Left Table in all downstream SQL joins during the dataset assembly phase. This ensures zero missing timestamps in the final outputs.