Feat/15 min frequency#24
Merged
Merged
Conversation
Extend BaseModel, worker, and ModelPipeline to support sub-hourly forecasting. When freq="15min" the pipeline trains 96 sub-models per day instead of 24. Key changes: - Add _FREQ_TO_PERIODS map and validate freq early with a clear message - Replace hardcoded range(24) with range(_periods_per_day) throughout - Add _compute_period_index() for freq-aware period slot calculation - Use calendar-day shifts (not elapsed-24h) for DST-safe future index alignment in _preprocess() - Build day IDs from local calendar dates to prevent DST-induced day-number duplication - Store _predictors_fn separately so callable predictors are invoked each call instead of being overwritten on first evaluation - Guard worker tasks against insufficient samples, NaN/inf predictors, and ValueError during fit; filter None results cleanly - Propagate freq through worker config dict and ModelResultRef
EntsoeSource gains prefer_15min flag: when True the resolution priority is 15min > 30min > 60min; when False (default) it is 60min > 30min > 15min, preserving the previous behaviour. CalendarSource gains a freq parameter (default "1h"). At 15-min frequency the date-range is generated at 15-min intervals and _add_hour() maps each slot to a 0-95 period index (hour * 4 + minute // 15) instead of 0-23 hours.
Replace asfreq() + interpolate-only approach with a _METHOD_MAP that pairs an aggregation function (for downsampling) with a fill strategy (for upsampling). Add new methods: sum, first, last. The docstring now includes a clear direction/aggregation/fill table. Validation error message lists all valid method names from the map.
ENTSOE week-ahead (A31) data uses a fixed UTC+2 anchor for period starts. In winter (UTC+1 zones like CET) this places daily values at 23:00 local time instead of midnight, causing a subsequent ffill to cover the wrong calendar day. Fix: add 1h before normalizing so that 23:xx snaps to the next midnight while values already at 00:xx-02:xx snap to the current one. Also fix the mutable default-argument anti-pattern in __init__ by defaulting daily_columns to None and assigning the list inside the body. Update the docstring to describe the UTC+2 anchor behaviour.
CsvExporter: use tz_convert(None) instead of tz_localize(None) when stripping timezone from an already tz-aware DatetimeIndex, which would raise TypeError rather than silently drop the tz. ExcelExporter: rewrite details-sheet assembly to merge model DataFrames on sort keys instead of aligning by positional index. Handles models with different numbers of valid rows (e.g. after NaN-row filtering) and avoids column-order issues when multiple models are present. rMAEEvaluator: improve error message for empty model_dfs to tell the caller what kwarg is required, not which model is absent.
TimezoneTransformer: five new TestTimezoneTransformerDailySnap cases covering winter 23:xx → next midnight, summer midnight no-shift, winter 01:xx → current midnight, already-at-midnight, and non-daily-column passthrough. Models: three new tests — day IDs follow calendar days across the CET → CEST DST boundary, result rows correctly align actual values to target dates after DST, and a multi-horizon run spanning DST completes with finite predictions. EntsoeSource: fix Germany integration test to use DE_LU bidding zone. rMAEEvaluator: update match string to reflect the improved error message.
- Add prefer_15min and freq params to EntsoeSource and CalendarSource reference pages and llms.txt - Add Germany/DE_LU bidding zone caution to EntsoeSource docs - Replace ResampleTransformer interpolation-only method table with a full aggregation/fill table covering all six methods; fix duplicate "Selecting Columns" heading - Add freq parameter to ModelPipeline.run() reference in overview and workflow YAML example - Update llms.txt pipeline.run() snippet with freq and expand ResampleTransformer behaviour description
01: fix hardcoded API key (use ENTSOE_API_KEY env var), extend lag window, drop price from null validator, rename output file. 02–04: add RMSEEvaluator, rMAEEvaluator, CsvExporter; add missing TimezoneTransformer to example 04. 06 (new): end-to-end 15-min forecasting example for DE using prefer_15min=True, CalendarSource(freq="15min"), ResampleTransformer(freq="15min"), and ModelPipeline.run(freq="15min").
Add data_pipeline.yaml, model_pipeline.yaml, and experiment.yaml as example configuration files for the YAML-driven workflow (example 04). API key placeholder used — set ENTSOE_API_KEY or replace inline before running.
Allows fetching 3-hourly ("hourly_3"), 6-hourly ("hourly_6"), or
model-native resolution ("native") data in addition to the default 1 h.
When temporal_resolution is not None the parameter is forwarded to the
Open-Meteo API. For "native" the points-per-day value is inferred from
the delta between the first two returned timestamps instead of being
hardcoded to 24, so forecast index offsets stay correct regardless of
model step size. The value is included in the cache config key so
different resolutions cache independently.
There was a problem hiding this comment.
Pull request overview
Adds first-class support for sub-hourly (15-minute) forecasting runs across the data/model pipeline, while addressing DST/timezone edge cases that previously caused misalignment of “day” ids and target dates.
Changes:
- Add
freqsupport toModelPipeline.run()/BaseModel.run()with 15-minute period indexing (96/day) and propagate frequency into results. - Improve timezone/DST handling (calendar-day shifting for targets; midnight snapping for sparse daily ENTSOE A31 columns).
- Extend data sources/transformers and documentation for 15-minute workflows (ENTSOE
prefer_15min, CalendarSourcefreq, ResampleTransformer down/up sampling behaviors) and add a dedicated 15-min example.
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_rmse_rmae_csv.py | Update rMAE error-message expectation to match new evaluator behavior. |
| tests/models/test_models.py | Add regression tests for DST day-id continuity and target-date/actual alignment. |
| tests/data/transformers/test_transformers.py | Add tests for daily-column midnight snapping across winter/summer cases. |
| tests/data/sources/test_entsoe.py | Use DE_LU bidding zone code for German price tests. |
| pyproject.toml | Bump project version to 2.2.0. |
| model_pipeline.yaml | Add a sample model pipeline YAML config. |
| llms.txt | Document new options (freq, prefer_15min, temporal_resolution) and workflow examples. |
| experiment.yaml | Add a sample workflow YAML config. |
| examples/06_15min_forecasting.py | New end-to-end 15-minute workflow example. |
| examples/04_yaml_workflow.py | Update example workflow to include TimezoneTransformer and RMSE evaluator. |
| examples/03_full_workflow.py | Add RMSE evaluator and CSV exporter to the full workflow example. |
| examples/02_model_pipeline_only.py | Add RMSE/rMAE evaluators and CSV export to the model-only example. |
| examples/01_data_pipeline_only.py | Adjust example to load-only, expanded lags, and updated output naming/time span. |
| epftoolbox2/results/ref.py | Add freq to the model result reference metadata. |
| epftoolbox2/pipelines/model_pipeline.py | Thread freq through to model runs. |
| epftoolbox2/models/base.py | Implement frequency-aware loop sizing, DST-safe calendar-day shifts, and 15-min period indexing. |
| epftoolbox2/models/_worker.py | Make worker output optional (skip sparse/invalid slices) and include freq in rows. |
| epftoolbox2/exporters/excel.py | Rework details sheet building via merges to align predictions across models. |
| epftoolbox2/exporters/csv.py | Adjust tz handling for extra-column joining (see review comment). |
| epftoolbox2/evaluators/rmae.py | Improve error message when pipeline context (model_dfs) is missing. |
| epftoolbox2/data/transformers/timezone.py | Change daily snapping logic to round to nearest local midnight for ENTSOE A31. |
| epftoolbox2/data/transformers/resample.py | Extend resampling to handle both up/down sampling with method mapping. |
| epftoolbox2/data/sources/open_meteo.py | Add temporal_resolution support and caching keying for it. |
| epftoolbox2/data/sources/entsoe.py | Add prefer_15min to control price-resolution selection order. |
| epftoolbox2/data/sources/calendar.py | Add freq support and 15-min hour-of-day encoding. |
| docs/src/content/docs/workflow/overview.mdx | Document freq in workflow YAML and parameter table (currently not wired in code). |
| docs/src/content/docs/transformers/resample.mdx | Update transformer docs for new down/up sampling semantics and methods. |
| docs/src/content/docs/model-pipeline/overview.mdx | Document freq parameter on ModelPipeline.run(). |
| docs/src/content/docs/data-sources/openmeteo.mdx | Document temporal_resolution and usage examples. |
| docs/src/content/docs/data-sources/entsoe.mdx | Document prefer_15min and DE_LU note for German prices. |
| data_pipeline.yaml | Add a sample data pipeline YAML config including timezone + resampling. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Workflow: add model_freq parameter (default "1h") to __init__, serialize it in to_dict(), parse it in load(), and forward it to ModelPipeline.run() via freq=. Previously the freq key in YAML was silently ignored. CsvExporter: revert tz_convert(None) back to tz_localize(None) for stripping timezone from the source_data index. tz_convert(None) converts to UTC first, so a wall-clock hour like 18:00+02:00 becomes 16:00 naive — causing the target_date/hour join against model results (which use local wall-clock time) to silently produce wrong or missing rows. tz_localize(None) drops the tz label without shifting the time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.