Feat/pipeline improvements

[dawidlinek]

No description provided.

[Copilot]

Pull request overview

This PR adds significant pipeline functionality improvements to epftoolbox2, including data caching, validation framework, pipeline serialization, and a new calendar-based data source.

Key changes:

• Introduces a caching system for data sources with automatic gap detection and incremental fetching
• Adds a validation framework with three validators (ContinuityValidator, NullCheckValidator, EdaValidator)
• Implements pipeline serialization to/from YAML with component registry
• Adds CalendarSource for country-specific calendar features (holidays, weekdays, daylight hours)
• Centralizes logging configuration across the codebase
• Adds ResampleTransformer for time-series resampling

Reviewed changes

Copilot reviewed 22 out of 24 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
epftoolbox2/pipelines/data_pipeline.py	Core pipeline enhancements: adds validators, caching (both file and database-backed), serialization (save/load), timestamp parsing, and changes DataFrame merging from join to concat
epftoolbox2/logging.py	New centralized logging module using Rich for consistent formatting across components
epftoolbox2/data/cache_manager.py	New cache manager implementing hash-based caching with range tracking, gap detection, and incremental data storage
epftoolbox2/data/validators/*.py	New validation framework with base class, result container, and three concrete validators for data quality checks
epftoolbox2/data/sources/calendar.py	New calendar data source providing country-specific features like holidays, weekday encoding, and astronomical data
epftoolbox2/data/sources/base.py	Adds abstract get_cache_config method to DataSource interface
epftoolbox2/data/sources/{entsoe,open_meteo,csv}.py	Updates existing sources with get_cache_config implementations and migrates to centralized logging
epftoolbox2/data/transformers/resample.py	New transformer for resampling time-series data with multiple interpolation methods
epftoolbox2/data/transformers/init.py	Exports ResampleTransformer
tests/pipelines/test_pipeline.py	Adds tests for caching and serialization features
tests/data/validators/test_validators.py	Comprehensive tests for all three validators
tests/data/test_cache_manager.py	Tests for cache key generation, read/write operations, and gap detection
tests/data/sources/test_calendar.py	Tests for CalendarSource features and validation
requirements.txt, pyproject.toml	Adds pyyaml, holidays, and astral dependencies
.gitignore	Adds .cache/ directory and main.py to ignored files

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.