feat: introduce PEMMDB data and tyndp trajectories

[daniel-rdt]

Closes #128.

Changes proposed in this Pull Request

This PR introduces the processing of PEMMDb v2.4 data and TYNDP trajectories for power plants. The final processed output of the newly added rules will be three files:

• resources/tyndp_trajectories.csv in long format including information on trajectories (p_nom_min and p_nom_max) for RES technologies, nuclear power plants, electrolyzers and batteries.

• resources/pemdb_capacties_{planning_horizon}.csv in long format including information on p_nom, e_nom(optional), efficiency, carrier, index_carrier, bus, country, unit

• resources/pemmdb_profiles_{planning_horizon}.nc with the following structure:

  Field	Dimensions	Description
  profile	time, bus, carrier, index_carrier	the per unit hourly availability (p_max_pu) and must-run obligations (p_min_pu) for each bus and technology

As documented in the 2024 TYNDP methodology report, p.13, the data collected within the PEMMDB used for the Scenario Building includes information on NT capacities, must-run obligations and availabilities for power plants. The following information is covered within the dataset:

• Renewable energy sources (NT capacities and minimum and maximum
  trajectories for deviation scenarios)
• Thermal unit capacity & must-runs (p_min_pu)
  • no must runs for DE and GA after 2030
• Other Non-RES / CHPs (Other Non-RES are assumed to represent CHP plants according to Methodology report p.37) with price bands, capacity, efficiency, emission factor and max available capacity (p_max_pu)
  • "For Other Non-RES, minimum supply at zero cost has been kept capturing CHP operation not directly linked to the wholesale market price" (Methodology report, p.37)
• Nuclear unit capacity & must-runs
• Electrolyser (connected to electricity grid)
• Battery capacities (prosumer and market participating)
• Demand-side response (capacity & activation price bands)

Note: Similar to #71 and #77 , this PR serves to facilitate a sub-workflow for extraction and processing of PEMMDB data but does not yet attach them to any technologies yet. This will be added in a follow up PR.

Additionally required steps are:

• Inclusion of TYNDP generation technologies using PEMMDB capacities, must-runs, availabilities and trajectories ([PARENT] Inclusion of existing infrastructure from PEMMDB & investment candidates #64)

Tasks

• Set up architecture to build PEMMDB data
• include base function to process capacity data
• include base function to process must-run profiles
• Cover conventionals
• Cover Hydrogen reconversion
• Cover nuclear units
• Include must-run for thermal units as zero for DE and GA after 2030 (keep Other Non-RES minimum supply at zero cost to capture CHP bands operation not directly linked to the wholesale market price)
• Extract p_max_pu time series for CHP (Other Non-RES)
• Cover RES
• Cover Other RES
• Cover Electrolysers
• Cover Batteries
• Cover DSR
• Set up architecture to build TYNDP trajectories
• Map PEMMDB carriers and types to TYNDP carrier names
• Use technology mapping for PECD rules
• Fix TYNDP trajectories: Not for NT+ scenario and for DE and GA only > 2030 (except Nuclear)
• Refactor PEMMDB workflow for more efficient file loading
• Add new rules to documentation
• Add release note
• Add new feature to readme and documentation
• Update configuration `electricity.csv' for new configuration
• Investigate large netcdf file size for pemmcb_profiles.nc files
• Document open issues for transparency report

Workflow

The introduced workflow data pipeline for the PEMMDB capacity and must-run data includes the following steps:

• (new) retrieve_pemmdb_data: Rule to retrieve the 2024 TYNDP PEMMDB data and save in the tyndp_2024_bundle directory
• (new) build_pemmdb_data: Reads, processes and merges the available PEMMDB v2.4 capacities and profiles for different PEMMDB technologies from TYNDP data bundle for a given planning horizon. Returns a cleaned csv file with all NT capacities (p_nom) in long format and netcdf file with must run obligations (p_min_pu) and availability (p_max_pu) for all different PEMMDB technologies.
• (new) build_tyndp_trajectories: Rule to load and clean the 2024 TYNDP capacity trajectories from the investment dataset. Returns a csv file with trajectories for all available technologies and planning_horizons.
• (new) build_tyndp_datas: Collect rule to produce processed capacities and profiles for all specified planning_horizons.

Example workflow for three planning years (2030, 2035, 2040, 2045, 2050) and various different PEMMDB technologies:

Open issues

• It is not entirely clear how the "Hours" attribute of the DSR price bands is used. How do "Hours" specify DSR flexibility in the TYNDP?
• Are non-ENTSO-E exchanges and reserves from PEMMDB needed? We assume that not as it is not listed in the methodology report p.37)
• Are the buildout constraints in the BUILDOUTsheet of the TRAJECTORIES.xlsx used for the modelling and if so how?

Notes

Checklist

• I tested my contribution locally and it works as intended.
• Code and workflow changes are sufficiently documented.
• Changes in configuration options are added in config/config.default.yaml.
• Changes in configuration options are documented in doc/configtables/*.csv.
• Changes in configuration options are added in config/test/*.yaml.
• OET license identifier is added to all edited or newly created code files.
• Sources of newly added data are documented in doc/data_sources.rst.
• A release note doc/release_notes.rst is added.
• Major features are listed in README and doc/index.rst.

[daniel-rdt]

@tgilon I've addressed all of your comments now which included improvements on the size of the PEMMDB profiles netcdf file and efficiency gains in processing thermal capacities and profiles. Once the CI has passed, you can have another look at the changes.

[tgilon]

Thank you @daniel-rdt for all the improvements! They have certainly improved the code. I do have a few follow up comments, though. However, nothing major and the PR is in good shape to be merged soon.

[tgilon]

Looks good! No additional comments on my side. Thanks @daniel-rdt

[tgilon]

LGTM