Skip to content

Commit 4541286

Browse files
daniel-rdttgiloncoroapre-commit-ci[bot]
authored
feat: introduce PEMMDB data and tyndp trajectories (#97)
* feat: add clean rule for pemmdb capacity and must-run data * feat: add placeholder for tyndp trajectories * feat: add retrieve rule for tyndp pemmdb data * fix: add safe pyear to clean_tyndp_hydro_inflows.py * fix: improve input_pemmdb_data handling for disabled feature * fix: typo in clean hydro inflows param * feat: add handling of must-runs for DE and GA after 2030 and small fixes * feat: add capacities for other non-RES * feat: refactor workflow to split into clean rules for each technology and build rule to collect the data in one file per planning year * feat: add pemmdb profiles that combine p_min_pu and p_max_pu profiles * feat: refactor clean_pemmdb scripts into modular functions for improved readability and improve error handling * feat: generalise tyndp data retrieval (cherry picked from commit ce66349d5dca6bb5328a011190a939695fa9db88) * doc: add release note * fix: resolve to support relative path * fix: resolve to support relative path * fix: keep rootpath * Revert "fix: keep rootpath" This reverts commit 0cd9505. * refactor: remove unused rootpath * feat: add processing of renewable capacities * feat: minor improvements to processing of renewable capacities * feat: include RES PEMMDB capacities and profiles in workflow * feat: add Other RES PEMMDB capacities and profiles * feat: add pemmdb subfolder in resources * feat: add collect rule for pemmdb_datas * feat: include processing of electrolyser data * fix: increase latency wait to 10s to avoid missing directory in CI * feat: include processing of battery capacities, update configs and fix type hints * feat: include processing of DSR capacities and profiles * feat: process and clean tyndp trajectories * feat: add tyndp trajectories into workflow * feat: separate e_nom information to separate column * fix: add missing trajectories file to retrieve bundle rule output * feat: add mapping from PEMMDB carriers to a tyndp carrier name and index carrier * feat: ignore index when saving csv files * feat: ignore index when saving csv files * fix: bunch of minor fixes * feat: use tyndp_technology_map.csv for PECD carrier mapping * feat: remove spaces from pemmdb technology wildcards * fix: add fix for no h2 interzonal connections in NT+ scenario * feat: map carrier names for investment dataset trajectory file * ci: introduce multiple retries to hydro and pemmdb clean rules after retrieve * feat: process trajectories to only include values for DE and GA for after 2030 apart from Nuclear * fix for removed spaces in technology wildcards * fix PV carrier mapping * feat: refactor for consolidated reading and processing of pemmdb data in one rule (#116) * feat: refactor to read full set of pemmdb data and consolidate processing of different technologies in one rule * doc: add new nprocesses parameter for pemmdb data reading to documentation * fix: correct processing of different price bands for Other Non-RES data * fix: improve performance of profile merging by using pandas instead of xarray * fix: improve performance of reading by only reading required sheets * fix: use config_provider for threads definition * use vectorized function instead of loop * move convert_units to _helpers * fix: fix for thermal profiles DE and GA after 2030 * doc: add release note * doc: update feature's development status in documentation and readme * feat: remove outdated clean pemmdb capacities and profiles scripts * feat: addition to use tyndp_technology_map.csv for PECD carrier mapping * doc: update electricity.csv * fix: minor fixes from code review * fix: remove unnecessary lambda * apply code suggestion Co-authored-by: Jonas Hörsch <coroa@posteo.de> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: adjust use of convert_units to new function parameters * feat: reduce pemmdb inputs in prepare_sector_network to available pemmdb years * sort index in thermal profiles for more efficient indexing * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply suggestions from code review Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * doc: change description of pemmdb feature to clarify processing * feat: specify 16 GB resources for build_pemmdb_data rule * fix: fix missing wildcard argument to input_pemmdb_data * doc: add missing config options to electricity.csv * feat: include trajectories for DE and GA for 2030 * reuse: add tyndp_technology_map.csv to REUSE.toml * doc: update retrieve rule documentation * feat: include hydrogen in tyndp pemmdb mapping * feat: update typehints for safe pyear * doc: update docstring of build_tyndp_trajectories.py * feat: improve map_tyndp_carrier_names and introduce drop_merge_columns parameter * feat: improve robustness of snapshot filtering * feat: update pemmdb sheet mapping * doc: add missing rules to preparation.rst * fix: rename UK to GB in build_tyndp_trajectories.py * feat: consolidate pemmdb datas and trajectories in one collect rule * doc: include todo into PR template to document new rules * feat: minor improvement in parameter naming * doc: use consistent imperative style in docstrings as per PEP 257 * feat: use pattern matching to extract type and unit information from RES index * feat: various minor improvements and bug fixes to build_pemmdb_data.py * feat: add validate profile to filter out profiles with default PyPSA values to significantly reduce netcdf file * feat: rename index_year variable * feat: bundle processing of thermal technologies * doc: update output format description for pemmdb profiles * feat: improve on reduce pemmdb inputs in prepare_sector_network to available pemmdb years * Apply suggestions from code review Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: move and rename helper functions in build_pemmdb_data * ci: set tyndp tests as firsts * fix: correct use of drop_on_columns * log: improve verbosity of logger by setting logger levels appropriately * feat: minor improvements from code suggestion * apply code suggestion * feat: improve pemmdb collect rule --------- Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org> Co-authored-by: Jonas Hörsch <coroa@posteo.de> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent e411c81 commit 4541286

24 files changed

Lines changed: 1897 additions & 88 deletions

‎.github/pull_request_template.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,5 +27,6 @@ Closes # (if applicable).
2727
- [ ] Changes in configuration options are added in `config/test/*.yaml`.
2828
- [ ] Open-TYNDP SPDX license header added to all touched files.
2929
- [ ] Sources of newly added data are documented in `doc/data_sources.rst`.
30+
- [ ] New rules are documented in the appropriate `doc/*.rst` files.
3031
- [ ] A release note `doc/release_notes.rst` is added.
3132
- [ ] Major features are documented with up-to-date information in `README` and `doc/index.rst`.

‎Makefile‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -79,29 +79,29 @@ install-pinned-macos: _conda_check
7979
# Run default tests
8080
test:
8181
set -e
82+
snakemake -call --configfile config/test/config.tyndp.yaml
83+
snakemake -call --configfile config/config.tyndp.yaml -n
8284
snakemake -call solve_elec_networks --configfile config/test/config.electricity.yaml
8385
snakemake -call --configfile config/test/config.overnight.yaml
8486
snakemake -call --configfile config/test/config.myopic.yaml
8587
snakemake -call make_summary_perfect --configfile config/test/config.perfect.yaml
8688
snakemake -call resources/test-elec-clusters/networks/base_s_adm.nc --configfile config/test/config.clusters.yaml
8789
snakemake -call --configfile config/test/config.scenarios.yaml -n
88-
snakemake -call --configfile config/test/config.tyndp.yaml
89-
snakemake -call --configfile config/config.tyndp.yaml -n
9090
echo "All tests completed successfully."
9191

9292
unit-test:
9393
pytest test
9494

9595
# Cleans all output files from tests
9696
clean-tests:
97+
snakemake -call --configfile config/test/config.tyndp.yaml --delete-all-output
98+
snakemake -call --configfile config/config.tyndp.yaml -n --delete-all-output
9799
snakemake -call solve_elec_networks --configfile config/test/config.electricity.yaml --delete-all-output
98100
snakemake -call --configfile config/test/config.overnight.yaml --delete-all-output
99101
snakemake -call --configfile config/test/config.myopic.yaml --delete-all-output
100102
snakemake -call make_summary_perfect --configfile config/test/config.perfect.yaml --delete-all-output
101103
snakemake -call resources/test-elec-clusters/networks/base_s_adm.nc --configfile config/test/config.clusters.yaml --delete-all-output
102104
snakemake -call --configfile config/test/config.scenarios.yaml -n --delete-all-output
103-
snakemake -call --configfile config/test/config.tyndp.yaml --delete-all-output
104-
snakemake -call --configfile config/config.tyndp.yaml -n --delete-all-output
105105

106106
# Removes all created files except for large cutout files (similar to fresh clone)
107107
reset:

‎README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ The back-casting of the 2024 TYNDP cycle involves developing new features based
4545
- Added processing and preparation of TYNDP 2024 PECD v3.1 renewable profiles for offshore wind farms, replacing default ERA5-based profiles processed with Atlite (see PR [#53](https://github.com/open-energy-transition/open-tyndp/pull/53)).
4646
- Introduced TYNDP offshore wind hubs topology with both electric and hydrogen infrastructure, offshore electrolysers, and detailed wind farm characteristics (see PR [#54](https://github.com/open-energy-transition/open-tyndp/pull/54)).
4747
- Introduce a benchmarking framework that assesses Open-TYNDP quality against published TYNDP 2024 data (see PR [#73](https://github.com/open-energy-transition/open-tyndp/pull/73)) and Visualisation Platform data (see PR [#117](https://github.com/open-energy-transition/open-tyndp/pull/117)).
48+
- Added processing and preparation of PEMMDB v2.4 capacity, must-run, and availability data, along with expansion trajectories for conventional and renewable power generation, electrolysers, batteries, and DSR (see PR [#97](https://github.com/open-energy-transition/open-tyndp/pull/97)).
4849

4950
| Feature | TYNDP 2024 topology | Open-TYNDP topology |
5051
|:--------------------:|:-------------------:|:-------------------:|
@@ -60,16 +61,16 @@ While multiple TYNDP features are already introduced to the Open-TYNDP model, th
6061
|:------------------------------------------------------:|:---------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------:|:------:|
6162
| **Visualizations and workflow automation** | Automated workflow | | ✅ |
6263
| | TYNDP plotting routines | | ⌛ |
63-
| **Automated tests and benchmarks** | Automated benchmarking routine | [#73](https://github.com/open-energy-transition/open-tyndp/pull/73), [#117](https://github.com/open-energy-transition/open-tyndp/pull/117) | ✅ |
64+
| **Automated tests and benchmarks** | Automated benchmarking routine | [#73](https://github.com/open-energy-transition/open-tyndp/pull/73), [#117](https://github.com/open-energy-transition/open-tyndp/pull/117) | ✅ |
6465
| **TYNDP modelling features** | Perfect foresight optimization | | ⌛ |
6566
| | Security of Supply (SoS) loop | | ⌛ |
6667
| **Existing infrastructure and associated parameters** | Electricity reference grid | [#18](https://github.com/open-energy-transition/open-tyndp/pull/18) | ✅ |
6768
| | Hydrogen reference grid | [#17](https://github.com/open-energy-transition/open-tyndp/pull/17), [#36](https://github.com/open-energy-transition/open-tyndp/pull/36) | ✅ |
6869
| | Offshore grid | [#54](https://github.com/open-energy-transition/open-tyndp/pull/54) | ✅ |
6970
| | PECD data | [#53](https://github.com/open-energy-transition/open-tyndp/pull/53), [#71](https://github.com/open-energy-transition/open-tyndp/pull/71) | ✅ |
7071
| | Hydro inflows | [#77](https://github.com/open-energy-transition/open-tyndp/pull/77) | ✅ |
71-
| | PEMMDB capacities & must-runs | [#97](https://github.com/open-energy-transition/open-tyndp/pull/97) | 🔨 |
72-
| | Investment candidates trajectories | [#97](https://github.com/open-energy-transition/open-tyndp/pull/97) | 🔨 |
72+
| | PEMMDB capacities & must-runs processing | [#97](https://github.com/open-energy-transition/open-tyndp/pull/97) | ✅ |
73+
| | Investment candidates trajectories processing | [#97](https://github.com/open-energy-transition/open-tyndp/pull/97) | ✅ |
7374
| **TYNDP demand** | Electricity | [#14](https://github.com/open-energy-transition/open-tyndp/pull/14) | ✅ |
7475
| | Hydrogen | | ⌛ |
7576
| | Methane | | ⌛ |

‎REUSE.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ SPDX-License-Identifier = "CC-BY-4.0"
2121
[[annotations]]
2222
path = [
2323
"data/tyndp_electricity_loss_factors.csv",
24+
"data/tyndp_technology_map.csv",
2425
"doc/configtables/benchmarking.csv",
2526
"doc/img/tyndp/*",
2627
]

‎config/config.default.yaml‎

Lines changed: 30 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -141,8 +141,10 @@ electricity:
141141
everywhere_powerplants: []
142142

143143
conventional_carriers: [nuclear, oil, OCGT, CCGT, coal, lignite, geothermal, biomass]
144+
tyndp_conventional_carriers: []
144145
renewable_carriers: [solar, solar-hsat, onwind, offwind-ac, offwind-dc, offwind-float, hydro]
145146
tyndp_renewable_carriers: []
147+
tyndp_stores: []
146148

147149
pecd_renewable_profiles:
148150
enable: false
@@ -160,15 +162,10 @@ electricity:
160162
- 2040
161163
- 2050
162164
technologies:
163-
Wind_Offshore:
164-
- offwind-ac-fb-r
165-
- offwind-ac-fl-r
166-
- offwind-dc-fb-r
167-
- offwind-dc-fl-r
168-
- offwind-dc-fb-oh
169-
- offwind-dc-fl-oh
170-
- offwind-h2-fb-oh
171-
- offwind-h2-fl-oh
165+
- Wind_Offshore
166+
- LFSolarPVRooftop
167+
- LFSolarPVUtility
168+
- Wind_Onshore
172169

173170
pemmdb_hydro_profiles:
174171
enable: false
@@ -183,6 +180,30 @@ electricity:
183180
- PS Open
184181
- PS Closed
185182

183+
pemmdb_capacities:
184+
enable: false
185+
nprocesses: 4
186+
available_years:
187+
- 2030
188+
- 2040
189+
- 2050
190+
technologies:
191+
- Solar
192+
- Wind
193+
- Hydro
194+
- Other_RES
195+
- Gas
196+
- Nuclear
197+
- Hard_coal
198+
- Lignite
199+
- Light_oil
200+
- Heavy_oil
201+
- Oil_shale
202+
- Other_Non-RES
203+
- Hydrogen
204+
- Electrolyser
205+
- Battery
206+
186207
estimate_renewable_capacities:
187208
enable: true
188209
from_gem: true

‎config/config.tyndp.yaml‎

Lines changed: 61 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,37 @@ electricity:
4242
base_network: tyndp
4343

4444
# TODO: remove redundant onwind and solar from renewable_carriers list once technologies are included as tyndp_renewable_carriers from PEMMDB data
45+
tyndp_conventional_carriers:
46+
- gas
47+
- coal
48+
- lignite
49+
- nuclear
50+
- oil-light
51+
- oil-heavy
52+
- oil-shale
4553
renewable_carriers: [hydro, solar, solar-hsat, onwind]
46-
tyndp_renewable_carriers: [hydro, onwind, solar-pv-utility, solar-pv-rooftop, offwind-ac-fb-r, offwind-ac-fl-r, offwind-dc-fb-r, offwind-dc-fl-r, offwind-dc-fb-oh, offwind-dc-fl-oh, offwind-h2-fb-oh, offwind-h2-fl-oh]
54+
tyndp_renewable_carriers:
55+
- other-res
56+
- hydro-ror
57+
- hydro-reservoir
58+
- hydro-pondage
59+
- hydro-phs
60+
- hydro-phs-pure
61+
- onwind
62+
- solar-pv-utility
63+
- solar-pv-rooftop
64+
- offwind-ac-fb-r
65+
- offwind-ac-fl-r
66+
- offwind-dc-fb-r
67+
- offwind-dc-fl-r
68+
- offwind-dc-fb-oh
69+
- offwind-dc-fl-oh
70+
- offwind-h2-fb-oh
71+
- offwind-h2-fl-oh
72+
tyndp_stores:
73+
- battery
74+
- h2_cavern
75+
- h2_tank
4776

4877
pecd_renewable_profiles:
4978
enable: true
@@ -62,21 +91,10 @@ electricity:
6291
- 2030
6392
- 2040
6493
technologies:
65-
Wind_Offshore:
66-
- offwind-ac-fb-r
67-
- offwind-ac-fl-r
68-
- offwind-dc-fb-r
69-
- offwind-dc-fl-r
70-
- offwind-dc-fb-oh
71-
- offwind-dc-fl-oh
72-
- offwind-h2-fb-oh
73-
- offwind-h2-fl-oh
74-
LFSolarPVRooftop:
75-
- solar-pv-rooftop
76-
LFSolarPVUtility:
77-
- solar-pv-utility
78-
Wind_Onshore:
79-
- onwind
94+
- Wind_Offshore
95+
- LFSolarPVRooftop
96+
- LFSolarPVUtility
97+
- Wind_Onshore
8098

8199
pemmdb_hydro_profiles:
82100
enable: true
@@ -85,11 +103,35 @@ electricity:
85103
- 2040
86104
- 2050
87105
technologies:
88-
- Run of River
106+
- Run_of_River
89107
- Pondage
90108
- Reservoir
91-
- PS Open
92-
- PS Closed
109+
- PS_Open
110+
- PS_Closed
111+
112+
pemmdb_capacities:
113+
enable: true
114+
nprocesses: 4
115+
available_years:
116+
- 2030
117+
- 2040
118+
- 2050
119+
technologies:
120+
- Solar
121+
- Wind
122+
- Hydro
123+
- Other_RES
124+
- Gas
125+
- Nuclear
126+
- Hard_coal
127+
- Lignite
128+
- Light_oil
129+
- Heavy_oil
130+
- Oil_shale
131+
- Other_Non-RES
132+
- Hydrogen
133+
- Electrolyser
134+
- Battery
93135

94136
estimate_renewable_capacities:
95137
# NOTE: technologies that are covered by TYNDP renewable carriers need to be removed from estimation

‎config/test/config.tyndp.yaml‎

Lines changed: 61 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -49,8 +49,37 @@ electricity:
4949
Link: [H2 pipeline]
5050

5151
# TODO: remove redundant onwind and solar from renewable_carriers list once technologies are included as tyndp_renewable_carriers from PEMMDB data
52+
tyndp_conventional_carriers:
53+
- gas
54+
- coal
55+
- lignite
56+
- nuclear
57+
- oil-light
58+
- oil-heavy
59+
- oil-shale
5260
renewable_carriers: [solar, solar-hsat, onwind]
53-
tyndp_renewable_carriers: [hydro, onwind, solar-pv-utility, solar-pv-rooftop, offwind-ac-fb-r, offwind-ac-fl-r, offwind-dc-fb-r, offwind-dc-fl-r, offwind-dc-fb-oh, offwind-dc-fl-oh, offwind-h2-fb-oh, offwind-h2-fl-oh]
61+
tyndp_renewable_carriers:
62+
- other-res
63+
- hydro-ror
64+
- hydro-reservoir
65+
- hydro-pondage
66+
- hydro-phs
67+
- hydro-phs-pure
68+
- onwind
69+
- solar-pv-utility
70+
- solar-pv-rooftop
71+
- offwind-ac-fb-r
72+
- offwind-ac-fl-r
73+
- offwind-dc-fb-r
74+
- offwind-dc-fl-r
75+
- offwind-dc-fb-oh
76+
- offwind-dc-fl-oh
77+
- offwind-h2-fb-oh
78+
- offwind-h2-fl-oh
79+
tyndp_stores:
80+
- battery
81+
- h2_cavern
82+
- h2_tank
5483

5584
pecd_renewable_profiles:
5685
enable: true
@@ -69,21 +98,10 @@ electricity:
6998
- 2030
7099
- 2040
71100
technologies:
72-
Wind_Offshore:
73-
- offwind-ac-fb-r
74-
- offwind-ac-fl-r
75-
- offwind-dc-fb-r
76-
- offwind-dc-fl-r
77-
- offwind-dc-fb-oh
78-
- offwind-dc-fl-oh
79-
- offwind-h2-fb-oh
80-
- offwind-h2-fl-oh
81-
LFSolarPVRooftop:
82-
- solar-pv-rooftop
83-
LFSolarPVUtility:
84-
- solar-pv-utility
85-
Wind_Onshore:
86-
- onwind
101+
- Wind_Offshore
102+
- LFSolarPVRooftop
103+
- LFSolarPVUtility
104+
- Wind_Onshore
87105

88106
pemmdb_hydro_profiles:
89107
enable: true
@@ -92,11 +110,35 @@ electricity:
92110
- 2040
93111
- 2050
94112
technologies:
95-
- Run of River
113+
- Run_of_River
96114
- Pondage
97115
- Reservoir
98-
- PS Open
99-
- PS Closed
116+
- PS_Open
117+
- PS_Closed
118+
119+
pemmdb_capacities:
120+
enable: true
121+
nprocesses: 4
122+
available_years:
123+
- 2030
124+
- 2040
125+
- 2050
126+
technologies:
127+
- Solar
128+
- Wind
129+
- Hydro
130+
- Other_RES
131+
- Gas
132+
- Nuclear
133+
- Hard_coal
134+
- Lignite
135+
- Light_oil
136+
- Heavy_oil
137+
- Oil_shale
138+
- Other_Non-RES
139+
- Hydrogen
140+
- Electrolyser
141+
- Battery
100142

101143
estimate_renewable_capacities:
102144
# NOTE: technologies that are covered by TYNDP renewable carriers need to be removed from estimation

0 commit comments

Comments
 (0)