Skip to content

feat: process and prepare PECD capacity factor time series for offshore generators - #53

Merged
daniel-rdt merged 55 commits into
masterfrom
feat/49-pecd-offshore
Jul 3, 2025
Merged

feat: process and prepare PECD capacity factor time series for offshore generators#53
daniel-rdt merged 55 commits into
masterfrom
feat/49-pecd-offshore

Conversation

@daniel-rdt

@daniel-rdt daniel-rdt commented May 27, 2025

Copy link
Copy Markdown
Member

Closes #49. This PR is necessary for addressing #48.

Changes proposed in this Pull Request

This PR prepares the workflow to use PECD v3.1 data as input for renewable capacity factor time series of offshore generators. TYNDP generators are differentiated by their foundation type and offshore section type. This is described by their suffixes as described in the methodology report on p.47:

  • fb = "fixed-bottom"
  • fl = "floating"
  • r = "radial"
  • oh = "offshore-hub"

All TYNDP Offshore generator technologies use the same PECD profile Offshore_Wind.

The openly published TYNDP 2024 input data from PECD includes renewable time series for the years, 2030, 2040 and 2050.

To reproduce and test the implementation of this PR, you can run the 'Make' Rule:

make tyndp

To produce a dag of the adapted workflow of this PR with PECD profiles instead of default atlite renewable profiles as specified in the configuration file, you can call:

snakemake solve_sector_networks --configfile config/config.tyndp.yaml --dag | dot -Tsvg > dag.svg

The PECD renewable profiles output files resources/profile_pecd_{clusters}_{technology}.nc have the following structure:

===================  ====================  =========================================================
Field                Dimensions            Description
===================  ====================  =========================================================
profile              year, bus, bin, time  the per unit hourly availability factors for each bus
===================  ====================  =========================================================

Tasks

  • Prepare workflow by skipping default renewables profiles in the workflow
  • Add rule to clean PECD data
  • Add rule to process cleaned PECD data into one renewable profile xarray Dataset
  • validate the workflow for offshore input data
  • Add retrieve of PECD data from gdrive until data is validated and added to Zenodo data bundle
  • Add documentation to the configuration and rules doc files
  • add release note
  • add 2050 in PECD renewable profiles explicitly
  • early filter for snapshots when cleaning data
  • differentiate renewable carrier names for tyndp from renewable profiles
  • Add a note in the config to address config conflicts
  • Clean naming convention of PECD / TYNDP carriers
  • Split up renewable carriers into two separate lists

Workflow

  1. (new) retrieve rule: retrieve_tyndp_pecd_data which downloads the TYNDP PECD data from Google Drive and extracts it in the data/tyndp_2024_bundle subdirectory. This rule will be deprecated once the data bundle has been updated ([PARENT] Update TYNDP 2024 data bundle on Zenodo #87).

  2. (new) configuration option electricity/tyndp_renewable_profiles where one can choose whether to use PECD renewable profiles and for which technologies. There on can specify both the PECD technology and the TYNDP renewable carriers associated to that profile. This two layer approach serves helpful to reduce the amount of processing required as multiple TYNDP renewable carriers use the same PECD profile (e.g. all offshore wind technologies).
    The PECD technology can be any of {Wind_Offshore, Wind_Onshore, LFSolarPV}.
    The tyndp_renewable_carriers are listed seperately to the already existing renewable_carriers list. All tyndp_renewable_carriers listed must also be mapped to a PECD technology and must not be included in estimate_renewable_capacities technologies.

  3. Adapted inputs to rules add_electricity, prepare_sector_network and add_brownfield based on this new setting

  4. (new) rule and script clean_pecd_data

    • Loads and cleans the available PECD capacity factor generation time series based on PECD weather data.
    • The rule is executed for a given technology, and planning horizon.
    • Data is available for climate years between 1982-2019.
    • For intermediate planning_years, fallback is to the previous available year.
  5. (new) rule and script build_renewable_profiles_pecd

    • Creates renewable profiles for each region from PECD and TYNDP input data containing the available generation time series (based on PECD weather data) from the node for onshore wind, AC-connected offshore wind, DC-connected offshore wind and solar PV generators.
    • The rule is executed for a given technology and combines all planning_horizon years into one xarray Dataset for this technology
    • For planning_year 2050, fallback is to 2040 because of missing data for 2050.
  6. Hotfix: In add_existing_baseyear and add_brownfield skipped addition of existing renewables and p_max_pu respectively for specified technologies that use pecd data until their TYNDP generator components are added to the system

Open Issues

  • The openly published PECD input data from the TYNDP 2024 downloads page does not cover all offshore buses for the year 2040 and 2050.
  • While for 2030 the available PECD offshore zones match with those 45 zones where there is actually any zonal potential specified in the offshore generators input file, for 2040 data is only available for the same 45 offshore zones that are also available for 2030.
  • For 2050, data for only very few Offshore zones are available in the input dataset (namely: BEIOH01, BEOH001, DEOH001). Therefore, for planning_horizon 2050, fallback is the PECD data from 2040.

Notes

The updated workflow is described by the following simplified dag showcasing all the rules between the clean_pecd_data rules and the solve rules when calling
snakemake solve_sector_networks --configfile config/config.tyndp.yaml

image

Checklist

  • I tested my contribution locally and it works as intended.
  • Code and workflow changes are sufficiently documented.
  • Changed dependencies are added to envs/environment.yaml.
  • Changes in configuration options are added in config/config.default.yaml.
  • Changes in configuration options are documented in doc/configtables/*.csv.
  • Changes in configuration options are added in config/test/*.yaml.
  • Sources of newly added data are documented in doc/data_sources.rst.
  • A release note doc/release_notes.rst is added.

@daniel-rdt
daniel-rdt requested a review from tgilon May 27, 2025 13:57
@tgilon tgilon added this to the Release v0.2 milestone May 28, 2025

@tgilon tgilon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just wanted to share a few thoughts from our last chat. The main request is for a definition of TYNDP specific renewables carriers.

Comment thread rules/build_electricity.smk
Comment thread config/config.default.yaml Outdated
Comment thread scripts/clean_pecd_data.py Outdated
Comment thread scripts/clean_pecd_data.py
Comment thread scripts/build_renewable_profiles_pecd.py

@tgilon tgilon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @daniel-rdt, for the updates. I'm happy with the current set of configuration levels. This is a good balance between redundancy and user-friendliness. I have a few minor comments below.

Comment thread doc/configtables/electricity.csv Outdated
Comment thread doc/configtables/electricity.csv Outdated
Comment thread scripts/add_electricity.py Outdated
Comment thread scripts/add_electricity.py Outdated
Comment thread config/config.tyndp.yaml Outdated
Comment thread config/test/config.tyndp.yaml Outdated
@daniel-rdt
daniel-rdt requested a review from tgilon July 1, 2025 11:57

@tgilon tgilon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have much more to say about this PR. Looks (almost) ready to be merged!

Edit: A conflict with master needs to be resolved.

Comment thread doc/configtables/electricity.csv Outdated
Comment thread scripts/retrieve_tyndp_pecd_data.py
Comment thread doc/configtables/electricity.csv Outdated

@coroa coroa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm .. looks good.

The key thing i am wondering is more about where all the other characteristics of the tyndp renewable carriers are being defined. Think capacities, costs, ..., and whether profile then should not be a setting of that set of configuration items?

I am also wondering whether you can come up with a new name for the Tyndp technology name, ie. pecd_tech or tyndp_tech and then use that consistently to as variable names to differentiate between carrier and carrier :). It looks like in the config snippet suggestion you are using pecd_carrier for that.

Comment thread rules/build_sector.smk Outdated
Comment thread scripts/add_electricity.py
Comment thread scripts/add_electricity.py
@coroa

coroa commented Jul 3, 2025

Copy link
Copy Markdown
Member

I just now quickly tried the PR from a clean slate.

I am getting a

MissingInputException: Missing input files for rule clean_pecd_data:
    output: resources/tyndp/pecd_data_Wind_Offshore_2040.csv
    wildcards: technology=Wind_Offshore, planning_horizons=2040
    affected files:
        data/tyndp_2024_bundle/Offshore hubs/NODE.xlsx

This is understandable, because the whole tyndp bundle is only there after extraction. So this would have to be added to the output of retrieve_tyndp_bundle

@daniel-rdt

Copy link
Copy Markdown
Member Author

I just now quickly tried the PR from a clean slate.

I am getting a

MissingInputException: Missing input files for rule clean_pecd_data:
    output: resources/tyndp/pecd_data_Wind_Offshore_2040.csv
    wildcards: technology=Wind_Offshore, planning_horizons=2040
    affected files:
        data/tyndp_2024_bundle/Offshore hubs/NODE.xlsx

This is understandable, because the whole tyndp bundle is only there after extraction. So this would have to be added to the output of retrieve_tyndp_bundle

Good catch! Thanks @coroa. Curious that it most of the time ran through in the CIs still. Sometimes the retrieve timing works naturally and then there is no missing input, maybe the download speed is quicker on for the CI. Good to have the dependency fixed within Snakemake.

@daniel-rdt
daniel-rdt merged commit 2f13208 into master Jul 3, 2025
4 checks passed
@daniel-rdt
daniel-rdt deleted the feat/49-pecd-offshore branch July 3, 2025 14:22
@daniel-rdt daniel-rdt mentioned this pull request Jul 14, 2025
21 tasks
@tgilon tgilon added the major feature Major feature for the Open TYNDP. label Jul 22, 2025
daniel-rdt added a commit that referenced this pull request Sep 8, 2025
daniel-rdt added a commit that referenced this pull request Sep 8, 2025
…tus (#99)

* doc: add more prominent disclaimer to README.md about development status

* feat: add disclaimer to index file and include reference to dev status

* feat: update limitations section to be specific to TYNDP and Open-TYNDP

* feat: introduce overview of already implemented features and features in development

* fix: fix see also in readme

* fix: fix see also in readme (again)

* feat: update feature table with PR links

* feat: update feature table with separate PR column and new pending symbol

* Apply suggestions from code review

Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org>

* fix: move PR #53 to pecd row in dev status table

* fix: improve consistency in link to github issues

* fix: remove emojis from readthedocs headers

---------

Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

major feature Major feature for the Open TYNDP.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PECD Offshore capacity factors

3 participants