feat: PEMMDB v2.4 brownfield integration — Phase 1 (thermal conventionals) - #86
Closed
pworschischek-aggmag wants to merge 0 commit into
Closed
feat: PEMMDB v2.4 brownfield integration — Phase 1 (thermal conventionals)#86pworschischek-aggmag wants to merge 0 commit into
pworschischek-aggmag wants to merge 0 commit into
Conversation
Contributor
Reviewer's GuideImplements Phase 1 of PEMMDB v2.4 brownfield integration for thermal conventional technologies by building PEMMDB-based capacity tables from TYNDP data, wiring them into the Snakemake pipeline, and applying them post-solve as p_nom_min floors to non-AT thermal assets when enabled via configuration. Sequence diagram for applying PEMMDB brownfield floors post-solvesequenceDiagram
participant SM as Snakemake
participant Solve as solve_network.py
participant Net as pypsa.Network
participant Apply as apply_pemmdb_brownfield
participant CSV as pemmdb_capacities_{planning_horizons}.csv
participant Prior as _get_ppm_spatial_prior
participant Setter as _set_p_nom_min_for_carrier
SM->>Solve: call create_optimization_model
Solve->>Net: build and solve network
Solve->>Solve: check config.mods.pemmdb_brownfield.enabled
alt PEMMDB brownfield enabled
Solve->>Apply: apply_pemmdb_brownfield(Net, Snakemake)
Apply->>Apply: read config and planning_horizon
Apply->>Apply: validate horizon in planning_horizons
Apply->>CSV: read PEMMDB capacities CSV
CSV-->>Apply: capacities table
Apply->>Apply: filter skip_countries and aggregate by bus,country,carrier
loop for each (country,carrier)
Apply->>Net: identify model buses for country
Apply->>Prior: _get_ppm_spatial_prior(Net, buses, carrier)
Prior-->>Apply: ppm_p_nom per bus
Apply->>Apply: compute spatial weights (PPM or uniform)
loop for each target bus
Apply->>Setter: _set_p_nom_min_for_carrier(Net, bus, carrier, node_p_nom)
Setter-->>Apply: updated_count
Apply->>Apply: accumulate applied_count / skipped_count
end
end
Apply-->>Solve: Net updated in place
else PEMMDB brownfield disabled or horizon not allowed
Solve->>Solve: skip apply_pemmdb_brownfield
end
Solve->>Net: export_to_netcdf
ER diagram for PEMMDB capacities table used for brownfield floorserDiagram
pemmdb_capacities {
string bus
string pemmdb_carrier
string pemmdb_type
string open_tyndp_carrier
string open_tyndp_type
string pypsa_eur_carrier
float p_nom
float e_nom
float efficiency
string country
int planning_horizon
string unit
}
buses {
string bus
string country
string carrier
}
pemmdb_capacities ||--o{ buses : maps_to_model_bus
Flow diagram for PEMMDB v2.4 thermal brownfield data pipelineflowchart LR
subgraph OpenTYNDP_data
A_retrieve_open_tyndp[retrieve_open_tyndp rule]
A_xlsx["PEMMDB_{node}_NationalTrends_{year}.xlsx (per country/year)"]
end
subgraph Build_phase
B_build_rule[build_pemmdb_data rule]
B_script[build_pemmdb_data.py script]
B_map[tyndp_technology_map.csv]
B_csv["pemmdb_capacities_{planning_horizons}.csv"]
end
subgraph Solve_phase
C_solve_rule[solve_sector_network_myopic rule]
C_solve_script[solve_network.py]
C_apply[apply_pemmdb_brownfield function]
C_network["PyPSA Network (post-solve)"]
C_export["Exported network (NetCDF) with p_nom_min floors"]
end
A_retrieve_open_tyndp --> A_xlsx
A_xlsx --> B_build_rule
B_build_rule --> B_script
B_script --> B_map
B_script --> B_csv
B_csv --> C_solve_rule
C_solve_rule --> C_solve_script
C_solve_script --> C_network
C_solve_script -->|if mods.pemmdb_brownfield.enabled and horizon allowed| C_apply
B_csv --> C_apply
C_apply -->|update p_nom_min on Links/Generators/StorageUnits| C_network
C_network --> C_export
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- The helper
_get_ppm_spatial_priorclaims in its docstring to consider Links, Generators, and StorageUnits but currently only aggregates Links and Generators, so either includestorage_unitsin the prior or adjust the docstring to match the implementation. - In
solve_network.pyyou guard the call toapply_pemmdb_brownfieldwith a config check even though the function itself re-checksenabledandplanning_horizons; consider relying on the internal checks only to avoid duplicated configuration logic.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The helper `_get_ppm_spatial_prior` claims in its docstring to consider Links, Generators, and StorageUnits but currently only aggregates Links and Generators, so either include `storage_units` in the prior or adjust the docstring to match the implementation.
- In `solve_network.py` you guard the call to `apply_pemmdb_brownfield` with a config check even though the function itself re-checks `enabled` and `planning_horizons`; consider relying on the internal checks only to avoid duplicated configuration logic.
## Individual Comments
### Comment 1
<location path="mods/network_updates.py" line_range="562-571" />
<code_context>
+ """
+ updated = 0
+
+ # Links (thermal plants: bus1 = electricity output)
+ links_mask = (n.links.carrier == carrier) & (n.links.bus1 == bus)
+ if links_mask.any():
+ n.links.loc[links_mask, "p_nom_min"] = p_nom_min
+ updated += links_mask.sum()
+
+ # Generators
+ gen_mask = (n.generators.carrier == carrier) & (n.generators.bus == bus)
+ if gen_mask.any():
+ n.generators.loc[gen_mask, "p_nom_min"] = p_nom_min
+ updated += gen_mask.sum()
+
+ # StorageUnits
+ su_mask = (n.storage_units.carrier == carrier) & (n.storage_units.bus == bus)
+ if su_mask.any():
+ n.storage_units.loc[su_mask, "p_nom_min"] = p_nom_min
+ updated += su_mask.sum()
+
</code_context>
<issue_to_address>
**issue (bug_risk):** Using the full node `p_nom_min` for every component at a bus/carrier likely over-allocates capacity when multiple components exist.
For each `(bus, carrier)` you apply the same `p_nom_min` to every matching Link/Generator/StorageUnit. With multiple components at a node, this multiplies the effective minimum capacity (`node_p_nom × num_components`) rather than enforcing a single node-level floor from PEMMDB. It would be more accurate to allocate `p_nom_min` across components (e.g. proportionally to existing `p_nom` or evenly) so their summed `p_nom_min` matches the intended node-level `node_p_nom`.
</issue_to_address>
### Comment 2
<location path="mods/network_updates.py" line_range="523-532" />
<code_context>
+ """
+ Return the existing PPM-derived ``p_nom`` per bus as a spatial weight proxy.
+
+ Looks across Links, Generators, and StorageUnits for the given carrier.
+ Returns a Series indexed by bus with p_nom sums (0 if not present).
+ """
+ result = pd.Series(0.0, index=buses)
+
+ # Links: bus1 is the electricity output bus
+ links_mask = (n.links.carrier == carrier) & (n.links.bus1.isin(buses))
+ if links_mask.any():
+ for bus in buses:
+ bus_mask = links_mask & (n.links.bus1 == bus)
+ result[bus] = result[bus] + n.links.loc[bus_mask, "p_nom"].sum()
+
+ # Generators
+ gen_mask = (n.generators.carrier == carrier) & (n.generators.bus.isin(buses))
+ if gen_mask.any():
+ for bus in buses:
+ bus_mask = gen_mask & (n.generators.bus == bus)
</code_context>
<issue_to_address>
**issue (bug_risk):** The spatial prior ignores StorageUnits despite the docstring and caller expectations.
The code only considers Links and Generators, so StorageUnit capacity is excluded from the spatial weights despite what the docstring and caller expect. This can bias PEMMDB capacity allocation for carriers partly modeled as storage. Either include `n.storage_units` in the aggregation or update the docstring and callers so the behavior is consistent and not misleading.
</issue_to_address>
### Comment 3
<location path="scripts/pypsa-at/build_pemmdb_data.py" line_range="204-206" />
<code_context>
+ if caps is not None:
+ frames.append(caps)
+
+ if not frames:
+ logger.warning(f"No PEMMDB thermal capacities found for pyear={pyear}")
+ return pd.DataFrame()
+
+ all_caps = pd.concat(frames, ignore_index=True)
</code_context>
<issue_to_address>
**issue (bug_risk):** Returning an empty DataFrame with no columns will produce an empty CSV that `pd.read_csv` cannot parse later.
Here you return `pd.DataFrame()` with no columns, so `to_csv` will create an entirely empty file. The later `pd.read_csv(pemmdb_file)` call will then raise `EmptyDataError`. Please either:
- return an empty DataFrame with the expected columns so the CSV has headers, or
- avoid writing the file in this case and update downstream logic to treat a missing file as "no PEMMDB data".
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+562
to
+571
| # Links (thermal plants: bus1 = electricity output) | ||
| links_mask = (n.links.carrier == carrier) & (n.links.bus1 == bus) | ||
| if links_mask.any(): | ||
| n.links.loc[links_mask, "p_nom_min"] = p_nom_min | ||
| updated += links_mask.sum() | ||
|
|
||
| # Generators | ||
| gen_mask = (n.generators.carrier == carrier) & (n.generators.bus == bus) | ||
| if gen_mask.any(): | ||
| n.generators.loc[gen_mask, "p_nom_min"] = p_nom_min |
Contributor
There was a problem hiding this comment.
issue (bug_risk): Using the full node p_nom_min for every component at a bus/carrier likely over-allocates capacity when multiple components exist.
For each (bus, carrier) you apply the same p_nom_min to every matching Link/Generator/StorageUnit. With multiple components at a node, this multiplies the effective minimum capacity (node_p_nom × num_components) rather than enforcing a single node-level floor from PEMMDB. It would be more accurate to allocate p_nom_min across components (e.g. proportionally to existing p_nom or evenly) so their summed p_nom_min matches the intended node-level node_p_nom.
Comment on lines
+523
to
+532
| Looks across Links, Generators, and StorageUnits for the given carrier. | ||
| Returns a Series indexed by bus with p_nom sums (0 if not present). | ||
| """ | ||
| result = pd.Series(0.0, index=buses) | ||
|
|
||
| # Links: bus1 is the electricity output bus | ||
| links_mask = (n.links.carrier == carrier) & (n.links.bus1.isin(buses)) | ||
| if links_mask.any(): | ||
| for bus in buses: | ||
| bus_mask = links_mask & (n.links.bus1 == bus) |
Contributor
There was a problem hiding this comment.
issue (bug_risk): The spatial prior ignores StorageUnits despite the docstring and caller expectations.
The code only considers Links and Generators, so StorageUnit capacity is excluded from the spatial weights despite what the docstring and caller expect. This can bias PEMMDB capacity allocation for carriers partly modeled as storage. Either include n.storage_units in the aggregation or update the docstring and callers so the behavior is consistent and not misleading.
Comment on lines
+204
to
+206
| if not frames: | ||
| logger.warning(f"No PEMMDB thermal capacities found for pyear={pyear}") | ||
| return pd.DataFrame() |
Contributor
There was a problem hiding this comment.
issue (bug_risk): Returning an empty DataFrame with no columns will produce an empty CSV that pd.read_csv cannot parse later.
Here you return pd.DataFrame() with no columns, so to_csv will create an entirely empty file. The later pd.read_csv(pemmdb_file) call will then raise EmptyDataError. Please either:
- return an empty DataFrame with the expected columns so the CSV has headers, or
- avoid writing the file in this case and update downstream logic to treat a missing file as "no PEMMDB data".
| "pemmdb_capacities_{planning_horizons}.csv" | ||
| ) | ||
| } | ||
| if config.get("mods", {}) |
Collaborator
Author
There was a problem hiding this comment.
Should use config getter function
| : | ||
| Updates ``n`` in place. | ||
| """ | ||
| cfg = snakemake.config.get("mods", {}).get("pemmdb_brownfield", {}) |
Collaborator
Author
There was a problem hiding this comment.
require mods.pemmdb_brownfield key. Fail if missing
| skip_countries = cfg.get("skip_countries", ["AT"]) | ||
| pemmdb_file = Path(snakemake.input.pemmdb_capacities) | ||
|
|
||
| if not pemmdb_file.is_file(): |
Collaborator
Author
There was a problem hiding this comment.
require the pemmdb capacities file, fail if missing.
| "pemmdb_capacities_{planning_horizons}.csv" | ||
| ) | ||
| } | ||
| if config.get("mods", {}) |
Collaborator
Author
There was a problem hiding this comment.
use config getter helper function
Collaborator
Author
|
Critical Breaker detected: The PEMMDB data source does not contain brownfield capacities. It only contains projections for 2030+ and is used for lower capacity constraints in Open-TYNDP. I'll close this PR as I need to rethink the design |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements Phase 1 of pypsa-at-planning#96: PEMMDB v2.4 as the EU brownfield capacity source for thermal conventional technologies.
Replaces PowerPlantMatching (PPM) capacity totals for non-AT countries with PEMMDB v2.4 data (the official ENTSO-E capacity database), while retaining PPM's spatial plant location information for multi-region countries (e.g. Germany).
Feature is disabled by default (
mods.pemmdb_brownfield.enabled: false). No existing runs are affected.Files changed
data/pypsa-at/tyndp_technology_map.csvscripts/pypsa-at/build_pemmdb_data.pyrules/open-tyndp/build.smkbuild_pemmdb_dataSnakemake rulemods/network_updates.pyapply_pemmdb_brownfield()+ 2 private helpersscripts/solve_network.pyapply_pemmdb_brownfield()when enabledrules/solve_myopic.smkpemmdb_capacitiesinput when enabledconfig/config.at.yamlmods.pemmdb_brownfieldconfig block (disabled)test/test_pemmdb_brownfield.pyArchitecture
Carrier mapping (
tyndp_technology_map.csv): closely mirrors the open-tyndp technology map used in Open-TYNDP PR #97, withpypsa_eur_carriercolumn added for PyPSA-AT/EUR conventions.Multi-region distribution (e.g. Germany 5/16 nodes): PEMMDB national totals are distributed using existing PPM
p_nomas spatial weights; falls back to uniform distribution if no PPM prior exists.AT always skipped (
skip_countries: ["AT"]): Austrian brownfield is tracked in a separate issue stream.Phase scope
Testing
Unit tests in
test/test_pemmdb_brownfield.pycover:_set_p_nom_min_for_carrier(): Link, Generator, unknown carrier_get_ppm_spatial_prior(): zero prior, known linksapply_pemmdb_brownfield(): disabled, wrong horizon, CCGT, nuclear, AT skipContract tests (
test_pemmdb_brownfield_contract) are parametrized over all planning horizons and skip automatically when solved networks are not present (CI-safe). They are allowed to fail during early development as documented in the issue.Closes part of AGGM-AG/pypsa-at-planning#96 (Phase 1).
Summary by Sourcery
Integrate PEMMDB v2.4 as an optional brownfield capacity source for thermal conventional technologies in non‑AT EU countries and wire it into the myopic solve workflow while keeping the feature disabled by default.
New Features:
Enhancements:
Tests: