Merge changes from upstream (incl. PyPSA-Eur v2026.02.0) - #754
Conversation
* fix missing or outdated configuration options * pixi run generate-config
* allow custom efficiency_classes for conventional generators * pixi run generate-config
* chore: pin grpcio<1.78 to silence warnings * doc: add release note * Update envs/environment.yaml --------- Co-authored-by: Lukas Trippe <lkstrp@pm.me>
* Update pixi `run-post-link-scripts` config * Remove `dot -c` from pixi tasks --------- Co-authored-by: Lukas Trippe <lkstrp@pm.me>
* distribute electricity demand by JRC energy atlas * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * reset_index when redistribute_attribute is called * add dependency and adapt to new data inventory * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add to data.csv * codespell fixes and update pixi.lock * update schema * fixes for new data layer * add rule messages * solve slow DAG evaluation The rule was using storage() in the input block, which caused Snakemake to make HTTP HEAD requests during DAG construction to verify the file exists. The UK government server is slow to respond to HEAD requests, causing a 30x slowdown (2 sec → 60 sec) in DAG evaluation. This fix removes the input block and handles the download directly in the run block using requests.get(), matching the pattern used by retrieve_ons_lad. This eliminates the existence check and restores DAG evaluation speed. * fix retrieval issues with storage() * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add raise_for_status * restore master pixi.lock * update pixi.lock * add to archive * more archive * handle archive * fix swiss * generate config * fix: url format * add retrieval for for archive and fix upload * another url fix * update pixi.lock --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: lkstrp <lkstrp@pm.me>
* Reverting to .pdf file ending. * Added release notes. --------- Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
Co-authored-by: lkstrp <62255395+lkstrp@users.noreply.github.com> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
…PSA#2087) * Allow overwriting docs URL; select correct default config in test * Minor fixes * Generalise pre-commit config file exclusion
* Link release badge to documentation I already caught myself several times clicking the release badge and expecting to get to the docs of that version. Now it should work * Link to stable instead of specific relewase * update release badge link; move docs badge
Conflicts: doc/release_notes.rst
Merge changes from upstream
* fix: lower the lower bound for power grouping years * doc: add release note
* refactor: use scripts path provider * doc: add release note --------- Co-authored-by: Lukas Trippe <lkstrp@pm.me>
![pypsa[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
* feat: set up pypsa-bot * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * trigger env change * another one * now authorized * [pypsa-bot] update sync-locks - `pixi run sync-locks` * trigger * [pypsa-bot] update sync-locks - `pixi run sync-locks` * test config schema trigger * [pypsa-bot] update sync-locks & generate-config - `pixi run sync-locks` - `pixi run generate-config` * test full new deps solve * [pypsa-bot] update sync-locks & generate-config - `pixi run sync-locks` - `pixi run generate-config` * Revert "test config schema trigger" This reverts commit 7e6eb23. * Revert "test full new deps solve" This reverts commit 11d16bc. * [pypsa-bot] update sync-locks - `pixi run sync-locks` * empty trigger after fix * [pypsa-bot] run `sync-locks` & `generate-config` - `pixi run sync-locks` - `pixi run generate-config` --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: pypsa[bot] <994572+pypsa[bot]@users.noreply.github.com> Co-authored-by: pypsa[bot] <181215446+pypsa[bot]@users.noreply.github.com>
daniel-rdt
left a comment
daniel-rdt
left a comment
There was a problem hiding this comment.
Thank you @tgilon for all this hard work and the super comprehensive documentation of what you did. Great job overall!
Largely, the changes look to be consistent and as you already pointed out wrt SB, the results are indeed unchanged. You rightfully pointed out that the additional missing carrier was already introduced as an artifact of undefined for generation. My two main comments are related to:
- newly introduced file dependencies from upstream
- Implicit minimum pixi version (≥ 0.67) required which we should make explicit in the release note, readme, PR description etc.
| | | ||
|
|
||
| !!! warning | ||
| Open-TYNDP is under active development and is not yet feature-complete. The current [development status](#development-status) and the general [Limitations](limitations.md) are important to understand before using the model. |
There was a problem hiding this comment.
When we do, we need to remember to also update this in the Readme accordingly. I added this to the issue that you linked.
Thank you @Meas for this overview! After analysing this further, I came to the conclusion that the different order of the links you mentioned is the root cause of the observed differences. My investigation suggests that the swapped positions induce swapped noisy costs both for capital and marginal costs. This induces slightly different dispatch and cascading tiny differences across the system. To validate this, I scanned the networks to search for differences with the code down below (see Test code). This confirms your observations. Could you please run another test to confirm our conclusions? I would like to avoid random list ordering for new links here. Currently, the Test codeDetailsdef get_all_attrs(n: pypsa.Network, attr: str) -> pd.Series:
return pd.concat([c.static[attr] for c in n.components if attr in c.static])
def compare_attrs(attr: str, eps: float = 1e-6) -> pd.DataFrame:
return (
pd.concat(
[
get_all_attrs(n_master, attr).rename("master"),
get_all_attrs(n_pr, attr).rename("pr"),
],
axis=1,
)
.rename_axis(attr)
.query("abs(master-pr) / master >@eps")
)
for attr in ["capital_cost", "marginal_cost", "p_nom", "s_nom", "e_nom"]:
print(compare_attrs(attr))
print("\n")
master pr
capital_cost
LV00-SE03-DC 68.679905 71.806374
SE03-LV00-DC 71.806374 68.679905
master pr
marginal_cost
LV00-SE03-DC 0.010808 0.009412
SE03-LV00-DC 0.009412 0.010808
Empty DataFrame
Columns: [master, pr]
Index: []
Empty DataFrame
Columns: [master, pr]
Index: []
Empty DataFrame
Columns: [master, pr]
Index: [] |
Conflicts:
.github/workflows/test.yaml
.github/workflows/update-lockfile.yaml
Co-authored-by: Daniel Rüdt <117752024+daniel-rdt@users.noreply.github.com>
…ity_demand_base_tyndp
Conflicts: .github/workflows/test.yaml
daniel-rdt
left a comment
daniel-rdt
left a comment
There was a problem hiding this comment.
Thank you @tgilon for addressing all my comments. This is looking good for me now. Only one minor comment.
Main task before merging this PR I see relates to solving the CBA differences @measrainsey identified. Let's wait for the new test runs.
Co-authored-by: Daniel Rüdt <117752024+daniel-rdt@users.noreply.github.com>
…ity_demand_base_tyndp
|
Thanks @tgilon for your suggestions! While we wait for the latest high-res runs to be uploaded onto Google Drive in case you also want to look at them, I'll just share what I have so far. I ran both the master branch and this PR branch with the suggested test of Thus, I used your test code to compare four high-res runs:
The full test code: Detailsimport pandas as pd
import pypsa
path_main = "/Users/meas/Dropbox/oet/projects/tyndp/zib-results"
path_master = "NT-1H-20260622-master"
path_pr = "NT-1H-20260622-pinned-cba"
path_master_sort_links = "NT-1H-20260701-master-sort-links-sb"
path_pr_sort_links = "NT-1H-20260701-upstream-sort-links"
# list relative paths
path_sb_2030 = "networks/base_s_all___2030.nc"
path_sb_2040 = "networks/base_s_all___2040.nc"
# load networks
n_master_2040 = pypsa.Network(f"{path_main}/{path_master}/{path_sb_2040}")
n_pr_2040 = pypsa.Network(f"{path_main}/{path_pr}/{path_sb_2040}")
n_master_sort_links_2040 = pypsa.Network(
f"{path_main}/{path_master_sort_links}/{path_sb_2040}"
)
n_pr_sort_links_2040 = pypsa.Network(f"{path_main}/{path_pr_sort_links}/{path_sb_2040}")
# test code
def compare_links(n1, n2, name1=None, name2=None):
name1 = name1 or "n1"
name2 = name2 or "n2"
def get_all_attrs(n: pypsa.Network, attr: str) -> pd.Series:
return pd.concat([c.static[attr] for c in n.components if attr in c.static])
def compare_attrs(attr: str, eps: float = 1e-6) -> pd.DataFrame:
return (
pd.concat(
[
get_all_attrs(n1, attr).rename(name1),
get_all_attrs(n2, attr).rename(name2),
],
axis=1,
)
.rename_axis(attr)
.query(f"abs({name1}-{name2}) / {name1} >@eps")
)
for attr in ["capital_cost", "marginal_cost", "p_nom", "s_nom", "e_nom"]:
print(compare_attrs(attr))
print("\n")As you already showed in your comment, if comparing the compare_links(n_master_2040, n_pr_2040, "master", "pr")
master pr
capital_cost
LV00-SE03-DC 68.679905 71.806374
SE03-LV00-DC 71.806374 68.679905
master pr
marginal_cost
LV00-SE03-DC 0.010808 0.009412
SE03-LV00-DC 0.009412 0.010808
Empty DataFrame
Columns: [master, pr]
Index: []
Empty DataFrame
Columns: [master, pr]
Index: []
Empty DataFrame
Columns: [master, pr]
Index: []Comparing both the compare_links(
n_master_sort_links_2040,
n_pr_sort_links_2040,
"master_sort_links",
"pr_sort_links",
)
Empty DataFrame
Columns: [master_sort_links, pr_sort_links]
Index: []
Empty DataFrame
Columns: [master_sort_links, pr_sort_links]
Index: []
master_sort_links pr_sort_links
p_nom
NOM1 electricity distribution grid-2030 4651.936179 4651.931225
NON1 electricity distribution grid-2030 3992.818140 3992.811975
NOS0 electricity distribution grid-2030 20687.517123 20687.579318
Empty DataFrame
Columns: [master_sort_links, pr_sort_links]
Index: []
Empty DataFrame
Columns: [master_sort_links, pr_sort_links]
Index: []If we compare the compare_links(n_master_2040, n_pr_sort_links_2040, "master", "pr_sort_links")
Empty DataFrame
Columns: [master, pr_sort_links]
Index: []
Empty DataFrame
Columns: [master, pr_sort_links]
Index: []
Empty DataFrame
Columns: [master, pr_sort_links]
Index: []
Empty DataFrame
Columns: [master, pr_sort_links]
Index: []
Empty DataFrame
Columns: [master, pr_sort_links]
Index: [] |
willu47
left a comment
willu47
left a comment
There was a problem hiding this comment.
2nd review of documentation and I found no problems.
|
In general, I'm noticing some differences in the CBA results compared to on My assessment is I don't think there's anything to be changed with respect to this PR (as in I don't think it's necessarily introducing anything that's messing with the CBA results inherently), this is just something that probably needs to be improved within the CBA workflow. Run settingsI ran two new runs this morning:
For each run, I made sure to re-install my I used the same config file for both runs: Detailsrun:
name: "NT"
clustering:
temporal:
resolution_sector: 1H
solving:
options:
assign_all_duals: true
include_objective_constant: false
solver:
name: gurobi
options: gurobi-default
partition: big
mem_mb: 40000
runtime: 1h
cba:
projects:
- t1-t200
msv_extraction:
solving:
options:
include_objective_constant: false
solver:
name: gurobi
options: gurobi-default
solver_options:
gurobi-default:
BarConvTol: 1.0e-08
partition: big
mem_mb: 40000
runtime: 1h
# Rolling horizon solving settings
solving:
options:
include_objective_constant: false
load_sinks:
enable: true
all_carriers: false
carriers:
"co2 sequestered": 3000
partition: big
mem_mb: 40000
runtime: 1h
solver:
name: highs
options: highs-simplex
solver_options:
highs-simplex:
primal_feasibility_tolerance: 1.0e-08
dual_feasibility_tolerance: 1.0e-08
Checking outputsAttached is a CSV that compares the CBA outputs of all the indicators included (t1-200): While some projects maintain mostly the same (or close to the same) results, some projects do show large changes. For example, project In In the PR branch (+ sort links): I used the following code to check the SB networks and the CBA reference grid across the two runs: Detailsimport pandas as pd
import pypsa
path_main = "/Users/meas/Dropbox/oet/projects/tyndp/zib-results"
path_master = "NT-1H-20260703-master"
path_pr_sort_links = "NT-1H-20260703-pr-sort-links"
path_reference_2040 = "results/cba/networks/reference_2040.nc"
path_sb_2040 = "results/networks/base_s_all___2040.nc"
n_reference_master = pypsa.Network(f"{path_main}/{path_master}/{path_reference_2040}")
n_reference_pr = pypsa.Network(
f"{path_main}/{path_pr_sort_links}/{path_reference_2040}"
)
n_sb_master = pypsa.Network(f"{path_main}/{path_master}/{path_sb_2040}")
n_sb_pr = pypsa.Network(f"{path_main}/{path_pr_sort_links}/{path_sb_2040}")
# compare links
def compare_links(n1, n2, name1=None, name2=None):
name1 = name1 or "n1"
name2 = name2 or "n2"
def get_all_attrs(n: pypsa.Network, attr: str) -> pd.Series:
return pd.concat([c.static[attr] for c in n.components if attr in c.static])
def compare_attrs(attr: str, eps: float = 1e-6) -> pd.DataFrame:
return (
pd.concat(
[
get_all_attrs(n1, attr).rename(name1),
get_all_attrs(n2, attr).rename(name2),
],
axis=1,
)
.rename_axis(attr)
.query(f"abs({name1}-{name2}) / {name1} >@eps")
)
for attr in ["capital_cost", "marginal_cost", "p_nom", "s_nom", "e_nom"]:
print(compare_attrs(attr))
print("\n")
# compare expanded_capacity
def compare_expanded_capacity(n1, n2, name1="n1", name2="n2"):
def get_expanded_capacity(n: pypsa.Network) -> pd.DataFrame:
return pd.DataFrame(
{
f"expanded_capacity_{name1 if n is n1 else name2}": n.statistics.expanded_capacity(),
},
)
df1 = get_expanded_capacity(n1)
df2 = get_expanded_capacity(n2)
out = df1.merge(
df2,
left_index=True,
right_index=True,
how="outer",
suffixes=(f"_{name1}", f"_{name2}"),
)
out["expanded_capacity_diff"] = (
out[f"expanded_capacity_{name2}"] - out[f"expanded_capacity_{name1}"]
)
return out.sort_values("expanded_capacity_diff", key=abs, ascending=False)
# compare load shedding
def compare_load_shedding(n1, n2, name1="n1", name2="n2"):
def get_load_shedding(n: pypsa.Network) -> pd.DataFrame:
gens = n.generators[n.generators.carrier == "load"].copy()
return pd.DataFrame(
{
"bus": gens.bus,
f"e_sum_{name1 if n is n1 else name2}": n.generators_t.p[
gens.index
].sum(),
f"e_max_{name1 if n is n1 else name2}": n.generators_t.p[
gens.index
].max(),
},
index=gens.index,
)
df1 = get_load_shedding(n1)
df2 = get_load_shedding(n2)
out = df1.merge(
df2,
left_index=True,
right_index=True,
how="outer",
suffixes=(f"_{name1}", f"_{name2}"),
)
out["e_sum_diff"] = out[f"e_sum_{name2}"] - out[f"e_sum_{name1}"]
out["e_max_diff"] = out[f"e_max_{name2}"] - out[f"e_max_{name1}"]
return out.sort_values("e_sum_diff", key=abs, ascending=False)
# compare opex
def compare_opex(n1, n2, name1="n1", name2="n2"):
def get_opex(n: pypsa.Network) -> pd.DataFrame:
return pd.DataFrame(
{
f"opex_{name1 if n is n1 else name2}": n.statistics.opex(
aggregate_time="sum"
).sum()
/ 1e6,
},
index=[0],
)
df1 = get_opex(n1)
df2 = get_opex(n2)
out = df1.merge(
df2,
left_index=True,
right_index=True,
how="outer",
suffixes=(f"_{name1}", f"_{name2}"),
)
out["opex_diff"] = out[f"opex_{name2}"] - out[f"opex_{name1}"]
return out.sort_values("opex_diff", key=abs, ascending=False)SB networks (2040)If comparing links between the SB networks, again the only difference is a small arithmetic change in the compare_links(n_sb_master, n_sb_pr, "master", "pr_sort_links")
Empty DataFrame
Columns: [master, pr_sort_links]
Index: []
Empty DataFrame
Columns: [master, pr_sort_links]
Index: []
master pr_sort_links
p_nom
NON1 electricity distribution grid-2030 3992.81814 3992.812356
Empty DataFrame
Columns: [master, pr_sort_links]
Index: []
Empty DataFrame
Columns: [master, pr_sort_links]
Index: []Again it seems there are still some differences in the solved SB networks, such as in the expanded capacities of coal, uranium, and co2 sequestered (I notice the SB networks' objective values are similar but just slightly different between the two runs). print(compare_expanded_capacity(n_sb_master, n_sb_pr, "master", "pr_sort_links").head())
expanded_capacity_master \
component carrier
Store Lignite Primary 1.920069e+07
uranium 1.952611e+07
Coal Primary 1.991604e+07
co2 sequestered 1.661369e+08
Oil 6.351064e+02
expanded_capacity_pr_sort_links \
component carrier
Store Lignite Primary 1.435378e+07
uranium 1.470310e+07
Coal Primary 1.517621e+07
co2 sequestered 1.661370e+08
Oil 5.379969e+02
expanded_capacity_diff
component carrier
Store Lignite Primary -4.846909e+06
uranium -4.823011e+06
Coal Primary -4.739828e+06
co2 sequestered 1.020223e+02
Oil -9.710944e+01 CBA reference grids (2040)I focus on the reference grid because if just the reference grid is changing, then the CBA results will change as the indicators are all relative to the reference grid. Realistically there will also likely be differences in the project networks between the two runs, but I haven't checked those in depth. If comparing the links between the two runs' 2024 CBA reference grids, there are more links that are different. Some of these changes are again small arithmetic changes, but actually some seem somewhat significant changes, such as in lignite, coal, uranium. compare_links(n_reference_master, n_reference_pr, "master", "pr_sort_links")
Empty DataFrame
Columns: [master, pr_sort_links]
Index: []
Empty DataFrame
Columns: [master, pr_sort_links]
Index: []
master pr_sort_links
p_nom
EU lignite 9436.405542 9455.137333
EU coal 59321.479215 59257.535274
EU uranium 297250.612841 297249.271207
EU gas 374246.027076 374333.888937
co2 sequestered 45925.439535 45925.362544
EU solid biomass biomass to liquid 0.012660 0.010764
BA00 electricity distribution grid-2040 176.036841 176.037999
CH00 electricity distribution grid-2040 2051.010349 2051.007705
CY00 electricity distribution grid-2040 310.931530 310.933234
CZ00 electricity distribution grid-2040 2701.864827 2701.859495
DKE1 electricity distribution grid-2040 1309.898342 1309.891100
DKW1 electricity distribution grid-2040 2768.288588 2768.284919
EE00 electricity distribution grid-2040 434.540983 434.539998
FI00 electricity distribution grid-2040 926.864462 926.862967
GR03 electricity distribution grid-2040 127.990634 127.991155
HR00 electricity distribution grid-2040 334.665428 334.665000
HU00 electricity distribution grid-2040 1778.625042 1778.623000
ITCA electricity distribution grid-2040 0.033306 0.028264
ITCN electricity distribution grid-2040 326.173145 326.169022
ITN1 electricity distribution grid-2040 3063.322504 3063.313667
ITSA electricity distribution grid-2040 281.401913 281.402299
ITSI electricity distribution grid-2040 514.853297 514.852357
LUB1 electricity distribution grid-2040 0.033291 0.028261
LUF1 electricity distribution grid-2040 40.553233 40.552001
LUG1 electricity distribution grid-2040 0.033676 0.030691
LV00 electricity distribution grid-2040 0.071256 0.060042
ME00 electricity distribution grid-2040 187.930010 187.927319
MT00 electricity distribution grid-2040 173.838924 173.837261
NOM1 electricity distribution grid-2040 0.038165 0.032410
NON1 electricity distribution grid-2040 222.856719 222.862059
NOS0 electricity distribution grid-2040 0.299778 0.230296
PL00 electricity distribution grid-2040 7911.882735 7911.870637
RO00 electricity distribution grid-2040 886.844283 886.842469
RS00 electricity distribution grid-2040 214.506954 214.506682
SE01 electricity distribution grid-2040 177.806081 177.805188
SI00 electricity distribution grid-2040 770.429530 770.428063
SK00 electricity distribution grid-2040 676.547783 676.545553
NON1 electricity distribution grid-2030 3992.818140 3992.812356
Empty DataFrame
Columns: [master, pr_sort_links]
Index: []
master pr_sort_links
e_nom
EU lignite Store 1.920069e+07 1.435378e+07
EU coal Store 1.991604e+07 1.517621e+07
EU oil Store 6.351063e+02 5.379969e+02
EU uranium Store 1.952611e+07 1.470310e+07
co2 stored 7.373066e+00 6.229227e+00
EU gas Store 8.440684e+01 7.178147e+01There seems to be quite a difference in load shedding between the two reference grids: print(compare_load_shedding(n_reference_master, n_reference_pr, "master", "pr_sort_links")[["e_sum_master", "e_sum_pr_sort_links", "e_sum_diff"]])
e_sum_master e_sum_pr_sort_links e_sum_diff
name
DE00 load shedding 9.012529e+06 8.973102e+06 -39426.205419
NL00 load shedding 2.079794e+05 2.028662e+05 -5113.141648
CZ00 load shedding 3.661652e+05 3.623365e+05 -3828.711000
PL00 load shedding 4.804684e+05 4.780489e+05 -2419.460351
PL H2 load shedding 4.656095e+06 4.657646e+06 1551.774315
... ... ... ...
FROH003 H2 load shedding 0.000000e+00 0.000000e+00 0.000000
FROH002 load shedding 0.000000e+00 0.000000e+00 0.000000
FROH002 H2 load shedding 0.000000e+00 0.000000e+00 0.000000
FROH001 load shedding 0.000000e+00 0.000000e+00 0.000000
co2 stored load shedding 0.000000e+00 0.000000e+00 0.000000Comparatively, the differences in load shedding between the SB networks are not as significant: print(compare_load_shedding(n_sb_master, n_sb_pr, "master", "pr_sort_links")[["e_sum_master", "e_sum_pr_sort_links", "e_sum_diff"]])
e_sum_master e_sum_pr_sort_links e_sum_diff
name
DK H2 load shedding 6.510447e+05 6.529798e+05 1935.101571
PL H2 load shedding 4.631602e+06 4.629825e+06 -1777.148631
DKE1 load shedding 2.458078e+04 2.534971e+04 768.929005
DE H2 load shedding 1.500195e+06 1.500628e+06 432.219556
SE H2 load shedding 3.462370e+05 3.466639e+05 426.912323
... ... ... ...
RS H2 load shedding 0.000000e+00 0.000000e+00 0.000000
NO H2 load shedding 0.000000e+00 0.000000e+00 0.000000
MT H2 load shedding 0.000000e+00 0.000000e+00 0.000000
MK H2 load shedding 0.000000e+00 0.000000e+00 0.000000
AL H2 load shedding 0.000000e+00 0.000000e+00 0.000000And finally, the opex of the reference grids are also different across runs: compare_opex(n_reference_master, n_reference_pr, "master", "pr_sort_links")
opex_master opex_pr_sort_links opex_diff
0 545065.5901 544934.226723 -131.363377While it doesn't seem like a big difference, because the indicators such as the B1 indicator (which uses opex) are all relative to the reference grid's values, even small changes such as these can throw off the indicator a lot (even switching the sign of the indicator). Just to show that in comparison, the SB networks' opex doesn't change much: compare_opex(n_sb_master, n_sb_pr, "master", "pr_sort_links")
opex_master opex_pr_sort_links opex_diff
0 507638.536247 507638.378703 -0.157544 |
|
Thank you @Meas for the detailed analysis. I ran some other checks too using your latest networks and came to the same conclusion as you. Numerical tolerances and indicator sensitivity are probably to blame here. I took another look at the 2040 CBA reference networks and it looks plausible. I pushed the fix in 0d0c02e. Additionally, I created some figures to visualise the deviations >1% for the reference networks. I would argue that there is nothing we can do about in this PR. In the details, you'll also find a scatter plot with all the data points. I used Energy balance - AC
Details
Energy balance - H2
Details
Transmission - AC
Details
Transmission - H2(no differences >1% for transmission H2) 
Opex - AC
Details
Opex - H2
Details
|
measrainsey
left a comment
measrainsey
left a comment
There was a problem hiding this comment.
Thanks @tgilon for your additional and detailed analysis and plots! It's really helpful to be able to visualize the differences between the networks and to see that they're relatively minor (though they lead to larger differences in the overall CBA output but again that's something we should fix elsewhere, outside of this PR).
Then that marks the end of my review :)
18 participants
Closes #643.
Changes proposed in this Pull Request
This PR brings the latest changes from PyPSA/PyPSA-Eur (incl. PyPSA-Eur v2026.02.0 and up to PyPSA@ca2a493), incorporating significant upstream changes to the codebase.
Preview of the new documentation page: https://open-tyndp--754.org.readthedocs.build/en/754/
This PR does not affect the results, detailed benchmarks are provided in this comment.
Note: Requires
pixiv0.67 or higher to use relative exclude-newer (see PyPSA#2163). Runpixi self-update.Overview
Several key upstream contributions are worth noting:
Additional changes were required to integrate upstream modifications:
solving.optionsthrough to CBA solves #708)latesttag fornitrogen_statisticstyndp-archiveI successfully ran the
tyndp-sbandtyndp-cbatasks locally but I cannot validate meaningful results at 168H temporal aggregation. All the tests are passing locally, including thetyndp-cyears-test.Grype test fails because of one critical vulnerability. Unfortunately, this vulnerability is shared with upstream.
Next steps
Review requests:
Workflow
Files to focus on (active integration work):
README.md— Focus on Open-TYNDP contentdoc/*.mdfiles.github/workflows/update-lockfile.yaml— Integration ofpixi run sync-lockswhile keeping the feature of updating the environment on a PR branchSnakefile,rules/build_electricity.smk,rules/build_sector.smk,rules/retrieve.smk,rules/postprocess.smk,rules/collect.smk,rules/solve_*.smk— mainly to integrate thesnakefmtsortingscripts/solve_network.py,scripts/prepare_sector_network.py,scripts/add_brownfield.py,scripts/add_existing_baseyear.py,scripts/build_electricity_demand_base.py,scripts/build_snapshot_weightings.py,scripts/build_tyndp_network.py— Various merge conflictsconfig/config.default.yaml,config/schema.default.json,scripts/lib/validation/config/__init__.py,scripts/lib/validation/config/_schema.py,scripts/lib/validation/config/solving.py— Ensuringpixi run generate-configworks as expecteddata/versions.csv— Addition of new sources in the tabledoc/data_inventory.csv— Addition of Markdown links and new sources in the tablepixi.tomland all environment files — Version unpinned to match upstream, improvedclean-tyndptask, addedsync-locks.pre-commit-config.yaml,.github/pull_request_template.md,.github/workflows/test.yamlCITATION.cff— Attribute order aligned with upstream; upstream version updatedFollow-up tasks
Notes
Git merge conflicts:
Generative AI was used to assist with code migration and review. All content has been reviewed, verified, and is the responsibility of the author.
Checklist
pixi.toml(usingpixi add <dependency-name>).config/config.default.yaml.doc/configtables/*.csv.config/test/*.yaml.pixi run -e open-tyndp tyndp-cyears-test).doc/*.rstfiles.doc/release_notes.rstis added.doc/index.rst.