feat: edit marginal costs and dispatch of biomass/biogas generators in CBA - #719
Conversation
measrainsey
changed the title
measrainsey
changed the title
tgilon
left a comment
tgilon
left a comment
There was a problem hiding this comment.
Thank you @measrainsey for this PR!
I agree with your open issue on the methodology. I would recommend using a methodology similar to apply_msv_to_network where the marginal prices defines the marginal costs. Here, adding the two values together seems inappropriate. For your information, I ran my test with this fix.
To test this PR, I had to set cba.storage.soc_boundary_carriers = [] (see my config below). This relaxes the constraint on the reservoir state of charge at rolling horizon boundaries.
cba:
planning_horizons:
- 2030
cba_scenario_input:
use_presolved: true
sb_version: latest
projects:
- t130
storage:
cyclic_carriers:
- battery
soc_boundary_carriers: []
msv_extraction:
resolution: 6H
solving:
horizon: 168
overlap: 1With this in mind, I tested four setups:
- one with the current branch (including the fix mentioned in the first paragraph), which gave optimal results ;
- one with a slack of 0 and
apply_biomass_biogas_bus_marginal_pricescommented out, which gave infeasibility at the 49th iteration ; - one with the slack set to 0 only, which also gave infeasibility at the 49th iteration ;
- one with only the
apply_biomass_biogas_bus_marginal_pricescommented out, which give optimal results.
I cannot therefore conclude that the introduction of apply_biomass_biogas_bus_marginal_prices is strictly necessary at low resolution. However, it's a nice feature and probably worth keeping.
Finally, I would argue that setting the slack to 40% is arbitrary. I would recommend documenting the results that justified this, at least. It would also be good to define this as a config too.
Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org>
Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org>
Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org>
…ad of hardcoding
Conflicts: doc/release_notes.rst rules/cba.smk
for more information, see https://pre-commit.ci
|
Thank you @tgilon for your thorough testing and review! I think I've addressed your comments, which has led to some changes in the code, so please take another look to see if any additional improvements are needed :) I also updated the PR description quite a bit with more details about my tests, including how I landed on the 40% slack. I hope that is sufficient, but let me know if you think this information should be documented further/elsewhere. |
tgilon
left a comment
tgilon
left a comment
There was a problem hiding this comment.
Thank you, @measrainsey, for improving this PR! The features look good now and are well documented. I'm still wondering how / when we will be able to refine the CBA enough to tighten the slack. Nevertheless, this is beyond the scope of this PR. Should we track this somewhere so that we can revisit it at some point?
I also made some minor suggestions in the comments.
|
cc @lisazeyen |
Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org>
Co-authored-by: Thomas Gilon <thomas.gilon@openenergytransition.org>
|
Thanks @tgilon for your feedback! I know you already approved the PR, but I made a small change to the |
lisazeyen
left a comment
lisazeyen
left a comment
There was a problem hiding this comment.
@measrainsey thanks for preparing the PR! From my understanding of the water values / applying the marginal prices we do not need / want to include the e_sum_min / e_sum_max anymore, this was only needed since we did not have the marginal prices in place. I think these parts as well as the slack can be removed. Can we do one test run before merging where we compare the dispatch of the biomass / biogas generators with rolling horizon from this PR with
- master dispatch in rolling horizon
- dispatch in MSV with perfect foresight (from master or this PR should be the same)
for one project?
| area: tyndp # options: tyndp (TODO: entso-e, eu27) | ||
| remove_noisy_costs: true # Use pre-noise costs for indicators while keeping noisy costs in the solve | ||
| negative_toot_capacity: zero | ||
| # Slack on rolling-horizon biomass and biogas e_sum_min volume limits |
There was a problem hiding this comment.
from my understanding of the water value approach we should remove the limits on e_sum_min and e_sum_max, this should be covered by the introduced marginal prices
There was a problem hiding this comment.
Gotcha, you're probably right. These are now redundant.
No more comments from my side, except the point made by Lisa |
Thanks @lisazeyen for your review and suggestion! You're probably right. I ran three tests:
All tests were run on 1H resolution, on just the t16 project in 2040. For run 1, I used the following config (note that running on Detailsrun:
name: "NT"
clustering:
temporal:
resolution_sector: 1H
solving:
options:
assign_all_duals: true
include_objective_constant: false
solver:
name: gurobi
options: gurobi-default
cba:
planning_horizons:
- 2040
projects: [t16]
msv_extraction:
solving:
options:
include_objective_constant: false
solver:
name: gurobi
options: gurobi-default
solver_options:
gurobi-default:
BarConvTol: 1.0e-08
partition: big
mem_mb: 40000
runtime: 1h
# Rolling horizon solving settings
solving:
options:
include_objective_constant: false
load_sinks:
enable: true
all_carriers: false
carriers:
"co2 sequestered": 3000
partition: big
mem_mb: 40000
runtime: 1h
solver:
name: gurobi
options: gurobi-simplex
solver_options:
gurobi-simplex:
FeasibilityTol: 1.0e-08
OptimalityTol: 1.0e-08For runs 2-3, I used the following config: Detailsrun:
name: "NT"
clustering:
temporal:
resolution_sector: 1H
solving:
options:
assign_all_duals: true
include_objective_constant: false
solver:
name: gurobi
options: gurobi-default
cba:
planning_horizons:
- 2040
biomass_biogas_slack: 0.4
projects: [t16]
msv_extraction:
solving:
options:
include_objective_constant: false
solver:
name: gurobi
options: gurobi-default
solver_options:
gurobi-default:
BarConvTol: 1.0e-08
partition: big
mem_mb: 40000
runtime: 1h
# Rolling horizon solving settings
solving:
options:
include_objective_constant: false
partition: big
mem_mb: 40000
runtime: 1h
solver:
name: gurobi
options: gurobi-simplex
solver_options:
gurobi-simplex:
FeasibilityTol: 1.0e-08
OptimalityTol: 1.0e-08Comparing the dispatch of the 
I used the following code to generate the plots: Detailsfrom __future__ import annotations
import matplotlib.pyplot as plt
import pypsa
# load paths
path_results = "/Users/meas/oet/open-tyndp/results/tyndp"
path_resources = "/Users/meas/oet/open-tyndp/resources/tyndp/NT/cba"
path_master = "NT-gurobi-license-env"
path_pr_with_esum = "NT-pr-with-esum"
path_pr_without_esum = "NT-pr-without-esum"
path_proj_network = "cba/networks/project_t16_2040.nc"
path_msv_network = "msv_2040.nc"
# load project networks
n_proj_master = pypsa.Network(f"{path_results}/{path_master}/{path_proj_network}")
n_proj_pr_with_esum = pypsa.Network(
f"{path_results}/{path_pr_with_esum}/{path_proj_network}"
)
n_proj_pr_without_esum = pypsa.Network(
f"{path_results}/{path_pr_without_esum}/{path_proj_network}"
)
# load MSV networks
n_msv_master = pypsa.Network(f"{path_resources}/{path_master}/{path_msv_network}")
n_msv_pr_with_esum = pypsa.Network(
f"{path_resources}/{path_pr_with_esum}/{path_msv_network}"
)
n_msv_pr_without_esum = pypsa.Network(
f"{path_resources}/{path_pr_without_esum}/{path_msv_network}"
)
# for each network, plot the dispatch of "EU solid biomass" and "EU biogas" generators over time
generator_names = ["EU solid biomass", "EU biogas"]
# define networks dictionary
networks = {
"Master": {"proj": n_proj_master, "msv": n_msv_master},
"PR with esum constraint (with 40% slack)": {
"proj": n_proj_pr_with_esum,
"msv": n_msv_pr_with_esum,
},
"PR without esum constraint": {
"proj": n_proj_pr_without_esum,
"msv": n_msv_pr_without_esum,
},
}
# set runs as colors
colors = {
"Master": "C0",
"PR with esum constraint (with 40% slack)": "C1",
"PR without esum constraint": "C2",
}
# set network types as line styles
line_styles = {
"proj": "-", # solid line for rolling horizon
"msv": "--", # dashed line for MSV
}
# create plots
for generator_name in generator_names:
fig, axes = plt.subplots(3, 1, figsize=(12, 10), sharex=True, sharey=True)
for idx, (run_label, run_networks) in enumerate(networks.items()):
ax = axes[idx]
for net_type, network in run_networks.items():
dispatch = network.generators_t.p[generator_name]
label = "rolling horizon" if net_type == "proj" else "msv"
ax.plot(
dispatch.index,
dispatch.values,
label=label,
color=colors[run_label],
linestyle=line_styles[net_type],
)
ax.set_title(f"{run_label}")
ax.set_ylabel("Dispatch (MW)")
ax.legend()
# Only set xlabel on the bottom subplot
axes[-1].set_xlabel("Snapshot")
# Add overall title
fig.suptitle(generator_name, fontsize=14, y=0.995)
fig.tight_layout()
filename = f"dispatch_{generator_name.lower().replace(' ', '_')}.png"
plt.savefig(filename, dpi=300, bbox_inches="tight")
print(f"Plot saved as {filename}")
plt.close(fig) |
Conflicts: scripts/cba/solve_cba_network.py
|
@measrainsey that looks great! Ready to be merged from my side! |
|
Thanks @lisazeyen ! In my latest commits I removed the energy budget constraints on the biomass/biogas generators, in addition to the I've updated the release note and the PR description to reflect the final decision we've made to remove the energy budget constraint as well. |
3 participants
Closes:
Changes proposed in this Pull Request
Tasks
[x] Add slack toe_sum_minof dispatch of biomass/biogas generatorsWorkflow
scripts/cba/prepare_rolling_horizon.py: added functionapply_biomass_biogas_bus_marginal_prices()takes a list of carriers, finds the generators with these carriers, takes the marginal prices of buses attached to those generators, and set the corresponding bus' marginal prices as the generator's marginal costs.scripts/cba/solve_cba_network.py: removed thee_sum_minande_sum_maxconstraint on biomass and biogas generatorsconfig files andrules/cba.smk: added a config option calledcba.biomass_biogas_slackthat is passed through toscripts/cba/solve_cba_network.py. Currently the slack is set at 40%.scripts/cba/solve_cba_network.py: use the specifiedcba.biomass_biogas_slackfor the biomass and biogas generators'e_sum_min. Previously, bothe_sum_minande_sum_maxfollow the same exact value (window_energy) which causes infeasibilities at the last rolling horizon. This slack relaxes the constraint so infeasibilities are less likely to happen.Notes
Testing the changes to the marginal costs
Using the debugger and the following networks on my local machine:
Before
apply_biomass_biogas_bus_marginal_prices()is run, this is what the marginal costs and prices look like:Note that before
apply_biomass_biogas_bus_marginal_prices()is run, there is no time series of marginal costs for the 'EU biogas' and 'EU solid biomass' generators, only static marginal costs of 78.82 and 17.34, respectively. There are, however, time series of marginal prices for theEU biogasandEU biomassbuses.After
apply_biomass_biogas_bus_marginal_prices()is applied, the marginal costs look like:After
apply_biomass_biogas_bus_marginal_prices()is applied, the 'EU biogas' andEU solid biomassgenerators have time series marginal costs (n.generators_t.marginal_cost) have been set as then_msv.buses_t.marginal_priceof the corresponding buses (44.95509179 and 17.33346478, respectively).Testing the changes in infeasibilities using the slack
NOTE: this is no longer relevant - this was written during the development of this PR but this was not used in the end. I am keeping this here for tracking this information, but see this comment and the subsequent conversation for why the constraint was removed completely.
Details
On
masterThe following config (called
config/config.bio.yaml) is used (note that thesoc_boundary_carriers: []has to be set because otherwise there is an issue with hydro boundary conditions when using a lower resolution MSV):With the following command:
On the
masterbranch, the project solve optimization would fail due to infeasibilities, around the 50th rolling horizon window:The IIS computes the cause of infeasibility as related to the biomass constraint:
On PR branch
On this
feat/bio-dispatchbranch, the YAML file used has been changed slightly, to include thecba.biomass_biogas_slackconfig option:The same snakemake command is used:
In this branch, all rolling horizon windows solve optimally:
Testing amount of slack needed
To determine the amount of slack used in the default (in this case, 40%), I ran the high-res (1H) CBA workflows on ZIB, under different slack values.
To do so, I first added the following scenarios to
config/scenarios.tyndp.yaml:Then, I created a config file called
config/config.zib.yamlto run the scenarios:Note that here I am only testing on projects t1-t200.
Then, to run the workflow, I ran:
Up until 30% percent, there is at least one project that will still fail due to biomass related infeasibilities.
t16in 2040 is one particular case. For example, with a slack of 30%, thet16-2040 rolling horizon optimization is infeasible in the 52nd window.Checking for files with 30% that end up infeasible:
The log file shows the infeasibility in the 52nd window:
With a slack of 40%, though, the
t16-2040 optimization is successful:Then, with 40% slack, I ran the 1H CBA workflow for all projects (except
t339, as this is still failing in this branch and is being addressed with #751).The YAML used was:
In this case, no files show infeasible/unbounded (checked using
grep -l "infeasible\|unbounded" logs/slurm/*solve_cba_network*slack40*.out 2>/dev/null).Thus, I would suggest we keep 40% as the default slack, with the caveat that we can reduce the default slack in the future as we continue to improve our CBA methodology.
Checklist
[ ] Changed dependencies are added topixi.toml(usingpixi add <dependency-name>).[ ] Changes in configuration options are added inconfig/config.default.yaml.[ ] Changes in configuration options are documented indoc/configtables/*.csv.[ ] Changes in configuration options are added inconfig/test/*.yaml.pixi run -e open-tyndp tyndp-cyears-test).[ ] Open-TYNDP SPDX license header added to all touched files.[ ] For new data sources or versions, these instructions have been followed.[ ] New rules are documented in the appropriatedoc/*.rstfiles.doc/release_notes.rstis added.[ ] Major features are documented with up-to-date information indoc/index.rst.[ ] Module docstrings added to new Python scripts.