refac: restructure CBA outputs to align with SB outputs; add benchmarks for CBA rules by measrainsey · Pull Request #809 · open-energy-transition/open-tyndp

measrainsey · 2026-07-27T08:37:26Z

Closes:

Changes proposed in this Pull Request

PR to restructure the CBA outputs to be more aligned with SB outputs and to add benchmarks for CBA rules

Tasks

Outputs (plots, indicators, etc)

The CBA outputs have been restructured to be more aligned with the SB outputs structure.

Single scenario run (e.g., `NT`)

Outputs folder results/NT/cba:

results/cba-outputs/NT/cba/
├── all_scenarios.txt
├── graphs/
│   ├── 2030/
│   │   └── summary/   # summary plots
│   │       ├── b1_summary_toot.svg
│   │       ├── b1_summary_pint.svg
│   │       ├── b1_top_4_projects_toot.svg
│   │       ├── b1_top_1_projects_pint.svg
│   │       └── b1_capex_vs_opex_toot.svg
│   └── 2040/
│       └── summary/
│           ├── b1_summary_toot.svg
│           ├── b1_top_5_projects_toot.svg
│           └── b1_capex_vs_opex_toot.svg
│           └── ...
├── validation/ # project-level indicator plots comparing results to TYNDP 2024
│   ├── 2030/
│   │   ├── project_t4_2030.png
│   │   ├── project_t16_2030.png
│   │   ├── project_t28_2030.png
│   │   ├── project_t33_2030.png
│   │   ├── project_t35_2030.png
│   │   └── summary_benchmark_2030.png # summary plot for 2030
│   └── 2040/
│       ├── project_t4_2040.png
│       ├── ...
│       └── summary_benchmark_2040.png
├── results/ # indicator CSVs
│   ├── 2030/
│   │   ├── indicators_2030.csv  # combined indicators, all projects
│   │   ├── project_t4_2030.csv
│   │   ├── project_t16_2030.csv
│   │   ├── project_t28_2030.csv
│   │   ├── project_t33_2030.csv
│   │   └── project_t35_2030.csv
│   └── 2040/
│       ├── indicators_2040.csv
│       └── project_*_2040.csv
└── networks/
    ├── reference_2030.nc # one .nc per project per horizon
    ├── reference_2040.nc
    ├── project_t4_2030.nc
    ├── project_t16_2030.nc
    └── ...

Collection scenario run (e.g., `NT-cyears`)

Individual climate year, such as NT-cy2009:

results/cba-outputs-cyears/NT-cy2009/cba/
├── graphs/
│   ├── 2030/
│   │   ├── summary/                        # per-horizon indicator plots
│   │   │   └── b1_*.svg
│   │   ├── ensemble_t4_2030.png            # per-project weather benchmark plots
│   │   ├── ensemble_t16_2030.png
│   │   ├── ensemble_t28_2030.png
│   │   ├── ensemble_t33_2030.png
│   │   └── ensemble_t35_2030.png
│   └── 2040/
│       ├── summary/
│       │   └── b1_*.svg
│       └── ensemble_*_2040.png
├── validation/
│   ├── 2030/
│   │   ├── project_*_2030.png
│   │   └── summary_benchmark_2030.png
│   └── 2040/
│       ├── project_*_2040.png
│       └── summary_benchmark_2040.png
├── results/
│   ├── 2030/
│   │   ├── indicators_2030.csv
│   │   └── project_*_2030.csv
│   └── 2040/
│       ├── indicators_2040.csv
│       └── project_*_2040.csv
└── networks/
    ├── reference_2030.nc
    ├── reference_2040.nc
    └── project_*.nc

Collection of all climate years, NT-cyears:

results/cba-outputs-cyears/NT-cyears/cba/
├── all_scenarios.txt
├── graphs/
│   └── all/
│       ├── ensemble_all.png                       # all projects, all horizons
│       ├── ensemble_t4_all_horizons.png            # single project, all horizons
│       ├── ensemble_t16_all_horizons.png
│       ├── ensemble_t28_all_horizons.png
│       ├── ensemble_t33_all_horizons.png
│       └── ensemble_t35_all_horizons.png
├── validation/
│   ├── 2030/
│   │   └── summary_benchmark_2030.png             # cyear-weighted-average benchmark
│   └── 2040/
│       └── summary_benchmark_2040.png
└── results/
    └── all/
        ├── ensemble_indicators_t4_2030.csv         # cyear-weighted-average indicators
        ├── ensemble_indicators_t16_2030.csv
        ├── ensemble_indicators_t28_2030.csv
        ├── ensemble_indicators_t33_2030.csv
        ├── ensemble_indicators_t35_2030.csv
        ├── ensemble_indicators_t4_2040.csv
        ├── ensemble_indicators_t16_2040.csv
        ├── ensemble_indicators_t28_2040.csv
        ├── ensemble_indicators_t33_2040.csv
        └── ensemble_indicators_t35_2040.csv

Logs and benchmarks

Benchmarks have been added for CBA rules that did not have them before (in benchmarks/[run]/peformances/cba/), as well as logs (in logs/[run]/cba/). The exception for this would be the solve_* rules in the CBA workflow, whose logs and benchmarks are saved in the results/ folder of the run instead. For example, solve_cba_msv_extraction:

    log:
        solver=RESULTS + "logs/cba/msv/{planning_horizons}_solver.log",
        memory=RESULTS + "logs/cba/msv/{planning_horizons}_memory.log",
        python=RESULTS + "logs/cba/msv/{planning_horizons}_python.log",
    benchmark:
        RESULTS + "benchmarks/performances/cba/msv/{planning_horizons}"

Open issues

There seems to be duplication of plots in a number of places, when running NT-cyears. For example, in the NT-cy2009 folder:

The NT-cy2009/cba/graphs/[horizon]/ensemble_*_[horizon].png plots in the look the same as in the NT-cy2009/cba/validation/[horizon]/project_t4_[horizon].png
The plots in NT-cy2009/cba/graphs/[horizon]/indicators are the same as in NT-cy2009/cba/graphs/[horizon]/summary

Notes

Checklist

Required:

Security scans show no high-severity bugs, critical vulnerabilities, or exposed secrets.
Changes are tested locally and behave as expected.
Code and workflow changes are documented.
A release note entry is added to doc/release_notes.md.
The description is human-written and any AI-generated content is marked.

If applicable:

Changes in configuration options are reflected in scripts/lib/validation.
Changes in configuration options are added to config/test/*.yaml.
Multiple climate years test passes locally (pixi run -e open-tyndp tyndp-cyears-test).
For new data sources or versions, these instructions have been followed.
Open-TYNDP SPDX license header is added to all touched files.
Module docstrings are added to new Python scripts.
New rules are documented in the appropriate doc/*.md files.
Major features are documented in doc/index.md.

otherwise it shares output path with `plot_weather_benchmark` rule and that can sometimes lead to issue when running collected scenario

for more information, see https://pre-commit.ci

lisazeyen · 2026-07-31T15:50:38Z

@measrainsey concerning the duplication of the plots:

can we just delete the rule plot_weather_benchmark: ? it creates the same as the plot_cba_benchmark?

lisazeyen

@measrainsey changes look good to me. We can tackle the duplication also in a separate issue. Just as a small clean up (not related to the changes in this PR), can we remove the uncommented rule


# rule plot_all_cba_benchmark:
#     input:
#         indicators=rules.combine_indicators.output.indicators,
#     output:
#         plot_dir=directory(RESULTS + "cba/validation/{planning_horizons}"),
#     script:
#         scripts("cba/plot_benchmark_indicators.py")

measrainsey · 2026-08-03T08:16:25Z

@measrainsey changes look good to me. We can tackle the duplication also in a separate issue. Just as a small clean up (not related to the changes in this PR), can we remove the uncommented rule
# rule plot_all_cba_benchmark:
#     input:
#         indicators=rules.combine_indicators.output.indicators,
#     output:
#         plot_dir=directory(RESULTS + "cba/validation/{planning_horizons}"),
#     script:
#         scripts("cba/plot_benchmark_indicators.py")

Hmm I think so, but maybe we should check to be sure. At the time when we were first implementing the weather years, I had to comment out that rule because it was causing problems -- I created the issue #572 to document that.

I forget what this plot_all_cba_benchmark rule does, but if we now have all the plots we want (individual validation plots, as well as all projects' validation per Andreas' recent implementation) then maybe it doesn't matter and we can safely remove it?

measrainsey and others added 5 commits July 24, 2026 15:54

refac: add logs and benchmarks to CBA rules

d772294

refac: reorganize CBA outputs structure to align with SB conventions

6e822ee

fix: nest plot_indicators output in subdirectory

534e512

otherwise it shares output path with `plot_weather_benchmark` rule and that can sometimes lead to issue when running collected scenario

refac: move summary_benchmark_* plots to cba/validation/

9dbc59c

[pre-commit.ci] auto fixes from pre-commit.com hooks

258ae8b

for more information, see https://pre-commit.ci

This was linked to issues Jul 27, 2026

Add benchmarks and logs for CBA rules #660

Open

[SUB] Restructure CBA plots to match structure of SB plots #661

Open

measrainsey marked this pull request as ready for review July 30, 2026 09:00

tgilon requested a review from lisazeyen July 31, 2026 08:41

measrainsey self-assigned this Jul 31, 2026

measrainsey added the CBA Cost Benefit Analysis label Jul 31, 2026

tgilon mentioned this pull request Jul 31, 2026

Prepare release v0.8 #825

Open

31 tasks

lisazeyen approved these changes Jul 31, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refac: restructure CBA outputs to align with SB outputs; add benchmarks for CBA rules - #809

refac: restructure CBA outputs to align with SB outputs; add benchmarks for CBA rules#809
measrainsey wants to merge 5 commits into
masterfrom
refac/cba-logs-benchmarks

measrainsey commented Jul 27, 2026 •

edited

Loading

Uh oh!

lisazeyen commented Jul 31, 2026

Uh oh!

lisazeyen left a comment

Uh oh!

measrainsey commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

measrainsey commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this Pull Request

Tasks

Outputs (plots, indicators, etc)

Single scenario run (e.g., NT)

Collection scenario run (e.g., NT-cyears)

Logs and benchmarks

Open issues

Notes

Checklist

Uh oh!

lisazeyen commented Jul 31, 2026

Uh oh!

lisazeyen left a comment

Choose a reason for hiding this comment

Uh oh!

measrainsey commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

measrainsey commented Jul 27, 2026 •

edited

Loading

Single scenario run (e.g., `NT`)

Collection scenario run (e.g., `NT-cyears`)