refac: restructure CBA outputs to align with SB outputs; add benchmarks for CBA rules

[measrainsey]

Closes:

• Add benchmarks and logs for CBA rules #660
• [SUB] Restructure CBA plots to match structure of SB plots #661

Changes proposed in this Pull Request

PR to restructure the CBA outputs to be more aligned with SB outputs and to add benchmarks for CBA rules

Tasks

Outputs (plots, indicators, etc)

The CBA outputs have been restructured to be more aligned with the SB outputs structure.

Single scenario run (e.g., NT)

Outputs folder results/NT/cba:

results/cba-outputs/NT/cba/
├── all_scenarios.txt
├── graphs/
│   ├── 2030/
│   │   └── summary/   # summary plots
│   │       ├── b1_summary_toot.svg
│   │       ├── b1_summary_pint.svg
│   │       ├── b1_top_4_projects_toot.svg
│   │       ├── b1_top_1_projects_pint.svg
│   │       └── b1_capex_vs_opex_toot.svg
│   └── 2040/
│       └── summary/
│           ├── b1_summary_toot.svg
│           ├── b1_top_5_projects_toot.svg
│           └── b1_capex_vs_opex_toot.svg
│           └── ...
├── validation/ # project-level indicator plots comparing results to TYNDP 2024
│   ├── 2030/
│   │   ├── project_t4_2030.png
│   │   ├── project_t16_2030.png
│   │   ├── project_t28_2030.png
│   │   ├── project_t33_2030.png
│   │   ├── project_t35_2030.png
│   │   └── summary_benchmark_2030.png # summary plot for 2030
│   └── 2040/
│       ├── project_t4_2040.png
│       ├── ...
│       └── summary_benchmark_2040.png
├── results/ # indicator CSVs
│   ├── 2030/
│   │   ├── indicators_2030.csv  # combined indicators, all projects
│   │   ├── project_t4_2030.csv
│   │   ├── project_t16_2030.csv
│   │   ├── project_t28_2030.csv
│   │   ├── project_t33_2030.csv
│   │   └── project_t35_2030.csv
│   └── 2040/
│       ├── indicators_2040.csv
│       └── project_*_2040.csv
└── networks/
    ├── reference_2030.nc # one .nc per project per horizon
    ├── reference_2040.nc
    ├── project_t4_2030.nc
    ├── project_t16_2030.nc
    └── ...                                  

Collection scenario run (e.g., NT-cyears)

Individual climate year, such as NT-cy2009:

results/cba-outputs-cyears/NT-cy2009/cba/
├── graphs/
│   ├── 2030/
│   │   ├── summary/                        # per-horizon indicator plots
│   │   │   └── b1_*.svg
│   │   ├── ensemble_t4_2030.png            # per-project weather benchmark plots
│   │   ├── ensemble_t16_2030.png
│   │   ├── ensemble_t28_2030.png
│   │   ├── ensemble_t33_2030.png
│   │   └── ensemble_t35_2030.png
│   └── 2040/
│       ├── summary/
│       │   └── b1_*.svg
│       └── ensemble_*_2040.png
├── validation/
│   ├── 2030/
│   │   ├── project_*_2030.png
│   │   └── summary_benchmark_2030.png
│   └── 2040/
│       ├── project_*_2040.png
│       └── summary_benchmark_2040.png
├── results/
│   ├── 2030/
│   │   ├── indicators_2030.csv
│   │   └── project_*_2030.csv
│   └── 2040/
│       ├── indicators_2040.csv
│       └── project_*_2040.csv
└── networks/
    ├── reference_2030.nc
    ├── reference_2040.nc
    └── project_*.nc

Collection of all climate years, NT-cyears:

results/cba-outputs-cyears/NT-cyears/cba/
├── all_scenarios.txt
├── graphs/
│   └── all/
│       ├── ensemble_all.png                       # all projects, all horizons
│       ├── ensemble_t4_all_horizons.png            # single project, all horizons
│       ├── ensemble_t16_all_horizons.png
│       ├── ensemble_t28_all_horizons.png
│       ├── ensemble_t33_all_horizons.png
│       └── ensemble_t35_all_horizons.png
├── validation/
│   ├── 2030/
│   │   └── summary_benchmark_2030.png             # cyear-weighted-average benchmark
│   └── 2040/
│       └── summary_benchmark_2040.png
└── results/
    └── all/
        ├── ensemble_indicators_t4_2030.csv         # cyear-weighted-average indicators
        ├── ensemble_indicators_t16_2030.csv
        ├── ensemble_indicators_t28_2030.csv
        ├── ensemble_indicators_t33_2030.csv
        ├── ensemble_indicators_t35_2030.csv
        ├── ensemble_indicators_t4_2040.csv
        ├── ensemble_indicators_t16_2040.csv
        ├── ensemble_indicators_t28_2040.csv
        ├── ensemble_indicators_t33_2040.csv
        └── ensemble_indicators_t35_2040.csv

Logs and benchmarks

Benchmarks have been added for CBA rules that did not have them before (in benchmarks/[run]/peformances/cba/), as well as logs (in logs/[run]/cba/). The exception for this would be the solve_* rules in the CBA workflow, whose logs and benchmarks are saved in the results/ folder of the run instead. For example, solve_cba_msv_extraction:

    log:
        solver=RESULTS + "logs/cba/msv/{planning_horizons}_solver.log",
        memory=RESULTS + "logs/cba/msv/{planning_horizons}_memory.log",
        python=RESULTS + "logs/cba/msv/{planning_horizons}_python.log",
    benchmark:
        RESULTS + "benchmarks/performances/cba/msv/{planning_horizons}"

Open issues

There seems to be duplication of plots in a number of places, when running NT-cyears. For example, in the NT-cy2009 folder:

• The NT-cy2009/cba/graphs/[horizon]/ensemble_*_[horizon].png plots in the look the same as in the NT-cy2009/cba/validation/[horizon]/project_t4_[horizon].png
• The plots in NT-cy2009/cba/graphs/[horizon]/indicators are the same as in NT-cy2009/cba/graphs/[horizon]/summary

Notes

Checklist

Required:

• Security scans show no high-severity bugs, critical vulnerabilities, or exposed secrets.
• Changes are tested locally and behave as expected.
• Code and workflow changes are documented.
• A release note entry is added to doc/release_notes.md.
• The description is human-written and any AI-generated content is marked.

If applicable:

• Changes in configuration options are reflected in scripts/lib/validation.
• Changes in configuration options are added to config/test/*.yaml.
• Multiple climate years test passes locally (pixi run -e open-tyndp tyndp-cyears-test).
• For new data sources or versions, these instructions have been followed.
• Open-TYNDP SPDX license header is added to all touched files.
• Module docstrings are added to new Python scripts.
• New rules are documented in the appropriate doc/*.md files.
• Major features are documented in doc/index.md.

[lisazeyen]

@measrainsey concerning the duplication of the plots:

• can we just delete the rule plot_weather_benchmark: ? it creates the same as the plot_cba_benchmark?

[lisazeyen]

@measrainsey changes look good to me. We can tackle the duplication also in a separate issue. Just as a small clean up (not related to the changes in this PR), can we remove the uncommented rule

# rule plot_all_cba_benchmark:
#     input:
#         indicators=rules.combine_indicators.output.indicators,
#     output:
#         plot_dir=directory(RESULTS + "cba/validation/{planning_horizons}"),
#     script:
#         scripts("cba/plot_benchmark_indicators.py")