feat: reduce dependency to upstream retrieves

[tgilon]

Closes #775.

Changes proposed in this Pull Request

This PR suggests reducing dependency to upstream retrieves. To do so, conditional input files are defined in multiple rules.

The following retrieve rules have been removed from the dependencies:

• retrieve_country_hdd
• retrieve_gas_infrastructure_data
• retrieve_gem_europe_gas_tracker
• retrieve_ghg_emissions
• retrieve_h2_salt_caverns
• retrieve_ons_lad
• retrieve_powerplants

The conditional input files also removed the need for the following rules:

• build_gas_input_locations
• build_gas_network
• build_heat_totals
• build_salt_cavern_potentials
• cluster_gas_network

Tasks

• Remove dependencies from tyndp-archive (Remove unused dependencies from tyndp-archive #814)
• Contribute partially upstream (refactor: refactor input definitions to use conditional input files PyPSA/pypsa-eur#2248)

Workflow

Adding more conditional inputs reduces the total number of rules by 103 rules, from 2739 to 2636.

Current PR

Details

>>> snakemake -c1 all cba --configfile config/config.tyndp.yaml -n

Job stats:
job                                                    count
---------------------------------------------------  -------
add_brownfield                                            15
add_electricity                                           15
add_existing_baseyear                                     15
add_transmission_projects_and_dlr                         14
all                                                        1
average_indicators_per_project_and_planning_horizon       30
base_network                                              14
build_bidding_zones                                       14
build_biomass_potentials                                  30
build_central_heating_temperature_profiles                30
build_clustered_population_layouts                        15
build_cop_profiles                                        30
build_district_heat_share                                 15
build_electricity_demand_base_tyndp                       30
build_electricity_demand_tyndp                            30
build_energy_totals                                       15
build_eurostat_balances                                   14
build_existing_heating_distribution                       15
build_pemmdb_data                                         30
build_population_layouts                                  15
build_population_weighted_energy_totals                   15
build_powerplants                                          2
build_renewable_profiles_pecd                             60
build_shapes                                              14
build_shipping_demand                                     15
build_snapshot_weightings                                 15
build_statistics                                          30
build_swiss_energy_balances                               14
build_temperature_profiles                                15
build_transport_demand                                    15
build_tyndp_gas_demand                                    30
build_tyndp_h2_demand                                      4
build_tyndp_h2_imports                                    30
build_tyndp_h2_network                                    30
build_tyndp_hydro_profile                                 15
build_tyndp_network                                       14
build_tyndp_offshore_hubs                                 15
build_tyndp_trajectories                                  15
build_tyndp_transmission_projects                         15
cba                                                        1
clean_pecd_data                                          120
clean_projects                                            14
clean_tyndp_electricity_demand                            15
clean_tyndp_h2_imports                                    15
clean_tyndp_h2_storages                                   30
clean_tyndp_hydro_inflows                                150
clean_tyndp_indicators                                    10
clean_tyndp_output_benchmark                              26
clean_tyndp_report_benchmark                              15
clean_tyndp_smr                                           30
clean_tyndp_vp_data                                       15
cluster_network                                           14
collect_cba_scenario                                      15
combine_indicators                                         2
fix_reference_sb_to_cba                                   30
launch_explorer                                           15
make_benchmark                                            15
make_cumulative_costs                                     15
make_global_summary                                       15
make_indicators                                          100
make_summary                                              30
plot_balance_map                                         150
plot_balance_map_interactive                             150
plot_balance_timeseries                                   30
plot_base_hydrogen_network                                30
plot_base_network                                         15
plot_base_offshore_network                                60
plot_benchmark                                            15
plot_cba_benchmark                                        10
plot_cop_profiles                                         30
plot_heatmap_timeseries                                   30
plot_hydrogen_network                                     30
plot_indicators                                            2
plot_interactive_bus_balance                              30
plot_offshore_network                                     60
plot_power_network                                        30
plot_power_network_clustered                              15
plot_summary                                              15
plot_summary_projects_benchmark                            8
prepare_network                                           15
prepare_project                                          100
prepare_reference                                         20
prepare_rolling_horizon                                   20
prepare_sector_network                                    30
process_cost_data                                         45
retrieve_cutout                                            2
simplify_network                                          14
simplify_sb_network                                       20
solve_cba_msv_extraction                                  20
solve_cba_network                                        100
solve_cba_reference_network                               20
solve_sector_network_myopic                               30
summarize_all_indicators                                   3
summarize_indicators_per_project                          15
temporal_aggregation                                      30
total                                                   2636

On fix/796-combine-indicators

Details

>>> snakemake -c1 all cba --configfile config/config.tyndp.yaml -n

Job stats:
job                                                    count                                    
---------------------------------------------------  -------                                    
add_brownfield                                            15                                    
add_electricity                                           15                                    
add_existing_baseyear                                     15                                    
add_transmission_projects_and_dlr                         14                                    
all                                                        1                                    
average_indicators_per_project_and_planning_horizon       30                                    
base_network                                              14                                    
build_bidding_zones                                       14                                    
build_biomass_potentials                                  30                                    
build_central_heating_temperature_profiles                30                                    
build_clustered_population_layouts                        15                                    
build_cop_profiles                                        30                                    
build_district_heat_share                                 15                                    
build_electricity_demand_base_tyndp                       30                                    
build_electricity_demand_tyndp                            30                                    
build_energy_totals                                       15                                    
build_eurostat_balances                                   14                                    
build_existing_heating_distribution                       15                                    
build_gas_input_locations                                 15                                    
build_gas_network                                         15                                    
build_heat_totals                                         15                                    
build_pemmdb_data                                         30                                    
build_population_layouts                                  15                                    
build_population_weighted_energy_totals                   30                                    
build_powerplants                                         15                                    
build_renewable_profiles_pecd                             60                                    
build_salt_cavern_potentials                              15                                    
build_shapes                                              14                                    
build_shipping_demand                                     15                                    
build_snapshot_weightings                                 15                                    
build_statistics                                          30                                    
build_swiss_energy_balances                               14                                    
build_temperature_profiles                                15                                    
build_transport_demand                                    15                                    
build_tyndp_gas_demand                                    30                                    
build_tyndp_h2_demand                                      4                                    
build_tyndp_h2_imports                                    30                                    
build_tyndp_h2_network                                    30                                    
build_tyndp_hydro_profile                                 15                                    
build_tyndp_network                                       14                                    
build_tyndp_offshore_hubs                                 15                                    
build_tyndp_trajectories                                  15                                    
build_tyndp_transmission_projects                         15                                    
cba                                                        1                                    
clean_pecd_data                                          120                                    
clean_projects                                            14                                    
clean_tyndp_electricity_demand                            15                                    
clean_tyndp_h2_imports                                    15                                    
clean_tyndp_h2_storages                                   30                                    
clean_tyndp_hydro_inflows                                150                                    
clean_tyndp_indicators                                    10                                    
clean_tyndp_output_benchmark                              26                                    
clean_tyndp_report_benchmark                              15                                    
clean_tyndp_smr                                           30                                    
clean_tyndp_vp_data                                       15                                    
cluster_gas_network                                       15                                    
cluster_network                                           14                                    
collect_cba_scenario                                      15                                    
combine_indicators                                         2                                    
fix_reference_sb_to_cba                                   30                                    
launch_explorer                                           15                                    
make_benchmark                                            15                                    
make_cumulative_costs                                     15                                    
make_global_summary                                       15                                    
make_indicators                                          100                                    
make_summary                                              30                                    
plot_balance_map                                         150                                    
plot_balance_map_interactive                             150                                    
plot_balance_timeseries                                   30                                    
plot_base_hydrogen_network                                30                                    
plot_base_network                                         15                                    
plot_base_offshore_network                                60                                    
plot_benchmark                                            15                                    
plot_cba_benchmark                                        10                                    
plot_cop_profiles                                         30                                    
plot_heatmap_timeseries                                   30                                    
plot_hydrogen_network                                     30                                    
plot_indicators                                            2                                    
plot_interactive_bus_balance                              30                                    
plot_offshore_network                                     60                                    
plot_power_network                                        30                                    
plot_power_network_clustered                              15                                    
plot_summary                                              15                                    
plot_summary_projects_benchmark                            8                                    
prepare_network                                           15                                    
prepare_project                                          100                                    
prepare_reference                                         20                                    
prepare_rolling_horizon                                   20                                    
prepare_sector_network                                    30                                    
process_cost_data                                         45                                    
retrieve_cutout                                            2                                    
simplify_network                                          14                                    
simplify_sb_network                                       20                                    
solve_cba_msv_extraction                                  20                                    
solve_cba_network                                        100                                    
solve_cba_reference_network                               20                                    
solve_sector_network_myopic                               30                                    
summarize_all_indicators                                   3                                    
summarize_indicators_per_project                          15                                    
temporal_aggregation                                      30                                    
total                                                   2739

Open issues

The issue #775 also lists other dependencies. These dependencies would require more invasive work, since the files are actually used to build the models.

Notes

Checklist

Required:

• Security scans show no high-severity bugs, critical vulnerabilities, or exposed secrets.
• Changes are tested locally and behave as expected.
• Code and workflow changes are documented.
• A release note entry is added to doc/release_notes.md.
• The description is human-written and any AI-generated content is marked.

If applicable:

• Changes in configuration options are reflected in scripts/lib/validation.
• Changes in configuration options are added to config/test/*.yaml.
• Multiple climate years test passes locally (pixi run -e open-tyndp tyndp-cyears-test).
• For new data sources or versions, these instructions have been followed.
• Open-TYNDP SPDX license header is added to all touched files.
• Module docstrings are added to new Python scripts.
• New rules are documented in the appropriate doc/*.md files.
• Major features are documented in doc/index.md.

[daniel-rdt]

Thank you @tgilon for this refactoring! This will already improve the dependencies a lot and make our workflow as lean as we can currently make it without many disruptive changes. I have some minor comments and one requested change for the powerplants file in add_electricity.

[daniel-rdt]

Thank you @tgilon for the adjustments. This is looking good now and I see that the CI also passes again with your latest fix to the input_gas_network logic 🙌