Skip to content

Hydro horizons missing from temporal causes tab file to disappear #1102

@misyntropy

Description

@misyntropy

Relevant to #819 and ultimately #6 .

gen_hydro projects are linked to horizons in inputs_project_hydro_operational_characteristics. The way hydro data is read from the database is to pick the intersection of these horizons, with those specified in the inputs_temporal_horizons_timepoints table :

-- Limit to bt-horizons from this temporal scenario ID
JOIN (
SELECT DISTINCT balancing_type_horizon, horizon
FROM inputs_temporal_horizon_timepoints
WHERE temporal_scenario_id = {subscenarios.TEMPORAL_SCENARIO_ID}
AND subproblem_id = {subproblem}
AND stage_id = {stage}
) as hrz_tbl
ON (
balancing_type_project = balancing_type_horizon
AND prj_tbl.horizon = hrz_tbl.horizon
)

When horizons are missing from the temporal table, the intersection is also the null set and therefore the tab files aren't even written:

# Only write if we have data
data_list = [row for row in data.fetchall()]
if data_list:
with open(out_file, append_mode, newline="") as f:

This results in a file not found error:

ERROR:root:[Errno 2] No such file or directory: '../scenarios_etc-dev/2horizons_w_hydro_w_balancing_types/inputs/hydro_conventional_horizon_params.tab' and FileNotFoundError: [Errno 2] No such file or directory: '../scenarios_etc-dev/2horizons_w_hydro_w_balancing_types/inputs/hydro_conventional_horizon_params.tab' .

I see three equally valid ways to handle this:

  1. Make sure all empty files are written, with (perhaps?) the result that the hydro projects are never simulated. This aligns with existing practice of silently dropping mismatched horizons.
  2. Make mismatched horizons an error, and use LEFT JOIN instead of INNER JOIN while fetching the hydro opchar data. This entails a lot of changes throughout the test suite.
  3. Keep existing behaviour as by-design.

I am in the process of writing validation to catch this case, but before that wanted to understand if current operational behaviour is intended.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions