Handling year-specific data

[irm-codebase]

Problem description

Calliope currently lacks the ability to specify yearly data. This, in combination with the lack of non-timeseries data loading (see issue #92) causes projects to balloon in files quite quickly. Eg:

• fuel_constraints_2018.yaml
• fuel_constraints_2017.yaml
• fuel_constraints_2016.yaml
• ...

My proposal has three dimensions:

• Ensure data loading accounts for both location and year variability.
• Agree on how to specify that a value is year/location agnostic (this matters if we want to load data from one file).
• Ensure year data unrelated to the current run is discarded.

Assumptions

For now, let's assume the following order when defining years (not a must, but it makes the most sense to me).

locations:
    CHE:
        techs:
            open_field_pv:
                constraints:
                    energy_cap_equals:
                        years:
                            2020: 0.39  # (100,000 MW)
                            2030: 0.45  # (100,000 MW)
                            2040: 0.50  # (100,000 MW)

Problem description

Key issue here is how to handle the mix of data that is:

• Location agnostic, year agnostic (e.g. lifetime)
• Location specific, year agnostic (maximum capacity, energy_cap_max)
• Location agnostic, year specific (CAPEX, monetary.energy_cap)
• Location specific, year specific (installed capacity, capacity_equals)

As an example of this, see the following table in machine-readable format.

location	technology	type	parameter	year	value	unit	reference	comment
all	open_field_pv	constraints	lifetime	all	55	yr	Schmidt:2019
ALB	open_field_pv	constraints	energy_cap_max	all	300	0.1 TW	Joe:2011	Maximum capacity available
CHE	open_field_pv	constraints	energy_cap_max	all	200	0.1 TW	Schmoe:2011	Maximum capacity available
all	open_field_pv	costs	monetary.energy_cap	2020	85	10,000 EUR2015/MW	DEA:2020
all	open_field_pv	costs	monetary.energy_cap	2030	70	10,000 EUR2015/MW	DEA:2020
ALB	open_field_pv	constraints	energy_cap_equals	2020	1	0.1 TW	IEA:2018
ALB	open_field_pv	constraints	energy_cap_equals	2030	3	0.1 TW	IEA:2018
CHE	open_field_pv	constraints	energy_cap_equals	2020	0.05	0.1 TW	IEA:2018
CHE	open_field_pv	constraints	energy_cap_equals	2030	0.2	0.1 TW	IEA:2018

Proposal

An issue I see in the #92 proposal is that defining rows as techs and columns as a value stops working well if you have two sensitive variables (location, year). We should also have a way of specifying that data should be loaded from a file like the one above, and how to tell Calliope which data is "generic" (applies to all in a given dimension), and which isn't.
Here is a rough example:

data_sources:
    my_file:
        location="data/my_file.csv"
        type: "atomic"  # just to specify it is columns-only. Any other name works too.
        generic: "all"
        columns:
            techs: "technology"
            locations: "location"
            constraints: "parameter" if "type"=="constraints"
            costs: "parameter" if "type"=="costs"
            years: "year"
            value: "value"

locations: data_source=my_file
    CHE:
        techs:
            open_field_pv:
                constraints:
                    energy_cap_equals:
                        years:
                            2020: 0.39  # (100,000 MW) # This is just for show, in reality everything would be fetched from the file!
                            2030: 0.45  # (100,000 MW)
                            2040: 0.50  # (100,000 MW)

Another thing that would be necessary is to discard data that is not related to the current run by specifying the relevant year(s) in model.yaml (to enable us to define year relevant data in one configuration, even if a year index is not yet implemented in the constraints).

model:
    name: 'Euro-Calliope'
    calliope_version: 0.6.8
    timeseries_data_path: './'
    year: 2020
    subset_time: ['2020-01-01', '2020-12-31']