[WIP] Change data input to have prefix 'input_'

Author: abelsiqueira

benchmark/benchmarks.jl

@@ -19,6 +19,7 @@ function input_setup()
         connection,
         input_folder;
         schemas = TulipaEnergyModel.schema_per_table_name,
+        table_name_prefix = "input_",
     )
     return connection
 end

benchmark/profiling/common.jl

@@ -7,7 +7,12 @@ dir = norse_dir
 
 function _read_dir_and_return_connection(dir)
     con = DBInterface.connect(DuckDB.DB)
-    TulipaIO.read_csv_folder(con, dir; schemas = TulipaEnergyModel.schema_per_table_name)
+    TulipaIO.read_csv_folder(
+        con,
+        dir;
+        schemas = TulipaEnergyModel.schema_per_table_name,
+        table_name_prefix = "input_",
+    )
 
     return con
 end

docs/src/20-how-to-use.md

@@ -27,7 +27,7 @@ All tests should pass.
 ## Finding an input parameter
 
 !!! tip "Are you looking for an input parameter?"
-    Please visit the [Model Parameters](@ref schemas) section for a description and location of all model input parameters.
+    Please visit the [Data](@ref data) section for a description and location of all model input parameters.
 
 ## Running a Scenario
 

docs/src/30-concepts.md

@@ -638,7 +638,7 @@ unstacked_map[!,["k=1", "k=2", "k=3"]] = convert.(Float64, unstacked_map[!,["k=1
 unstacked_map # hide
 ```
 
-The file `assets-timeframe-partitions` has the information on how often we want to evaluate the inter-temporal constraints that combine the information of the representative periods. In this example, the file is missing in the folder, meaning that the default of a `uniform` distribution of one period will be use in the model, see [model parameters](@ref schemas) section. This assumption implies that the model will check the inter-storage level every day of the week timeframe.
+The file `assets-timeframe-partitions` has the information on how often we want to evaluate the inter-temporal constraints that combine the information of the representative periods. In this example, the file is missing in the folder, meaning that the default of a `uniform` distribution of one period will be use in the model, see the [schemas](@ref schemas) section. This assumption implies that the model will check the inter-storage level every day of the week timeframe.
 
 !!! info
     For the sake of simplicity, we show how using three representative days can recover part of the chronological information of one week. The same method can be applied to more representative periods to analyze the seasonality across a year or longer timeframe.
@@ -651,7 +651,12 @@ using DuckDB, TulipaIO, TulipaEnergyModel
 input_dir = "../../test/inputs/Storage" # hide
 # input_dir should be the path to the Storage example
 connection = DBInterface.connect(DuckDB.DB)
-read_csv_folder(connection, input_dir; schemas = TulipaEnergyModel.schema_per_table_name)
+read_csv_folder(
+    connection,
+    input_dir;
+    schemas = TulipaEnergyModel.schema_per_table_name,
+    table_name_prefix = "input_",
+)
 energy_problem = run_scenario(connection)
 ```
 

docs/src/50-schemas.md

@@ -1,6 +1,56 @@
-# [Model Parameters](@id schemas)
+# [Data pipeline/workflow](@id data)
 
-The optimization model parameters with the input data must follow the schema below for each table. To create these tables we currently use CSV files that follow this same schema and then convert them into tables using TulipaIO, as shown in the basic example of the [Tutorials](@ref basic-example) section.
+---
+TODO:
+
+- diagrams
+- Replace
+  > To create these tables we currently use CSV files that follow this same schema and then convert them into tables using TulipaIO, as shown in the basic example of the [Tutorials](@ref basic-example) section.
+- Review below
+
+---
+
+Tulipa uses a DuckDB database to store the input data, the representation of variables, constraints, and other internal tables, as well as the output.
+This database is informed through the `connection` argument in various parts of the API. Most notably, for [`run_scenario`](@ref) and [`EnergyProblem`](@ref).
+
+## [Minimum data and using defaults](@id minimum_data)
+
+Since `TulipaEnergyModel` is at a late stage in the workflow, its input data requirements are stricter.
+Therefore, the input data required by the Tulipa model must follow the schema in the follow section.
+
+Dealing with defaults is hard. A missing value might represent two different things to different people. That is why we require the tables to be complete.
+However, we also understand that it is not reasonable to expect people to fill a lot of things that they don't need for their models.
+Therefore, we have created the function [`populate_with_defaults!`](@ref) to fill the remaining columns of your tables with default values.
+
+To know the defaults, check the table [Schemas](@ref schemas) below.
+
+!!! warning "Beware implicit assumptions"
+    When data is missing and you automatically fill it with defaults, beware of your assumptions on what that means.
+    Check what are the default values and decide if you want to use them or not.
+    If you think a default does not make sense, open an issue, or a discussion thread.
+
+### Example of using `populate_with_defaults!`
+
+```@example
+using TulipaEnergyModel, TulipaIO, DuckDB
+```
+
+## Namespaces
+
+After creating a `connection` and loading data in a way that follows the schema (see the previous section on [minimum data](@ref minimum_data)), then Tulipa will create tables to handle the model data and various internal tables.
+To differentiate between these tables, we use a prefix. This should also help differentiate between the data you might want to create yourself.
+Here are the different namespaces:
+
+- `input_`: Tables expected by `TulipaEnergyModel`.
+- `var_`: Variable indices.
+- `cons_`: Constraints indices.
+- `expr_`: Expressions indices.
+- `resolution_`: Unrolled partition blocks of assets and flows.
+- `t_*`: Temporary tables.
+
+## [Schemas](@id schemas)
+
+The optimization model parameters with the input data must follow the schema below for each table.
 
 The schemas can be found in the `input-schemas.json`. For more advanced users, they can also access the schemas at any time after loading the package by typing `TulipaEnergyModel.schema_per_table_name` in the Julia console. Here is the complete list of model parameters in the schemas per table (or CSV file):
 

src/constraints/capacity.jl

@@ -376,19 +376,19 @@ function _append_capacity_data_to_indices_compact_method(connection, table_name)
             ANY_VALUE(asset_commission.investment_limit) AS investment_limit,
             ANY_VALUE(assets_profiles.profile_name) AS profile_name,
         FROM cons_$table_name AS cons
-        LEFT JOIN asset
+        LEFT JOIN input_asset as asset
             ON cons.asset = asset.asset
-        LEFT JOIN asset_commission
+        LEFT JOIN input_asset_commission as asset_commission
             ON cons.asset = asset_commission.asset
             AND cons.year = asset_commission.commission_year
         LEFT JOIN expr_available_asset_units_compact_method AS expr_avail
             ON cons.asset = expr_avail.asset
             AND cons.year = expr_avail.milestone_year
-        LEFT OUTER JOIN assets_profiles
+        LEFT OUTER JOIN input_assets_profiles as assets_profiles
             ON cons.asset = assets_profiles.asset
             AND cons.year = assets_profiles.commission_year
             AND assets_profiles.profile_type = 'availability'
-        LEFT OUTER JOIN assets_profiles AS avail_profile
+        LEFT OUTER JOIN input_assets_profiles AS avail_profile
             ON cons.asset = avail_profile.asset
             AND expr_avail.commission_year = avail_profile.commission_year
             AND avail_profile.profile_type = 'availability'
@@ -420,19 +420,19 @@ function _append_capacity_data_to_indices_simple_method(connection, table_name)
             asset_commission.investment_limit AS investment_limit,
             assets_profiles.profile_name AS profile_name,
         FROM cons_$table_name AS cons
-        LEFT JOIN asset
+        LEFT JOIN input_asset as asset
             ON cons.asset = asset.asset
-        LEFT JOIN asset_commission
+        LEFT JOIN input_asset_commission as asset_commission
             ON cons.asset = asset_commission.asset
             AND cons.year = asset_commission.commission_year
         LEFT JOIN expr_available_asset_units_simple_method AS expr_avail
             ON cons.asset = expr_avail.asset
             AND cons.year = expr_avail.milestone_year
-        LEFT OUTER JOIN assets_profiles
+        LEFT OUTER JOIN input_assets_profiles as assets_profiles
             ON cons.asset = assets_profiles.asset
             AND cons.year = assets_profiles.commission_year
             AND assets_profiles.profile_type = 'availability'
-        LEFT OUTER JOIN assets_profiles AS avail_profile
+        LEFT OUTER JOIN input_assets_profiles AS avail_profile
             ON cons.asset = avail_profile.asset
             AND expr_avail.commission_year = avail_profile.commission_year
             AND avail_profile.profile_type = 'availability'

src/constraints/consumer.jl

@@ -66,12 +66,12 @@ function _create_consumer_table(connection)
             asset_milestone.peak_demand,
             assets_profiles.profile_name,
         FROM cons_balance_consumer AS cons
-        LEFT JOIN asset
+        LEFT JOIN input_asset as asset
             ON cons.asset = asset.asset
-        LEFT JOIN asset_milestone
+        LEFT JOIN input_asset_milestone as asset_milestone
             ON cons.asset = asset_milestone.asset
             AND cons.year = asset_milestone.milestone_year
-        LEFT OUTER JOIN assets_profiles
+        LEFT OUTER JOIN input_assets_profiles as assets_profiles
             ON cons.asset = assets_profiles.asset
             AND cons.year = assets_profiles.commission_year
             AND assets_profiles.profile_type = 'demand' -- This must be a ON condition not a where (note 1)

src/constraints/energy.jl

@@ -71,10 +71,10 @@ function _append_energy_data_to_indices(connection, table_name, min_or_max)
             asset_milestone.$(min_or_max)_energy_timeframe_partition,
             assets_timeframe_profiles.profile_name
         FROM cons_$table_name AS cons
-        LEFT JOIN asset_milestone
+        LEFT JOIN input_asset_milestone as asset_milestone
             ON cons.asset = asset_milestone.asset
             AND cons.year = asset_milestone.milestone_year
-        LEFT OUTER JOIN assets_timeframe_profiles
+        LEFT OUTER JOIN input_assets_timeframe_profiles as assets_timeframe_profiles
             ON cons.asset = assets_timeframe_profiles.asset
             AND cons.year = assets_timeframe_profiles.commission_year
             AND assets_timeframe_profiles.profile_type = '$(min_or_max)_energy'

src/constraints/group.jl

@@ -70,9 +70,9 @@ function _get_assets_in_group(connection, group)
             asset.group,
             asset.capacity,
         FROM var_assets_investment AS var
-        JOIN asset
+        JOIN input_asset as asset
             ON var.asset = asset.asset
-        JOIN group_asset
+        JOIN input_group_asset as group_asset
             ON asset.group = group_asset.name
         WHERE asset.group IS NOT NULL
               AND  asset.group = '$group'

src/constraints/ramping-and-unit-commitment.jl

@@ -271,9 +271,9 @@ function _append_ramping_data_to_indices(connection, table_name)
             asset.max_ramp_down,
             assets_profiles.profile_name
         FROM cons_$table_name AS cons
-        LEFT JOIN asset
+        LEFT JOIN input_asset as asset
             ON cons.asset = asset.asset
-        LEFT OUTER JOIN assets_profiles
+        LEFT OUTER JOIN input_assets_profiles as assets_profiles
             ON cons.asset = assets_profiles.asset
             AND cons.year = assets_profiles.commission_year
             AND assets_profiles.profile_type = 'availability'
@@ -299,7 +299,7 @@ function _append_available_units_data_compact_method(connection, table_name)
         LEFT JOIN expr_available_asset_units_compact_method AS expr_avail
             ON cons.asset = expr_avail.asset
             AND cons.year = expr_avail.milestone_year
-        LEFT JOIN asset
+        LEFT JOIN input_asset as asset
             ON cons.asset = asset.asset
         WHERE asset.investment_method = 'compact'
         GROUP BY cons.id
@@ -324,7 +324,7 @@ function _append_available_units_data_simple_method(connection, table_name)
         LEFT JOIN expr_available_asset_units_simple_method AS expr_avail
             ON cons.asset = expr_avail.asset
             AND cons.year = expr_avail.milestone_year
-        LEFT JOIN asset
+        LEFT JOIN input_asset as asset
             ON cons.asset = asset.asset
         WHERE asset.investment_method in ('simple', 'none')
         ORDER BY cons.id

src/constraints/storage.jl

@@ -270,7 +270,7 @@ function _append_storage_data_to_indices(connection, table_name)
                 cons.period_block_start,
                 SUM(mapping.num_timesteps) AS duration_period_block
             FROM cons_balance_storage_over_clustered_year AS cons
-            LEFT JOIN timeframe_data AS mapping
+            LEFT JOIN input_timeframe_data AS mapping
                 ON mapping.year = cons.year
                 AND mapping.period BETWEEN cons.period_block_start AND cons.period_block_end
             GROUP BY cons.asset, cons.year, cons.period_block_start
@@ -301,26 +301,26 @@ function _append_storage_data_to_indices(connection, table_name)
             min_storage_level_profile.profile_name AS min_storage_level_profile_name,
             expr_avail.id AS avail_energy_capacity_id
         FROM cons_$table_name AS cons
-        LEFT JOIN asset
+        LEFT JOIN input_asset as asset
             ON cons.asset = asset.asset
-        LEFT JOIN asset_commission
+        LEFT JOIN input_asset_commission as asset_commission
             ON cons.asset = asset_commission.asset
             AND cons.year = asset_commission.commission_year
-        LEFT JOIN asset_milestone
+        LEFT JOIN input_asset_milestone as asset_milestone
             ON cons.asset = asset_milestone.asset
             AND cons.year = asset_milestone.milestone_year
         LEFT JOIN expr_available_energy_capacity_simple_method AS expr_avail
             ON cons.asset = expr_avail.asset
             AND cons.year = expr_avail.milestone_year
-        LEFT OUTER JOIN assets_profiles AS inflows_profile
+        LEFT OUTER JOIN input_assets_profiles AS inflows_profile
             ON cons.asset = inflows_profile.asset
             AND cons.year = inflows_profile.commission_year
             AND inflows_profile.profile_type = 'inflows'
-        LEFT OUTER JOIN assets_profiles AS max_storage_level_profile
+        LEFT OUTER JOIN input_assets_profiles AS max_storage_level_profile
             ON cons.asset = max_storage_level_profile.asset
             AND cons.year = max_storage_level_profile.commission_year
             AND max_storage_level_profile.profile_type = 'max_storage_level'
-        LEFT OUTER JOIN assets_profiles AS min_storage_level_profile
+        LEFT OUTER JOIN input_assets_profiles AS min_storage_level_profile
             ON cons.asset = min_storage_level_profile.asset
             AND cons.year = min_storage_level_profile.commission_year
             AND min_storage_level_profile.profile_type = 'min_storage_level'

src/constraints/transport.jl

@@ -108,14 +108,14 @@ function _append_transport_data_to_indices(connection)
             expr_avail.id AS avail_id,
             flows_profiles.profile_name AS profile_name,
         FROM cons_transport_flow_limit_simple_method AS cons
-        LEFT JOIN flow
+        LEFT JOIN input_flow as flow
             ON cons.from_asset = flow.from_asset
             AND cons.to_asset = flow.to_asset
         LEFT JOIN expr_available_flow_units_simple_method AS expr_avail
             ON cons.from_asset = expr_avail.from_asset
             AND cons.to_asset = expr_avail.to_asset
             AND cons.year = expr_avail.milestone_year
-        LEFT OUTER JOIN flows_profiles
+        LEFT OUTER JOIN input_flows_profiles as flows_profiles
             ON cons.from_asset = flows_profiles.from_asset
             AND cons.to_asset = flows_profiles.to_asset
             AND cons.year = flows_profiles.year