-
Notifications
You must be signed in to change notification settings - Fork 8
Description
Bug
I am using this script to migrate a bunch of single file (schema.yml) properties to individiual properties files (1 per model). Many of the declarations in the schema.yml do not include any column details/tests. After running dbt-invoke properites.migrate --models <mode-name> the declaration is removed from schema.yml and a new <model_name>.yml is created successfully. However, if I follow up with dbt-invoke properties.update --models <model-name> the script produces the error:
dbt-invoke | INFO | Searching for matching resources...
dbt-invoke | INFO | Found 1 matching resources in dbt project [REDACTED]
dbt-invoke | INFO | [START] Resource 1 of 1, models/<model-name>.sql
dbt-invoke | ERROR | [FAILURE] Resource 1 of 1, models/<model-name>.sql
dbt-invoke | ERROR | Tracebacks for all failures:
Resource 1 of 1, models/LEVEL_02_MODELS/L02_API_AFFILIATE_DECLINES.SQL
Traceback (most recent call last):
File ".../python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../python3.11/site-packages/dbt_invoke/properties.py", line 714, in _create_property_file
property_file_dict = _structure_property_file_dict(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../python3.11/site-packages/dbt_invoke/properties.py", line 818, in _structure_property_file_dict
for item in property_file_dict[resource_type_plural][0]['columns']
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
File ".../python3.11/site-packages/ruamel/yaml/comments.py", line 853, in __getitem__
return ordereddict.__getitem__(self, key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'columns'
dbt-invoke | INFO | [DONE] Total: 1, Successes: 0, Failures: 1
Workaround
Updating the migrated .yml poperties file to contain a columns section resolves the issue and the properties file can be created successfully. It doesn't require a column name that is relevant to the table/view that it describes, only that the yml tier exists. Note that a column name most be present for the yml to parse correctly. Example
version: 2
models:
- name: <model-name>
description: 'some description that was migrated from the single schema.yml'
columns:
- name: ID
Running dbt-invoke properties.update --models <mode-name> after adding columns: -name: ID:
dbt-invoke | INFO | Searching for matching resources...
dbt-invoke | INFO | Found 1 matching resources in dbt project [REDACTED]
dbt-invoke | INFO | [START] Resource 1 of 1, models/<model-name>.sql
dbt-invoke | INFO | [SUCCESS] Resource 1 of 1, models/<model-name>.sql
dbt-invoke | INFO | [DONE] Total: 1, Successes: 1, Failures: 0