Open
Description
Kia ora, I am trying to use this module to create an external table in an Athena/Glue catalog.
When I apply the config (with dbt run-operation stage_external_sources
) I receive this output:
20:39:46 Running with dbt=1.8.9
20:39:47 Registered adapter: athena=1.8.4
20:39:47 Found 12 models, 5 sources, 626 macros, 5 unit tests
20:39:47 1 of 1 START external source intermediate.test_table
20:39:52 Encountered an error while running operation: Runtime Error
Compilation Error
'dict object' has no attribute 'vals'
> in macro refresh_external_table (macros/common/refresh_external_table.sql)
> called by macro athena__get_external_build_plan (macros/get_external_build_plan.sql)
> called by macro get_external_build_plan (macros/common/get_external_build_plan.sql)
> called by macro stage_external_sources (macros/common/stage_external_sources.sql)
> called by <Unknown>
My sources file is similar to this:
version: 2
sources:
- name: intermediate
tables:
- name: test_table
database: awscatalog
external:
location: '<s3 bucket path>'
row_format: "SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'"
table_properties: ('partition_filtering.enabled' = 'true','projection.day.type' = 'date','projection.day.range' = '2024-02-01,NOW','projection.day.format' = 'yyyy-MM-dd','projection.hour.type' = 'enum','projection.enabled' = 'true','projection.hour.values' = '00,01,02,03,04,05,06,07,08,09,10,11,12,13,14,15,16,17,18,19,20,21,22,23')
partitions:
- name: day
data_type: string
- name: hour
data_type: string
columns:
- name: timestamp
data_type: string
- name: dealid
data_type: string
- name: creativeid
data_type: string
- name: campaignid
data_type: string
- name: slotid
data_type: string
My installed packages file is as so:
packages:
- package: dbt-labs/dbt_utils
version: 1.3.0
- package: dbt-labs/dbt_external_tables
version: [ '0.8.7' ]
- git: https://github.com/dbt-athena/dbt-athena-external-tables.git
revision: 5af3301ce8e56adebaadf09a90be5740aaa15677 # pinned to a specific version rather than main
And my dbt_project.yml
includes:
dispatch:
- macro_namespace: dbt_external_tables
search_order: [dbt_athena_external_tables, dbt_external_tables]
The issue seems to be identical to dbt-labs/dbt-external-tables#208 . From looking at the example models, I tried adding hive_compatible_partitions: true
to the source config, and this cleared the issue and the table was created without issue.
If this is a misconfiguration, I feel the problem needs to be made clearer in the dbt output.
Metadata
Metadata
Assignees
Labels
No labels