Skip to content

Error applying external table with partitions without hive_compatible_partitions=true #9

Open
@cloventt

Description

@cloventt

Kia ora, I am trying to use this module to create an external table in an Athena/Glue catalog.

When I apply the config (with dbt run-operation stage_external_sources) I receive this output:

20:39:46  Running with dbt=1.8.9
20:39:47  Registered adapter: athena=1.8.4
20:39:47  Found 12 models, 5 sources, 626 macros, 5 unit tests
20:39:47  1 of 1 START external source intermediate.test_table
20:39:52  Encountered an error while running operation: Runtime Error
  Compilation Error
    'dict object' has no attribute 'vals'

    > in macro refresh_external_table (macros/common/refresh_external_table.sql)
    > called by macro athena__get_external_build_plan (macros/get_external_build_plan.sql)
    > called by macro get_external_build_plan (macros/common/get_external_build_plan.sql)
    > called by macro stage_external_sources (macros/common/stage_external_sources.sql)
    > called by <Unknown>

My sources file is similar to this:

version: 2

sources:
  - name: intermediate
    tables:
      - name: test_table
        database: awscatalog
        external:
          location: '<s3 bucket path>'
          row_format: "SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'"
          table_properties: ('partition_filtering.enabled' = 'true','projection.day.type' = 'date','projection.day.range' = '2024-02-01,NOW','projection.day.format' = 'yyyy-MM-dd','projection.hour.type' = 'enum','projection.enabled' = 'true','projection.hour.values' = '00,01,02,03,04,05,06,07,08,09,10,11,12,13,14,15,16,17,18,19,20,21,22,23')
          partitions:
            - name: day
              data_type: string
            - name: hour
              data_type: string
        columns:
          - name: timestamp
            data_type: string
          - name: dealid
            data_type: string
          - name: creativeid
            data_type: string
          - name: campaignid
            data_type: string
          - name: slotid
            data_type: string

My installed packages file is as so:

packages:
  - package: dbt-labs/dbt_utils
    version: 1.3.0
  - package: dbt-labs/dbt_external_tables
    version: [ '0.8.7' ]
  - git: https://github.com/dbt-athena/dbt-athena-external-tables.git
    revision: 5af3301ce8e56adebaadf09a90be5740aaa15677  # pinned to a specific version rather than main

And my dbt_project.yml includes:

dispatch:
  - macro_namespace: dbt_external_tables
    search_order: [dbt_athena_external_tables, dbt_external_tables]

The issue seems to be identical to dbt-labs/dbt-external-tables#208 . From looking at the example models, I tried adding hive_compatible_partitions: true to the source config, and this cleared the issue and the table was created without issue.

If this is a misconfiguration, I feel the problem needs to be made clearer in the dbt output.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions