Skip to content

insert+overwrite incremental strategy for iceberg tables incorrectly uses a partition clause #204

@amaralvieira

Description

@amaralvieira

From the docs:

Since Iceberg uses hidden partitioning it means you don't need a partition clause in your INSERT statements.

The current get_insert_overwrite_sql macro throws an error whenever the iceberg table uses partition transforms. This can be avoided altogether by omitting the partition clause. For example:

    {% if table_type == 'iceberg' -%}
        {%- if raw_strategy == 'insert_overwrite' %}
            insert overwrite {{ target }} ({{ dest_cols_csv }})
            select {{ dest_cols_csv }} from {{ source }}
        {%- elif raw_strategy == 'append' %}
            insert into {{ target }} ({{ dest_cols_csv }})
            select {{ dest_cols_csv }} from {{ source }}
        {%- endif %}
    {% else %}

It would probably be a good idea to add a check somewhere for the case someone tries to set a INSERT OVERWRITE incremental strategy for tables that use the BUCKET transform (as it is not allowed).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions