Skip to content

feat(athena): add build_strategy config for incremental and table materializations #1830

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dtaniwaki wants to merge 10 commits into
base: main
Choose a base branch
from
Open

feat(athena): add build_strategy config for incremental and table materializations #1830

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
5bd01f8
feat(athena): add build_with_subquery config to skip tmp table staging
dtaniwaki Apr 6, 2026
5e0f4ec
Update dbt-athena/src/dbt/adapters/athena/impl.py
dtaniwaki Apr 6, 2026
c31f06c
Merge branch 'main' into feat/athena-build-with-subquery
dtaniwaki Apr 6, 2026
f95a3ae
fix(athena): wrap build_with_subquery source SQL on its own line
dtaniwaki Apr 8, 2026
c2217e9
test(athena): refactor build_with_subquery SQL assertions to heredoc …
dtaniwaki Apr 8, 2026
7e23811
test(athena): drop blank lines in build_with_subquery SQL assertion n…
dtaniwaki Apr 8, 2026
bd9f34f
feat(athena): support build_with_subquery for table materialization (…
dtaniwaki Apr 29, 2026
221b1cd
refactor(athena): rename build_with_subquery (bool) to build_strategy…
dtaniwaki Apr 29, 2026
eba662d
test(athena): expand build_strategy functional coverage with record v…
dtaniwaki Apr 29, 2026
244fa2c
fix(athena): reword force_batch incompatibility error per review feed…
dtaniwaki Apr 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions dbt-athena/.changes/unreleased/Features-20260406-225458.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: build_with_subquery config option to skip tmp table staging for incremental merge and append strategies
time: 2026-04-06T22:54:58.739286+09:00
custom:
Author: dtaniwaki
Issue: "1829"
4 changes: 4 additions & 0 deletions dbt-athena/src/dbt/adapters/athena/impl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,9 @@ class AthenaConfig(AdapterConfig):
force_batch: Skip creating the table as ctas and run the operation directly in batch insert mode.
unique_tmp_table_suffix: Enforce the use of a unique id as tmp table suffix instead of __dbt_tmp.
temp_schema: Define in which schema to create temporary tables used in incremental runs.
build_with_subquery: Use a subquery directly instead of staging data into __dbt_tmp.
Creates an empty tmp table (WITH NO DATA) for schema comparison, then applies via subquery.
Supported for Iceberg merge and append strategies, and Hive append. Incompatible with force_batch.
"""

@iconara iconara Apr 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you aren't familiar with dbt-athena since before this change, you may be confused by the naming of this property. The name is a response to the current default, but the current default is actually not the default behavior that you would expect.

It's natural to add a feature with a config that enables it, but I'm thinking that in this case we are adding something that probably should have been the default mode, and it's only for historical reasons it ended up being added later. At least to me, using a temp table is the alternative way of doing this.

One way to convey this is to use a config that is not true or false, but enables one of two (or more) modes, like incremental_strategy with values "subquery" or "tmp_table" (default).

@dtaniwaki dtaniwaki Apr 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds reasonable. incremental_strategy is the same as dbt-core's config like incremental_strategy: insert_overwrite. How about build_strategy?

@dtaniwaki dtaniwaki Apr 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about applying this mode to table materialization? This behavior benefits all the materialization types rather than the incremental materialization.

I made another PR over this PR in dtaniwaki#12.

@iconara iconara Apr 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_strategy sounds good to me, that works for table materialization too.


work_group: Optional[str] = None
Expand All @@ -127,6 +130,7 @@ class AthenaConfig(AdapterConfig):
force_batch: bool = False
unique_tmp_table_suffix: bool = False
temp_schema: Optional[str] = None
build_with_subquery: bool = False


class AthenaAdapter(SQLAdapter):
Expand Down
31 changes: 24 additions & 7 deletions dbt-athena/src/dbt/include/athena/macros/materializations/models/incremental/helpers.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
target_relation,
existing_relation,
force_batch,
source_sql=none,
statement_name="main"
)
%}
Expand All @@ -65,18 +66,34 @@
{% if force_batch %}
{% do batch_incremental_insert(tmp_relation, target_relation, dest_cols_csv) %}
{% else %}
{%- set insert_full -%}
insert into {{ target_relation }} ({{ dest_cols_csv }})
(
select {{ dest_cols_csv }}
from {{ tmp_relation }}
);
{%- endset -%}
{%- if source_sql is not none -%}
{%- set insert_full -%}
insert into {{ target_relation }} ({{ dest_cols_csv }})
(
select {{ dest_cols_csv }}
from (
{{ source_sql }}
) _dbt_sbq
);
{%- endset -%}
{%- else -%}
{%- set insert_full -%}
insert into {{ target_relation }} ({{ dest_cols_csv }})
(
select {{ dest_cols_csv }}
from {{ tmp_relation }}
);
{%- endset -%}
{%- endif -%}

{%- set query_result = adapter.run_query_with_partitions_limit_catching(insert_full) -%}
{%- do log('QUERY RESULT: ' ~ query_result) -%}
{%- if query_result == 'TOO_MANY_OPEN_PARTITIONS' -%}
{%- if source_sql is not none -%}
{% do exceptions.raise_compiler_error('Runtime error: TOO_MANY_OPEN_PARTITIONS encountered with build_with_subquery=True. Disable build_with_subquery to enable automatic batching.') %}
{%- else -%}
{% do batch_incremental_insert(tmp_relation, target_relation, dest_cols_csv) %}
{%- endif -%}
{%- endif -%}
{%- endif -%}

Expand Down
125 changes: 89 additions & 36 deletions dbt-athena/src/dbt/include/athena/macros/materializations/models/incremental/incremental.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
{% set lf_grants = config.get('lf_grants') %}
{% set partitioned_by = config.get('partitioned_by') %}
{% set force_batch = config.get('force_batch', False) | as_bool -%}
{% set build_with_subquery = config.get('build_with_subquery', False) | as_bool -%}
{% set unique_tmp_table_suffix = config.get('unique_tmp_table_suffix', False) | as_bool -%}
{% set temp_schema = config.get('temp_schema') %}
{% set target_relation = this.incorporate(type='table') %}
Expand Down Expand Up @@ -46,6 +47,18 @@
{% endif %}
{% endif %}

{% if build_with_subquery %}
{% if model_language == 'python' %}
{% do exceptions.raise_compiler_error('build_with_subquery is not supported with Python models.') %}
{% endif %}
{% if force_batch %}
{% do exceptions.raise_compiler_error('build_with_subquery is incompatible with force_batch. Batching requires data in the tmp table.') %}

@iconara iconara Apr 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: I would say "Batching requires a temp table"

{% endif %}
{% if strategy == 'insert_overwrite' %}
{% do exceptions.raise_compiler_error('build_with_subquery is not supported with insert_overwrite strategy.') %}
{% endif %}
Comment on lines +57 to +59

Copilot AI Apr 6, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build_with_subquery validation block runs unconditionally, which will raise a compiler error even on --full-refresh runs (e.g., when build_with_subquery=True and force_batch=True). The new functional tests expect full-refresh to succeed and only the incremental run to fail, and it’s also reasonable for full-refresh to ignore this incremental-only optimization. Consider gating these checks to only apply on incremental runs (e.g., when existing_relation is not none and not should_full_refresh()), or update the tests/behavior consistently.

Suggested change
{% if force_batch %}
{% do exceptions.raise_compiler_error('build_with_subquery is incompatible with force_batch. Batching requires data in the tmp table.') %}
{% endif %}
{% if strategy == 'insert_overwrite' %}
{% do exceptions.raise_compiler_error('build_with_subquery is not supported with insert_overwrite strategy.') %}
{% endif %}
{% if existing_relation is not none and not should_full_refresh() %}
{% if force_batch %}
{% do exceptions.raise_compiler_error('build_with_subquery is incompatible with force_batch. Batching requires data in the tmp table.') %}
{% endif %}
{% if strategy == 'insert_overwrite' %}
{% do exceptions.raise_compiler_error('build_with_subquery is not supported with insert_overwrite strategy.') %}
{% endif %}
{% endif %}

Copilot uses AI. Check for mistakes.

@dtaniwaki dtaniwaki Apr 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there're pros and cons in ignoring the error in the full-refresh phase. The error will be eventually raised in the incremental phase, so we should catch it earlier.

{% endif %}

{{ run_hooks(pre_hooks, inside_transaction=False) }}

-- `BEGIN` happens here:
Expand Down Expand Up @@ -146,22 +159,36 @@

-- Append Strategy --
{% elif strategy == 'append' %}
{% if old_tmp_relation is not none %}
{% do drop_relation(old_tmp_relation) %}
{% if build_with_subquery %}
{% if old_tmp_relation is not none %}
{% do drop_relation(old_tmp_relation) %}
{% endif %}
{%- set empty_sql = 'SELECT * FROM (\n' ~ compiled_code ~ '\n) _dbt_sbq WITH NO DATA' -%}
{% do run_query(create_table_as(True, tmp_relation, empty_sql)) %}

{% set build_sql = incremental_insert(
on_schema_change, tmp_relation, target_relation, existing_relation, false, source_sql=compiled_code
)
%}
{% do to_drop.append(tmp_relation) %}
{% else %}
{% if old_tmp_relation is not none %}
{% do drop_relation(old_tmp_relation) %}
{% endif %}
{% set query_result = safe_create_table_as(True, tmp_relation, compiled_code, model_language, force_batch) -%}
{%- if model_language == 'python' -%}
{% call statement('create_table', language=model_language) %}
{{ query_result }}
{% endcall %}
{%- endif -%}
{% set build_sql = incremental_insert(
on_schema_change, tmp_relation, target_relation, existing_relation, force_batch
)
%}
{% do to_drop.append(tmp_relation) %}
{% endif %}
{% set query_result = safe_create_table_as(True, tmp_relation, compiled_code, model_language, force_batch) -%}
{%- if model_language == 'python' -%}
{% call statement('create_table', language=model_language) %}
{{ query_result }}
{% endcall %}
{%- endif -%}
{% set build_sql = incremental_insert(
on_schema_change, tmp_relation, target_relation, existing_relation, force_batch
)
%}
{% do to_drop.append(tmp_relation) %}

-- Iceberge Merge Stategy --
-- Iceberg Merge Strategy --
{% elif strategy == 'merge' and table_type == 'iceberg' %}
{% set unique_key = config.get('unique_key') %}
{% set incremental_predicates = config.get('incremental_predicates') %}
Expand All @@ -186,29 +213,55 @@
{% do exceptions.raise_compiler_error(inc_predicates_not_list) %}
{% endif %}
{% endif %}
{% if old_tmp_relation is not none %}
{% do drop_relation(old_tmp_relation) %}

{% if build_with_subquery %}
-- Create empty tmp table for schema comparison (no data scan)
{% if old_tmp_relation is not none %}
{% do drop_relation(old_tmp_relation) %}
{% endif %}
{%- set empty_sql = 'SELECT * FROM (\n' ~ compiled_code ~ '\n) _dbt_sbq WITH NO DATA' -%}
{% do run_query(create_table_as(True, tmp_relation, empty_sql)) %}

{% set build_sql = iceberg_merge(
on_schema_change=on_schema_change,
tmp_relation=tmp_relation,
target_relation=target_relation,
unique_key=unique_key,
incremental_predicates=incremental_predicates,
existing_relation=existing_relation,
delete_condition=delete_condition,
update_condition=update_condition,
insert_condition=insert_condition,
force_batch=false,
source_sql=compiled_code,
)
%}
{% do to_drop.append(tmp_relation) %}
{% else %}
{% if old_tmp_relation is not none %}
{% do drop_relation(old_tmp_relation) %}
{% endif %}
{% set query_result = safe_create_table_as(True, tmp_relation, compiled_code, model_language, force_batch) -%}
{%- if model_language == 'python' -%}
{% call statement('create_table', language=model_language) %}
{{ query_result }}
{% endcall %}
{%- endif -%}
{% set build_sql = iceberg_merge(
on_schema_change=on_schema_change,
tmp_relation=tmp_relation,
target_relation=target_relation,
unique_key=unique_key,
incremental_predicates=incremental_predicates,
existing_relation=existing_relation,
delete_condition=delete_condition,
update_condition=update_condition,
insert_condition=insert_condition,
force_batch=force_batch,
)
%}
{% do to_drop.append(tmp_relation) %}
{% endif %}
{% set query_result = safe_create_table_as(True, tmp_relation, compiled_code, model_language, force_batch) -%}
{%- if model_language == 'python' -%}
{% call statement('create_table', language=model_language) %}
{{ query_result }}
{% endcall %}
{%- endif -%}
{% set build_sql = iceberg_merge(
on_schema_change=on_schema_change,
tmp_relation=tmp_relation,
target_relation=target_relation,
unique_key=unique_key,
incremental_predicates=incremental_predicates,
existing_relation=existing_relation,
delete_condition=delete_condition,
update_condition=update_condition,
insert_condition=insert_condition,
force_batch=force_batch,
)
%}
{% do to_drop.append(tmp_relation) %}
{% endif %}

{% call statement("main", language=model_language) %}
Expand Down
21 changes: 17 additions & 4 deletions dbt-athena/src/dbt/include/athena/macros/materializations/models/incremental/merge.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@
update_condition,
insert_condition,
force_batch,
source_sql=none,
statement_name="main"
)
%}
Expand Down Expand Up @@ -143,9 +144,17 @@
{%- if force_batch -%}
{% do batch_iceberg_merge(tmp_relation, target_relation, merge_part, dest_cols_csv) %}
{%- else -%}
{%- set src_part -%}
merge into {{ target_relation }} as target using {{ tmp_relation }} as src
{%- endset -%}
{%- if source_sql is not none -%}
{%- set src_part -%}
merge into {{ target_relation }} as target using (
{{ source_sql }}
) as src
{%- endset -%}
{%- else -%}
{%- set src_part -%}
merge into {{ target_relation }} as target using {{ tmp_relation }} as src
{%- endset -%}
{%- endif -%}
{%- set merge_full -%}
{{ src_part }}
{{ merge_part }}
Expand All @@ -154,7 +163,11 @@
{%- set query_result = adapter.run_query_with_partitions_limit_catching(merge_full) -%}
{%- do log('QUERY RESULT: ' ~ query_result) -%}
{%- if query_result == 'TOO_MANY_OPEN_PARTITIONS' -%}
{% do batch_iceberg_merge(tmp_relation, target_relation, merge_part, dest_cols_csv) %}
{%- if source_sql is not none -%}
{% do exceptions.raise_compiler_error('Runtime error: TOO_MANY_OPEN_PARTITIONS encountered with build_with_subquery=True. Disable build_with_subquery to enable automatic batching.') %}
{%- else -%}
{% do batch_iceberg_merge(tmp_relation, target_relation, merge_part, dest_cols_csv) %}
{%- endif -%}
{%- endif -%}
{%- endif -%}

Expand Down
Loading
Loading