Skip to content

feat(athena): add disable_batch_fallback config option - #3

Open
dtaniwaki wants to merge 3 commits into
mainfrom
worktree-soft-chasing-ullman
Open

feat(athena): add disable_batch_fallback config option#3
dtaniwaki wants to merge 3 commits into
mainfrom
worktree-soft-chasing-ullman

Conversation

@dtaniwaki

@dtaniwaki dtaniwaki commented Apr 1, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add disable_batch_fallback config option (default False) to dbt-athena
  • When set to True, TOO_MANY_OPEN_PARTITIONS errors are raised instead of silently falling back to batch insert mode
  • Applies to both table and incremental materializations (CTAS, insert, and iceberg merge)

Motivation

Currently, dbt-athena automatically falls back to batch mode when Athena returns TOO_MANY_OPEN_PARTITIONS on partitioned tables. This is convenient but can mask issues silently. The new disable_batch_fallback option gives users explicit control to fail fast when this error occurs.

Usage

-- table materialization with partitions
{{ config(
    materialized='table',
    partitioned_by=['date_column'],
    disable_batch_fallback=true
) }}

-- incremental materialization with partitions
{{ config(
    materialized='incremental',
    partitioned_by=['date_column'],
    disable_batch_fallback=true
) }}

Note: TOO_MANY_OPEN_PARTITIONS only occurs on partitioned tables. This config has no effect on non-partitioned tables.

Test plan

  • Unit tests for run_query_with_partitions_limit_catching with disable_batch_fallback=True and False
  • Unit test for non-partition errors always raising regardless of config
  • Pre-commit lint (black, flake8, mypy) passes

…ITIONS behavior

When batch_fallback=False, TOO_MANY_OPEN_PARTITIONS errors are raised
instead of silently falling back to batch mode. Default is True
(preserving existing behavior).
Align with other bool configs by using False as the default value.
disable_batch_fallback=True raises TOO_MANY_OPEN_PARTITIONS instead of
silently falling back to batch mode.
dtaniwaki added a commit that referenced this pull request Apr 9, 2026
PR dbt-labs#1830/dbt-labs#1832 のテストが fork #3 で追加された disable_batch_fallback
引数を MockAdapter.run_query_with_partitions_limit_catching に含んでいなかった。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant