feat(athena): add model-level timeout support - #4
Open
dtaniwaki wants to merge 3 commits into
Open
Conversation
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Add two timeout mechanisms to prevent runaway queries and models: - query_timeout_seconds: per-query timeout configured in profiles.yml. Cancels the Athena query and raises AthenaQueryTimeoutError when a single query exceeds the limit. - model_timeout_seconds: per-model timeout configured in dbt model config. Checks deadline before each batch iteration and in _run_query(). Raises AthenaModelTimeoutError when total model execution exceeds the limit. Uses threading.local() for thread safety. Batch loop timeout checks are added to: - create_table_as_with_partitions - batch_incremental_insert - batch_iceberg_merge
…ut_seconds Remove per-query timeout (query_timeout_seconds) as only model-level timeout is needed. The model_timeout_seconds config covers the batch loop runaway scenario which is the actual problem.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
model_timeout_seconds(model config) to limit total model execution time including batch loopsthreading.local()for deadline tracking_run_query()AthenaModelTimeoutErrorwhen the deadline is exceededMotivation
The dbt-athena adapter has no model-level timeout mechanism. When Iceberg
force_batchgenerates a large number of batches, each batch query is fast but the total model execution time becomes unbounded. This makes it impossible to set a reasonable CodeBuild timeout without risking runaway models.Standardizing timeout handling is being discussed upstream at dbt-labs/dbt-core#8522, but progress has stalled with no concrete implementation plan. This PR provides a practical solution for the Athena adapter in the meantime.
Usage
In
dbt_project.yml:Or in a model file:
{{ config(model_timeout_seconds=3600) }} SELECT ...Changes
exceptions.py: AddAthenaModelTimeoutErrorimpl.py: Addmodel_timeout_secondstoAthenaConfig, addset/check/clear_model_timeout()adapter methods withthreading.local(), add timeout check in_run_query()andrun_operation_with_potential_multiple_runs()table.sql,incremental.sql: Set/clear model timeout at materialization boundariescreate_table_as.sql,helpers.sql,merge.sql: Check model timeout before each batch iterationTest plan
Recreated from #1, rebased on main for cleaner history.