Skip to content

fix(athena): skip retry of deterministic errors with configurable timeout handling - #1814

Closed
dtaniwaki wants to merge 13 commits into
dbt-labs:mainfrom
dtaniwaki:fix/no-retry-on-query-timeout
Closed

fix(athena): skip retry of deterministic errors with configurable timeout handling#1814
dtaniwaki wants to merge 13 commits into
dbt-labs:mainfrom
dtaniwaki:fix/no-retry-on-query-timeout

Conversation

@dtaniwaki

@dtaniwaki dtaniwaki commented Mar 30, 2026

Copy link
Copy Markdown
Contributor

resolves #1813
resolves #1820
docs N/A

Problem

The outer retry loop in AthenaCursor.execute retries all exceptions except TOO_MANY_OPEN_PARTITIONS. This means non-transient errors like query timeout (#1813) and query exhausted resources (#1820) are retried up to num_retries times, wasting compute cost and time.

Solution

  • Query exhausted resources: Always skip retry (deterministic — the query exceeds Athena's memory capacity).
  • Query timeout: Skip retry only when skip_retry_on_query_timeout: true is set in the profile config. Since timeout can be caused by transient load, this is opt-in rather than hardcoded.

Usage

# profiles.yml
my_profile:
  target: dev
  outputs:
    dev:
      type: athena
      skip_retry_on_query_timeout: true  # default: false
      ...

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
@cla-bot cla-bot Bot added the cla:yes The PR author has signed the CLA label Mar 30, 2026
@dtaniwaki
dtaniwaki marked this pull request as ready for review March 30, 2026 01:07
@dtaniwaki
dtaniwaki requested a review from a team as a code owner March 30, 2026 01:07
Copilot AI review requested due to automatic review settings March 30, 2026 01:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates dbt-athena’s query execution retry logic to avoid wasting time/cost on deterministic Athena failures (query timeouts), aligning behavior with issue #1813.

Changes:

  • Exclude Athena “Query timeout” failures from the outer AthenaCursor.execute retry loop.
  • Add an unreleased changelog entry describing the fix.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
dbt-athena/src/dbt/adapters/athena/connections.py Adjusts tenacity retry predicate to skip retries when the error message includes “Query timeout”.
dbt-athena/.changes/unreleased/Fixes-20260330-095539.yaml Records the fix in the changelog system.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dbt-athena/src/dbt/adapters/athena/connections.py
Comment thread dbt-athena/src/dbt/adapters/athena/connections.py Outdated
Comment thread dbt-athena/src/dbt/adapters/athena/connections.py
Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
@dtaniwaki
dtaniwaki force-pushed the fix/no-retry-on-query-timeout branch from b71a6af to 0e120ff Compare March 30, 2026 01:17
Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
@dtaniwaki dtaniwaki changed the title fix(athena): skip retry on query timeout fix(athena): skip retry on non-transient errors Mar 31, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dbt-athena/src/dbt/adapters/athena/connections.py Outdated
Comment thread dbt-athena/src/dbt/adapters/athena/connections.py
Comment thread dbt-athena/tests/unit/test_cursor_retry.py
Comment thread dbt-athena/.changes/unreleased/Fixes-20260330-095539.yaml Outdated
Comment thread dbt-athena/src/dbt/adapters/athena/connections.py Outdated
…meout handling

Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
@dtaniwaki dtaniwaki changed the title fix(athena): skip retry on non-transient errors fix(athena): skip retry of deterministic errors with configurable timeout handling Apr 1, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dbt-athena/src/dbt/adapters/athena/connections.py
Comment thread dbt-athena/src/dbt/adapters/athena/connections.py
@colin-k-rogers
colin-k-rogers requested a review from Copilot April 1, 2026 15:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@dtaniwaki

dtaniwaki commented Apr 21, 2026

Copy link
Copy Markdown
Contributor Author

@iconara As you mentioned on the other similar PR, do you think this PR is also covered by #1637 ?

@iconara

iconara commented Apr 26, 2026

Copy link
Copy Markdown
Contributor

@dtaniwaki yes, that PR will only retry on a few well-defined errors so the ones here should be covered too.

@dtaniwaki

Copy link
Copy Markdown
Contributor Author

@iconara I'm using your fix in my dbt-athena custom build for a week and found no issues so far.

@dtaniwaki

Copy link
Copy Markdown
Contributor Author

Superseded by #1637 (PyAthena→boto3 migration), now merged into main. The new boto3-based cursor only retries ICEBERG_COMMIT_ERROR (throttling API errors are handled separately), so ICEBERG_FILESYSTEM_ERROR / deterministic errors like query timeouts are no longer retried by design. Closing as no longer needed.

@dtaniwaki dtaniwaki closed this Jul 2, 2026
auto-merge was automatically disabled July 2, 2026 00:27

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla:yes The PR author has signed the CLA needs:docs

Projects

None yet

4 participants