fix(athena): skip retry of deterministic errors with configurable timeout handling - #1814
fix(athena): skip retry of deterministic errors with configurable timeout handling#1814dtaniwaki wants to merge 13 commits into
Conversation
Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
There was a problem hiding this comment.
Pull request overview
Updates dbt-athena’s query execution retry logic to avoid wasting time/cost on deterministic Athena failures (query timeouts), aligning behavior with issue #1813.
Changes:
- Exclude Athena “Query timeout” failures from the outer
AthenaCursor.executeretry loop. - Add an unreleased changelog entry describing the fix.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
dbt-athena/src/dbt/adapters/athena/connections.py |
Adjusts tenacity retry predicate to skip retries when the error message includes “Query timeout”. |
dbt-athena/.changes/unreleased/Fixes-20260330-095539.yaml |
Records the fix in the changelog system. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
b71a6af to
0e120ff
Compare
Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…meout handling Signed-off-by: Daisuke Taniwaki <daisuketaniwaki@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@iconara As you mentioned on the other similar PR, do you think this PR is also covered by #1637 ? |
|
@dtaniwaki yes, that PR will only retry on a few well-defined errors so the ones here should be covered too. |
|
@iconara I'm using your fix in my dbt-athena custom build for a week and found no issues so far. |
|
Superseded by #1637 (PyAthena→boto3 migration), now merged into |
Pull request was closed
resolves #1813
resolves #1820
docs N/A
Problem
The outer retry loop in
AthenaCursor.executeretries all exceptions exceptTOO_MANY_OPEN_PARTITIONS. This means non-transient errors like query timeout (#1813) and query exhausted resources (#1820) are retried up tonum_retriestimes, wasting compute cost and time.Solution
skip_retry_on_query_timeout: trueis set in the profile config. Since timeout can be caused by transient load, this is opt-in rather than hardcoded.Usage
Checklist