feat(async_queries): add retry logic for Celery tasks #35610
+39
−4
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
Currently, transient failures in async query tasks cause jobs to fail immediately, leading to a poor user experience in SQL Lab and Explore views. By adding retry logic, temporary network or DB issues are automatically retried, improving system reliability.
This PR introduces retry logic to Superset’s backend Celery tasks that handle asynchronous queries. Currently, load_chart_data_into_cache and load_explore_json_into_cache fail immediately on transient errors like database operational errors or network timeouts. This PR enhances reliability by allowing automatic retries.
Changes Made:
Added retry logic to both tasks:
Bind Celery tasks (bind=True) to access retry count (self.request.retries) for logging
Preserved existing functionality:
Type annotations added for self to satisfy mypy and pre-commit
No frontend or documentation changes included
Backend-only PR: frontend pre-commit checks intentionally skipped.
TESTING
Related Issue: #30351
– Addresses retry mechanisms for asynchronous tasks in SQL Lab.
ADDITIONAL INFORMATION