Extract log tail from failed Task Instances #31#68197
Open
sharan-s2k wants to merge 19 commits into
Open
Conversation
The polling loop that waits for the supervisor process to exit used:
while job.is_running:
await sleep(0.1)
This trips ruff ASYNC110 (await inside a while-condition loop).
Refactor to the ASYNC110-clean pattern used elsewhere:
while True:
if not job.is_running:
break
await sleep(0.1)
Same behaviour, passes static-checks CI.
Fix ASYNC110 violation in edge3 worker
docs(team_project): add team workspace with project plan and architec…
feat(team_project): add heuristic failure classifier with five-catego…
feat(team_project): add Airflow log normalization layer
…w failure patterns
feat(team_project): seed remediation knowledge base with eight Airflo…
Token expiration worker
Add automatic log tail capture for failed task instances Fetch the last N lines (default 200, configurable via [triage] in airflow.cfg) from task logs using Airflow's TaskLogReader so triage does not require opening the log viewer for every incident. A listener captures tails on failed and skipped tasks; get_task_log_tail supports on-demand UI access with an in-memory cache. Log read failures degrade gracefully without crashing the plugin.
sharan-s2k
requested review from
amoghrajesh,
ashb,
bbovenzi,
choo121600,
guan404ming,
kaxil,
pierrejeambrun,
ryanahamilton,
shubhamraj-git and
vatsrahul1001
as code owners
Add unit and integration-style tests for dag_triage log tail reading, in-memory cache, listener hooks, and triage configuration. Includes MockTaskLogReader helpers and Airflow stubs so tests run standalone without a full Airflow runtime. Wire pytest dev dependencies and pythonpath in team_project/pyproject.toml.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add automatic log tail capture for failed task instances
Fetch the last N lines (default 200, configurable via [triage] in
airflow.cfg) from task logs using Airflow's TaskLogReader so triage
does not require opening the log viewer for every incident. A listener
captures tails on failed and skipped tasks; get_task_log_tail supports
on-demand UI access with an in-memory cache. Log read failures degrade
gracefully without crashing the plugin.