Detect TPU bad-node stderr and promote to WORKER_FAILED

[rjpower]

Fixes #4783.

On non-zero container exit, the worker now scans the stderr tail for known JAX / libtpu bad-node signatures (/dev/vfio busy, Couldn't open iommu group, TPU initialization failed, Failed to initialize TPU system, No accelerator found). On a hit, the attempt transitions to TASK_STATE_WORKER_FAILED instead of TASK_STATE_FAILED, so it's billed against the preemption retry budget and rescheduled elsewhere — matching the existing infra-failure path. No controller changes required.

[claude]

Claude finished @rjpower's task in 3m 55s —— View job

---

Code review complete

• Check PR eligibility for review
• Identify relevant CLAUDE.md/AGENTS.md files (root AGENTS.md, lib/iris/AGENTS.md)
• Summarize PR changes
• Run parallel review agents (CLAUDE.md compliance + bug hunt)
• Validate flagged issues
• Post review findings

Summary

No high-signal issues found. Checked for bugs and AGENTS.md compliance.

• detect_tpu_init_failure is a simple substring scan over a bounded tail — empty-list slicing, None/empty-line guards, and pattern iteration all behave correctly.
• State transition in task_attempt.py fires exactly one transition_to under both branches, followed by the single break — no double-transition risk.
• The two new test_worker.py tests exercise both the WORKER_FAILED promotion path and the no-false-positive FAILED path.
• AGENTS.md compliance: top-level imports, no TYPE_CHECKING, top-level constant _TPU_STDERR_TAIL_LINES, no compatibility shims.

Posted summary comment: #4798 (comment).
· Branch

[rjpower]

@claude get rid of the slop comments & tests. A function with 5 lines doesn't need a 10 line comment.

you can keep the worker tests but get rid of the slop test_tpu_health tests and slop tpu health comments

[claude]

Trimmed slop comments & tests

• Slim lib/iris/src/iris/cluster/worker/tpu_health.py — one-line module docstring, one-line function docstring, no per-pattern commentary
• Trim verbose comments around TPU bad-node detection in lib/iris/src/iris/cluster/worker/task_attempt.py
• Delete lib/iris/tests/cluster/worker/test_tpu_health.py
• Shorten docstrings on the two new tests in lib/iris/tests/cluster/worker/test_worker.py
• ./infra/pre-commit.py --all-files --fix → OK (ruff, black, pyrefly, etc.)
• Commit + push (58d4585f2)

Net change: 4 files, +9 / -113. Behavior coverage for TPU bad-node detection is now solely via the two worker-level integration tests (test_tpu_bad_node_stderr_promotes_to_worker_failed, test_non_tpu_stderr_still_maps_to_failed) that exercise the real task_attempt._monitor_loop path end-to-end.

Test status: uv run pytest lib/iris/tests/cluster/worker/test_worker.py wouldn't return in this sandbox — two concurrent invocations ended up deadlocked on import/fixture setup and I lack permission to kill them here. Pre-commit (ruff + pyrefly) passes cleanly. The TPU-specific test logic was not touched by this commit — only docstrings around existing assertions.

View job run · branch

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 106b753a2f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[claude]

🤖 ## Code review

No issues found. Checked for bugs and CLAUDE.md compliance.