Merged
Conversation
Large TPU pods were timing out while queued resources became ready and while the last workers finished bootstrap. Scale the wait budgets by pod size and log the missing workers and probe errors that remain when bootstrap stalls.
Contributor
|
Claude finished @dlwh's task in 4m 57s —— View job PR Review complete
Summary: No issues found. Checked for bugs and CLAUDE.md/AGENTS.md compliance. Two candidate issues were raised by the compliance agents and validated:
Bug agents found no syntax, logic, timeout-semantic, or error-handling issues; tests exercise real boundary values (1 / 64 / 256). |
Contributor
Code review🤖 No issues found. Checked for bugs and CLAUDE.md/AGENTS.md compliance. |
rjpower
approved these changes
Apr 15, 2026
| return DEFAULT_TPU_BOOTSTRAP_TIMEOUT | ||
|
|
||
|
|
||
| def _format_probe_error(error: BaseException) -> str: |
Collaborator
There was a problem hiding this comment.
this is just str(error) ?
| # ============================================================================= | ||
|
|
||
|
|
||
| def test_recommended_tpu_timeouts_scale_with_pod_size(): |
Collaborator
|
@claude recommended.*timeout -> default.*timeout, fix up the slop as per PR comments |
Contributor
|
Claude finished @rjpower's task in 3m 43s —— View job Code changesAlready on the branch — PR author pushed
I had staged an equivalent local commit and reset it in favor of the author's version (which picks a slightly more defensive fallback for empty exception messages). No push from me was needed. Validation
|
rjpower
added a commit
that referenced
this pull request
Apr 15, 2026
Per-task slow_log timers on submit_task (500ms) and synchronous kill (2000ms) inside handle_heartbeat identify which task stalls a heartbeat. Worker service heartbeat entrypoint gets an outer slow_log (1000ms) and a DEBUG payload-size line to correlate with controller-side sync timing. Slice ready/failed transitions log registered worker counts and ids to expose partial bootstrap on large slices. Complements #4792 and #4793.
rjpower
added a commit
that referenced
this pull request
Apr 15, 2026
Per-task slow_log timers on submit_task (500ms) and synchronous kill (2000ms) inside handle_heartbeat identify which task stalls a heartbeat. The worker service heartbeat entrypoint gets an outer slow_log (1000ms) and a DEBUG payload-size line to correlate with controller-side sync timing. Slice ready/failed transitions log registered worker counts and ids to expose partial bootstrap on large slices. Complements #4792 and #4793.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scale TPU queued-resource and bootstrap waits by pod size and log the missing workers and probe errors when bootstrap stalls. This covers the 255/256 healthy provisioning path from base issue #4697.
Part of #4746