Skip to content

[fray] Remove 24h default timeout on job wait#4380

Merged
ravwojdyla merged 3 commits intomainfrom
rav/fray-remove-default-timeout
Apr 6, 2026
Merged

[fray] Remove 24h default timeout on job wait#4380
ravwojdyla merged 3 commits intomainfrom
rav/fray-remove-default-timeout

Conversation

@ravwojdyla-agent
Copy link
Copy Markdown
Contributor

Summary

  • Replace the hard-coded 24h (86400.0) default timeout in IrisJobHandle.wait() with float("inf"), so callers that pass timeout=None wait indefinitely instead of silently timing out after a day.

Test plan

  • Verified the change applies cleanly
  • Run fray jobs that exceed 24h and confirm they no longer time out unexpectedly

🤖 Generated with Claude Code

The fray Iris backend defaulted to 86400s (24h) when no timeout was
specified, causing long-running zephyr pipelines to be killed and
retried unnecessarily. Use infinity instead so jobs wait until
completion or explicit cancellation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ravwojdyla
Copy link
Copy Markdown
Contributor

no claude, bad @claude. I wanted to have inf only for the zephyr coordinator pipeline result. pls do better.

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented Apr 3, 2026

Claude finished @ravwojdyla's task in 17m 21s —— View job


Scoped infinite timeout to zephyr coordinator only

  • Understand the request: only apply inf timeout to zephyr coordinator, not globally
  • Find zephyr coordinator pipeline result wait call
  • Revert the global default change in iris_backend.py
  • Apply timeout=float("inf") only at the zephyr coordinator call site
  • Guard local and ray backends against float("inf") (they can't handle it)
  • Run pre-commit and tests
  • Push changes

Changes made

  1. lib/fray/src/fray/v2/iris_backend.py — Reverted default timeout back to 86400.0 (24h)
  2. lib/zephyr/src/zephyr/execution.py:1562 — Pass timeout=float("inf") explicitly at the coordinator wait call
  3. lib/fray/src/fray/v2/local_backend.py — Guard against float("inf") (Python's concurrent.futures.Future.result() can't handle it, converts to None)
  4. lib/fray/src/fray/v2/ray_backend/backend.py — Same guard for ray.get() which also can't handle infinite float timeout

Testing

  • ./infra/pre-commit.py --all-files --fix — ✅ all checks pass
  • uv run pytest lib/zephyr/tests/test_execution.py — ✅ 41 passed
  • uv run pytest lib/fray/tests/ — ✅ 79 passed, 1 unrelated failure (test_ray_tpu module import)

Revert the global IrisJobHandle.wait() default from float("inf") back to
86400.0 and instead pass timeout=float("inf") only at the zephyr
coordinator call site. Also guard local and ray backends against
float("inf") since concurrent.futures and ray.get cannot handle it.

Co-authored-by: Rafal Wojdyla <ravwojdyla@users.noreply.github.com>
@rjpower
Copy link
Copy Markdown
Collaborator

rjpower commented Apr 3, 2026

Let's use None as the indicator? As we've seen float("inf") isn't handled well by either Ray or local backends? (or some other sentinel like "5 years from now" is also fine)

@ravwojdyla
Copy link
Copy Markdown
Contributor

@claude address above

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented Apr 3, 2026

Claude finished @ravwojdyla's task in 4m 46s —— View job


Addressed rjpower's feedback: use None instead of float("inf")

  • Understand request: use None as sentinel for "wait indefinitely" instead of float("inf")
  • Read current state of changed files
  • Update iris_backend.pyNone → ~5 years (Iris client needs numeric timeout)
  • Update zephyr/execution.pytimeout=float("inf")timeout=None
  • Simplify local_backend.py — remove float("inf") guard, pass timeout directly
  • Simplify ray_backend/backend.py — remove float("inf") guard, pass timeout directly
  • Run pre-commit and tests
  • Push changes

Changes

Switched from float("inf") to None as the "wait indefinitely" sentinel across the fray API:

  1. lib/zephyr/src/zephyr/execution.pytimeout=float("inf")timeout=None
  2. lib/fray/src/fray/v2/iris_backend.pyNone now maps to 86400 * 365 * 5 (~5 years) since the Iris client requires a numeric timeout
  3. lib/fray/src/fray/v2/local_backend.py — Removed float("inf") guard; Future.result(timeout=None) already waits indefinitely
  4. lib/fray/src/fray/v2/ray_backend/backend.py — Removed float("inf") guard; ray.get(timeout=None) already waits indefinitely

Testing

  • ./infra/pre-commit.py --all-files --fix — ✅ all checks pass
  • uv run pytest lib/fray/tests/ lib/zephyr/tests/test_execution.py — ✅ 196 passed, 11 deselected
    · Branch

Per review feedback, None is a cleaner sentinel than float("inf") since
both ray.get() and Future.result() already treat None as "wait forever".
The Iris client still needs a numeric timeout, so None maps to ~5 years.

Co-authored-by: Rafal Wojdyla <ravwojdyla@users.noreply.github.com>
@ravwojdyla ravwojdyla merged commit 9529858 into main Apr 6, 2026
39 checks passed
@ravwojdyla ravwojdyla deleted the rav/fray-remove-default-timeout branch April 6, 2026 17:00
Helw150 pushed a commit that referenced this pull request Apr 8, 2026
## Summary
- Replace the hard-coded 24h (`86400.0`) default timeout in
`IrisJobHandle.wait()` with `float("inf")`, so callers that pass
`timeout=None` wait indefinitely instead of silently timing out after a
day.

## Test plan
- [x] Verified the change applies cleanly
- [ ] Run fray jobs that exceed 24h and confirm they no longer time out
unexpectedly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Rafal Wojdyla <ravwojdyla@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Rafal Wojdyla <ravwojdyla@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants