zephyr: exempt preemption requeues from MAX_SHARD_FAILURES

[ravwojdyla-agent]

• preemption-driven requeues (register_worker, heartbeat timeout) no longer consume the 3-attempt MAX_SHARD_FAILURES budget — previously three clean preemptions aborted the pipeline
• split _task_attempts into two counters: generation (bumps on every requeue, drives stale-result rejection in report_result) and _task_error_attempts (TASK-only, still capped at MAX_SHARD_FAILURES=3)
• add ShardFailureKind.{TASK,INFRA}; report_error → TASK, _maybe_requeue_worker_task → INFRA
• INFRA requeues retry unbounded ¹
• tests
  • replace test_heartbeat_death_aborts_at_max_shard_failures (codified the bug) with test_heartbeat_timeouts_do_not_count_toward_shard_failures — 15 timeouts + final success
  • add test_worker_reregistration_does_not_count_toward_shard_failures
  • add test_report_error_still_aborts_at_max_shard_failures_after_preemptions — pins that real task errors still abort at 3 even after prior preemptions

Footnotes

1. _check_worker_group already aborts if the worker job permanently exhausts Iris' own retry budget, so unbounded infra retries cannot loop forever ↩

[rjpower]

@claude fix all of the local imports in the test file, these violate our standards.

[claude]

Claude finished @rjpower's task in 5m 13s —— View job

---

Fixing local imports in test file

• Read lib/zephyr/tests/test_execution.py to identify local imports
• Move local imports to top of file
• Run ./infra/pre-commit.py --all-files --fix
• Run uv run pytest on affected tests
• Commit and push

Summary

Removed local imports from ~20 test functions and consolidated them at the module top. The added top-level imports are:

• stdlib: logging, os, from unittest.mock import MagicMock
• third-party: from fray.v2.local_backend import LocalClient
• project: from zephyr import counters, from zephyr.plan import compute_plan, and the combined zephyr.execution import now covers MAX_SHARD_FAILURES, ListShard, PickleDiskChunk, ShardTask, TaskResult, WorkerState, ZephyrCoordinator, ZephyrWorkerError (previously imported ad-hoc inside tests).

Net: +19 / -87 lines in lib/zephyr/tests/test_execution.py.

Verification

• ./infra/pre-commit.py --all-files --fix — OK (ruff, black, pyrefly all pass)
• uv run --with pytest pytest lib/zephyr/tests/test_execution.py -x — 42 passed in 77.31s (one unrelated Threads leaked from test warning on test_shared_data[iris], not introduced by this change)

Commit: a560ff75b on branch worktree-rav-peemption-shard-retires.