[Jobs] Add on_before_recovery runtime hook

[Michaelvll]

Summary

Adds extension points so a registered managed-jobs runtime can preserve a
recovering job's logs before its cluster is torn down, plus a dashboard slot to
surface them. When a managed job recovers, its previous run's logs are otherwise
lost with the replaced cluster; these hooks let a runtime snapshot them first.

Changes:

1. on_before_recovery hook. New
   ManagedJobRuntime.on_before_recovery(handle, backend, job_id, task_id, exit_codes, job_id_on_pool_cluster)
   protocol method + a module-level dispatch in sky/jobs/runtime.py. The
   controller (sky/jobs/controller.py) invokes it in _monitor_one_task's
   recovery branch — before cleanup/relaunch, while the cluster is (best-effort)
   still reachable — via asyncio.to_thread (so the blocking log I/O never
   stalls the event loop) and gated on is_registered().

   • exit_codes: the failed run's per-node exit codes (when recovery was
     triggered by a job failure), so the hook can record why each node exited.
   • job_id_on_pool_cluster: disambiguates which job's logs to pull on a
     shared pool cluster.

2. sync_down_logs(head_only=...). CloudVmRayBackend.sync_down_logs gains
   a head_only option to rsync only the head node's run.log (which already
   aggregates every node's output via the runtime's log streaming). During a
   multi-node recovery a node is often already gone, and the all-node fan-out
   exec's into every node — so one unreachable node aborts the whole download.
   Head-only sidesteps that and still yields the full aggregated log.

3. jobs.detail.logfilters dashboard slot. A PluginSlot beside the log
   node picker so a plugin can contribute extra log filters, keyed on the same jobId/taskId
   context as the log pane.

Defensive: the dispatch skips runtimes predating the hook, and the controller
swallows hook exceptions so a failure never blocks recovery. No behavior change
on the default OSS path (no runtime registered) or for runtimes that don't
implement the hook.

Test plan

• Existing managed-jobs unit tests pass.
• Verified the dispatch is a no-op with no runtime registered and under version
  skew (runtime lacking the method), and fires once per recovery otherwise.
• Verified head_only rsyncs only the head node.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces an on_before_recovery hook to the managed job runtime, allowing the system to capture logs from a failing cluster before it is torn down or relaunched. Feedback on the changes points out that calling this synchronous, I/O-heavy hook directly inside the asynchronous _monitor_one_task method will block the event loop, and recommends wrapping the call in asyncio.to_thread to run it in a separate thread.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[kevinmingtarja]

/smoke-test --kubernetes --jobs-consolidation --no-resource-heavy