Skip to content

Warm up Tart VM images and expose pull status in UI #81

Description

@STRRL

This was generated by AI during triage.

Problem

macOS arm64 runners backed by Tart can spend a long time pulling a remote VM image at job time. In the observed failure mode, fruit-runner-macos-arm64 received work, entered runner preparation, and then spent the preparation window running:

tart pull ghcr.io/cirruslabs/macos-tahoe-xcode:latest

Because the image is large, the job can be canceled before any VM is cloned, started, or registered as a usable runner. The dashboard currently does not make this distinction obvious enough: a runner can be counted as allocated/preparing while the real activity is still image pull.

Goals

  • Add a warm-up path for Tart-backed runner sets so configured remote macOS images can be pulled before the first job arrives.
  • Make image pull/preparation status visible in the dashboard for macOS runner sets.
  • Preserve capacity control semantics: a runner in preparation should reserve capacity so the controller does not overspawn, but UI/logs should clearly show that it is not yet a running/idle runner.

Expected Behavior

  • On startup, or via an explicit command/API path, the service can pre-pull Tart images for macOS runner sets whose image is a remote registry reference.
  • The UI shows a clear runner-set state such as warming image, pulling image, or preparing runner when the macOS backend is not yet able to start a VM.
  • Users can distinguish these states:
    • image not pulled yet
    • image pull in progress
    • VM cloning/starting
    • runner process started and waiting for assignment
    • runner busy
  • Logs and API responses expose enough structured state to diagnose slow image pulls without SSHing into the host.

Implementation Notes

  • The current controller has internal states for preparing, idle, and busy. This issue likely needs more granular backend preparation state for Tart, not just a single preparing bucket.
  • Tart remote images appear in tart list --quiet as OCI images, not in tart list --source local --quiet. Any warm-up implementation should account for pulled OCI images.
  • Avoid pulling the same image concurrently for multiple runner sets or repeated scaling messages. A per-image lock or singleflight-style guard would prevent duplicate large downloads.
  • Consider exposing warm-up status through the existing control-plane API so the dashboard can render it without parsing logs.

Acceptance Criteria

  • A configured macOS Tart image can be warmed before the first job needs a runner.
  • If a Tart image pull is running, the dashboard shows that state for the affected runner set.
  • The dashboard does not present a pulling/preparing macOS runner as an available runner.
  • Repeated scale messages while a Tart pull is in progress do not start duplicate pulls for the same image.
  • Tests cover at least the image-present/image-missing decision and the visible runner-set preparation state.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions