Skip to content

runner-bootstrap.md

Latest commit

 

History

History
170 lines (132 loc) · 7.62 KB

runner-bootstrap.md

File metadata and controls

170 lines (132 loc) · 7.62 KB
Raw

Runner Bootstrap

Read this when:

  • you are changing cloud-init or the per-OS bootstrap scripts;
  • you are debugging a managed box that never becomes SSH-ready;
  • you are changing the minimal runner contract or the crabbox-ready checks.

Bootstrap is what turns a freshly provisioned cloud instance into a box Crabbox can reach over SSH and run commands on. It is generated by the CLI for direct-provider leases and by the coordinator for brokered leases; both surfaces emit the same scripts, so a box behaves the same whether it was leased direct or through the broker. Code lives in internal/cli/bootstrap.go and worker/src/bootstrap.ts.

Bootstrapped boxes carry no coordinator credentials. The box never calls the broker; the CLI connects to it directly over SSH.

The minimal Linux contract

Brokered and direct cloud Linux runners are Ubuntu machines configured by cloud-init. The bootstrap deliberately does not run apt upgrade (package_upgrade: false), so a box comes up fast and predictable rather than chasing the latest package set.

Bootstrap creates:

  • the crabbox SSH user, in the sudo group with passwordless sudo;
  • key-only SSH (PasswordAuthentication no), authorizing the per-lease public key;
  • SSH listening on the primary port (default 2222) plus the configured fallback ports (default 22);
  • the work root (/work/crabbox by default), owned by crabbox;
  • shared package cache directories /var/cache/crabbox/pnpm and /var/cache/crabbox/npm.

Bootstrap installs only a small base set with --no-install-recommends:

  • openssh-server
  • ca-certificates
  • curl
  • git
  • rsync
  • jq

apt-get runs are wrapped in a retry loop (8 attempts, increasing backoff) so a transient mirror failure does not fail the whole boot.

Readiness: crabbox-ready

Bootstrap writes /usr/local/bin/crabbox-ready and runs it at the end of boot. A box does not count as ready until this script exits 0 over SSH. The base Linux script checks:

  • git, rsync, curl, and jq are present and runnable;
  • the marker file /var/lib/crabbox/bootstrapped exists;
  • the work root is writable.

Optional lease capabilities (below) and Tailscale extend crabbox-ready with additional checks, so readiness always reflects the full requested capability set.

What bootstrap does not install

Bootstrap intentionally omits project language runtimes and services — Go, Node, pnpm, Docker, databases, and the like. Those are repository-owned setup and belong in Actions hydration, devcontainers, Nix, mise/asdf, or repository setup scripts. Keeping bootstrap minimal keeps boot fast and keeps the contract stable enough that snapshots and prebaked images can replace cloud-init. See Prebaked runner images.

Optional lease capabilities

Interactive tooling is opt-in per lease, never part of the minimal bootstrap. Each requested capability appends both its install steps and its crabbox-ready checks, and is gated by the provider's declared feature set.

  • --desktop — installs a headless desktop. The default XFCE environment adds Xvfb on display :99, an XFCE session, and a loopback x11vnc on 127.0.0.1:5900. With --desktop-env wayland (labwc) or --desktop-env gnome it installs a Wayland session driven by wayvnc instead. Readiness checks the relevant systemd units and that something is listening on 127.0.0.1:5900.
  • --browser — installs Google Chrome stable, falling back to Chromium, plus the native-addon build helpers (build-essential, python3) that browser-channel QA often needs during dependency fallback installs. It writes a crabbox-browser wrapper and /var/lib/crabbox/browser.env; readiness verifies the wrapper runs --version.
  • --code — installs code-server (managed Linux only) for the authenticated portal editor; readiness verifies code-server --version.

Crabbox owns these machine capabilities; scenario systems still own browser automation and proof artifacts. For slow QA lanes, bake these capabilities into a provider image while keeping secrets, browser profiles, repository checkouts, and built artifacts out of the image. See Interactive desktop and VNC and Prebaked runner images.

Tailscale

--tailscale on a managed Linux lease is also optional. Bootstrap installs the Tailscale package, brings the box up on the configured tailnet, writes non-secret metadata under /var/lib/crabbox (such as tailscale-ipv4, tailscale-hostname, and exit-node details), and extends crabbox-ready with a bounded check that a 100.x address has appeared.

The auth key is piped to tailscale up through stdin and is not persisted or placed in process arguments. Brokered leases receive a one-off key minted by the coordinator; direct-provider leases read it from CRABBOX_TAILSCALE_AUTH_KEY. Set TS_CONTROL_URL on the operator shell to register the box against a self-hosted control plane (Headscale and similar) instead of the default Tailscale control plane.

When a lease also carries a --pond label, the Tailscale bootstrap installs a 30-second timer that rewrites /etc/hosts.cbx and a managed block in /etc/hosts so pond peers resolve as <slug>.cbx. Peers are discovered from the box-local tailscale status filtered by the pond ACL tag, so the broker never sees a Tailscale credential. See Tailscale and the pond notes in Providers.

Other targets

macOS and Windows (managed)

aws (Windows and Mac instances) and azure (Windows) bootstrap non-Linux boxes with per-OS scripts rather than cloud-init:

  • macOS — a shell script creates SSH access for the lease user, enables Remote Login on the configured ports, enables Screen Sharing, and writes a crabbox-ready that checks rsync/curl, a writable work root, an open SSH port, and the VNC port 5900.
  • Windows — a PowerShell script installs OpenSSH, configures key-only access for administrators, opens firewall rules on the SSH ports, and installs Git for Windows so git and tar are on the machine PATH. --windows-mode wsl2 additionally enables the WSL feature set, imports an Ubuntu rootfs, and runs the minimal Linux base inside WSL; --desktop adds TightVNC. The work root defaults to C:\crabbox (native) or the Linux default inside WSL.

Static / BYO SSH hosts

provider=ssh (aliases static, static-ssh) targets are not bootstrapped by Crabbox. They are assumed to be operator-managed and must already provide:

  • macOS and Windows WSL2 targets: SSH, bash, git, rsync, and tar;
  • native Windows targets: OpenSSH, PowerShell, git, and tar;
  • static.workRoot pointing at a writable directory for that target mode.

For native Windows, install Git before the Crabbox check or restart OpenSSH Server afterward, so new non-interactive SSH sessions inherit git and tar on PATH.

SSH port fallback

The CLI prefers the configured SSH port and falls back through ssh.fallbackPorts during early bootstrap or when operator-network egress restricts ports. The default is primary 2222 with a 22 fallback. Disable the fallback with ssh.fallbackPorts: [] in config or CRABBOX_SSH_FALLBACK_PORTS=none in the environment.

Once the bootstrap contract is stable, snapshots or provider images can replace slow cloud-init while preserving the same readiness contract.

Related docs