feat(infra): RUNNERS-count multi-runner provisioning; drop dead RUNNER_PRIVATE_IP

Add a `RUNNERS=N` knob to provision N runners on `sst deploy` with no API
changes. The default runner is still auto-seeded by the API at boot; extras
(runner-2..N) each get their own EC2 + minted token and are registered with the
control plane after deploy via the admin API. Pairing is token-based and v2
runners self-report their address via healthcheck, so the API needs no
multi-runner seed.

- sst.config.ts: add the `command` provider; create extra-runner EC2s
  (protect:true, same ignoreChanges as the default); RegisterExtraRunners
  (command.local.Command) runs scripts/register-runners.mjs after the EC2s.
- scripts/register-runners.mjs: poll GET /api/health, then POST
  /api/admin/runners per extra runner (Bearer admin key); 201 and 409 both
  treated as success, so redeploys are idempotent.
- Remove dead RUNNER_PRIVATE_IP / runnerEndpoint wiring + the
  DEFAULT_RUNNER_*URL env: they only fed the v0 seed branch, which this v2
  deploy never hits. Keep DEFAULT_RUNNER_NAME/_API_KEY (the v2 seed needs them).
- .env.example / README: document RUNNERS + "Adding a runner"; remove the stale
  RUNNER_PRIVATE_IP "after first deploy" step.

Known limitations (follow-ups): scale-down is not symmetric — protect:true
blocks a plain `RUNNERS` decrease (needs the decommission ceremony) and the
provisioner does not deregister the control-plane row; extra EC2 tags come out
as `boxlite-runner-runner-N`; `npx sst install` is required to fetch the
`command` provider before deploy.

Author: DorianZheng

apps/infra/.env.example

@@ -64,13 +64,11 @@ OIDC_CLIENT_ID=your-spa-client-id
 OIDC_AUDIENCE=https://dev.example.com/api
 
 # ─── 3. Runner ───────────────────────────────────────────────────────────────
-# [required at runtime] After the first deploy, the runner EC2 instance exists.
-# Get its private IP and redeploy so the API can route jobs to it. Unset → the
-# API points at localhost and no sandboxes work (deploy still succeeds).
-#   aws ec2 describe-instances --region ap-southeast-1 \
-#     --filters "Name=tag:Name,Values=boxlite-runner-default" \
-#     --query 'Reservations[].Instances[].PrivateIpAddress' --output text
-RUNNER_PRIVATE_IP=
+# [optional] Total number of runners (the default runner is #1). Set >1 to add
+# more: each extra gets its own EC2 + token and is auto-registered with the
+# control plane during `sst deploy` (admin API). Unset = 1 (default only).
+# v2 runners self-report their address via healthcheck, so no IP wiring is needed.
+#   RUNNERS=3
 
 # ─── 4. SSH Gateway ──────────────────────────────────────────────────────────
 # [required at runtime if you use SSH] Base64-encoded Ed25519 keys. Unset → the
@@ -121,12 +119,9 @@ SSH_HOST_KEY_B64=
 # DASHBOARD_BASE_API_URL=https://api.dev.example.com
 # APP_URL=
 
-# 5e. Runner wiring — derived from RUNNER_PRIVATE_IP; pin to override per-runner
-# endpoints.
+# 5e. Default runner name (rarely changed). v2 runners self-report their
+# address, so there are no domain/url knobs to pin.
 # DEFAULT_RUNNER_NAME=default
-# DEFAULT_RUNNER_DOMAIN=
-# DEFAULT_RUNNER_API_URL=
-# DEFAULT_RUNNER_PROXY_URL=
 
 # 5f. Auth0 Management API (account linking).
 # Gated behind OIDC_MANAGEMENT_API_ENABLED — off by default. Create a

apps/infra/README.md

@@ -38,21 +38,29 @@ from the failed step.
 
 ## After first deploy
 
-One value needs to be fed back into `.env`:
+Nothing needs to be fed back into `.env`. The runner EC2 self-registers with the
+API on boot — v2 runners report their address via healthcheck — so sandboxes
+work as soon as the runner reaches `READY` (~30–60s), visible in the dashboard
+Runner table or `GET /admin/runners`.
 
-```bash
-# Get runner private IP:
-aws ec2 describe-instances --region ap-southeast-1 \
-  --filters "Name=tag:Name,Values=boxlite-runner" \
-  --query 'Reservations[].Instances[].PrivateIpAddress' --output text
+### Adding a runner
 
-# Add to .env:
-echo "RUNNER_PRIVATE_IP=10.0.x.y" >> .env
+The default runner is auto-seeded by the API at boot. To run more, set the total
+count and redeploy:
 
-# Redeploy (~2 min):
+```bash
+echo "RUNNERS=3" >> .env     # default runner (#1) + runner-2 + runner-3
 npx sst deploy --stage dev
 ```
 
+Each extra runner gets its own EC2 + minted token. Because the API only
+auto-seeds the single default, the extras are registered with the control plane
+by a post-deploy step (`RegisterExtraRunners` in `sst.config.ts`, which runs
+`scripts/register-runners.mjs` against the admin API once the API is healthy).
+It's idempotent — re-running `sst deploy` won't duplicate rows. Scaling **down**
+is the deliberate decommission ceremony under [Runner lifecycle](#runner-lifecycle),
+applied per runner.
+
 > **Note:** `CLOUDFRONT_DOMAIN` is no longer needed — SST Router resolves
 > it automatically via your `STACK_DOMAIN`. The dashboard's API base URL
 > is likewise derived: `DASHBOARD_BASE_API_URL` defaults to
@@ -339,9 +347,12 @@ self-heals.
 **"Organization is suspended: Please verify your email address"** — Auth0 access_token
 missing `email_verified` claim. Deploy the Post-Login Action described above.
 
-**Runner never registers with API** — `RUNNER_PRIVATE_IP` in `.env` is stale or
-missing. Get the current IP and redeploy. The runner also self-registers via
-`RUNNER_DOMAIN` set from EC2 instance metadata.
+**Runner never reaches `READY`** — the runner pairs to its DB row by token
+(`BOXLITE_RUNNER_TOKEN`, baked into the EC2's user-data, must equal the row's
+`apiKey`), then self-reports its address via `POST /runners/healthcheck` using
+`RUNNER_DOMAIN` (set from EC2 instance metadata at boot). Check the runner's
+systemd logs (`aws ssm start-session` → `journalctl -u boxlite-runner`) for auth
+or connectivity errors to the API.
 
 **Sandbox preview URL returns 503** — Proxy service may need a force-redeploy after
 initial setup: `aws ecs update-service --force-new-deployment --service Proxy`.

apps/infra/scripts/register-runners.mjs

@@ -0,0 +1,83 @@
+// SPDX-License-Identifier: AGPL-3.0-only
+// Copyright (c) 2026 BoxLite AI
+
+/*
+ * Post-deploy registration of extra runners with the control plane.
+ *
+ * The API only auto-seeds the single default runner (from DEFAULT_RUNNER_*), so
+ * additional runners declared via RUNNERS must be registered through the admin
+ * API. This script is invoked by the `RegisterExtraRunners` command in
+ * sst.config.ts after the extra runner EC2s and the API service are up.
+ *
+ * Pairing is token-based: the runner row's apiKey must equal the
+ * BOXLITE_RUNNER_TOKEN baked into the matching EC2's user-data. SST mints one
+ * token per runner and passes the (name, token) pairs here via RUNNERS.
+ *
+ * Idempotent: a 409 (runner already exists in the region) is treated as
+ * success, so redeploys are safe.
+ *
+ * Env:
+ *   API_URL        base URL of the API service (e.g. https://api.example.com)
+ *   ADMIN_API_KEY  admin-scoped API key (Bearer) for POST /api/admin/runners
+ *   REGION_ID      region to register the runners in (default "us")
+ *   RUNNERS        JSON array of { name, apiKey }
+ */
+
+const { API_URL, ADMIN_API_KEY, REGION_ID = 'us', RUNNERS } = process.env
+
+const runners = JSON.parse(RUNNERS || '[]')
+if (runners.length === 0) {
+  process.exit(0)
+}
+if (!API_URL || !ADMIN_API_KEY) {
+  console.error('register-runners: API_URL and ADMIN_API_KEY are required')
+  process.exit(1)
+}
+
+const base = API_URL.replace(/\/+$/, '')
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
+
+// Wait for the API to start serving. /api/health returns 200 only after the
+// HTTP server is listening, which (in onApplicationBootstrap) is after the
+// default region and admin user have been seeded — both prerequisites for the
+// admin POST below.
+async function waitForApi() {
+  for (let attempt = 1; attempt <= 60; attempt++) {
+    try {
+      const res = await fetch(`${base}/api/health`)
+      if (res.ok) return
+    } catch {
+      // not up yet
+    }
+    await sleep(5000)
+  }
+  throw new Error(`register-runners: ${base}/api/health not ready after 5 minutes`)
+}
+
+async function register({ name, apiKey }) {
+  const res = await fetch(`${base}/api/admin/runners`, {
+    method: 'POST',
+    headers: {
+      'content-type': 'application/json',
+      authorization: `Bearer ${ADMIN_API_KEY}`,
+    },
+    body: JSON.stringify({ name, apiKey, apiVersion: '2', regionId: REGION_ID }),
+  })
+
+  if (res.status === 201) {
+    console.log(`register-runners: ${name} registered`)
+    return
+  }
+  if (res.status === 409) {
+    console.log(`register-runners: ${name} already registered`)
+    return
+  }
+  const body = await res.text().catch(() => '')
+  throw new Error(`register-runners: ${name} failed (${res.status}): ${body}`)
+}
+
+await waitForApi()
+for (const runner of runners) {
+  await register(runner)
+}
+console.log(`register-runners: done (${runners.length} runner(s))`)

apps/infra/sst.config.ts

@@ -90,14 +90,6 @@ const requireOidcIssuer = () => {
   return v;
 };
 
-// Runner endpoint overrides — use RUNNER_PRIVATE_IP shortcut when set.
-const runnerEndpoint = (override: string, port: number, scheme: string) =>
-  envOr(
-    override,
-    process.env.RUNNER_PRIVATE_IP
-      ? `${scheme}${process.env.RUNNER_PRIVATE_IP}:${port}`
-      : `${scheme}localhost:${port}`,
-  );
 
 // ── app config ───────────────────────────────────────────────────────────────
 export default $config({
@@ -110,6 +102,8 @@ export default $config({
         aws: { region: REGION, profile: envOr("AWS_PROFILE", "default") },
         cloudflare: "6.14.0",
         random: "4.16.6",
+        // Post-deploy runner registration (see RegisterExtraRunners in run()).
+        command: "1.0.1",
       },
     };
   },
@@ -341,12 +335,11 @@ export default $config({
         ...registryEnv("TRANSIENT", registry.url),
         ...registryEnv("INTERNAL", registry.url),
 
-        // Default runner — wire via RUNNER_PRIVATE_IP after the first deploy
+        // Default runner — the API seeds it at boot from name + apiKey. v2
+        // runners self-report their address via healthcheck, so no domain/url
+        // wiring is needed here.
         DEFAULT_RUNNER_NAME: envOr("DEFAULT_RUNNER_NAME", "default"),
         DEFAULT_RUNNER_API_KEY: envOr("DEFAULT_RUNNER_API_KEY", defaultRunnerApiKey.result),
-        DEFAULT_RUNNER_DOMAIN: runnerEndpoint("DEFAULT_RUNNER_DOMAIN", PORTS.RUNNER, ""),
-        DEFAULT_RUNNER_API_URL: runnerEndpoint("DEFAULT_RUNNER_API_URL", PORTS.RUNNER, "http://"),
-        DEFAULT_RUNNER_PROXY_URL: runnerEndpoint("DEFAULT_RUNNER_PROXY_URL", PORTS.PROXY, "http://"),
 
         // PostHog (enables the dashboard's "Create Sandbox" feature flag)
         ...(process.env.POSTHOG_API_KEY && {
@@ -595,6 +588,62 @@ export default $config({
       ignoreChanges: ["ami", "userDataBase64"],
       protect: true,
     });
+
+    // ── Extra runners (RUNNERS > 1) ──────────────────────────────────────────
+    // The default runner above is auto-seeded by the API at boot via
+    // DEFAULT_RUNNER_*. The API has no multi-runner seed, so any additional
+    // runners are provisioned here and registered with the control plane after
+    // deploy via the admin API (RegisterExtraRunners below). Each gets its OWN
+    // token — pairing is token-based (the runner row's apiKey must equal the
+    // BOXLITE_RUNNER_TOKEN baked into the matching EC2's user-data) — and the
+    // same protect/ignoreChanges options as the default so routine deploys never
+    // replace a state-holding runner.
+    const totalRunners = Math.max(1, parseInt(envOr("RUNNERS", "1"), 10) || 1);
+    const extraRunners = Array.from({ length: totalRunners - 1 }, (_, i) => {
+      const name = `runner-${i + 2}`; // default is runner #1
+      const apiKey = randomKey(`RunnerApiKey-${name}`);
+      const instance = new aws.ec2.Instance(`Runner-${name}`, {
+        ami: ubuntuAmi.then((a) => a.id),
+        instanceType: RUNNER.instanceType,
+        subnetId: vpc.publicSubnets[0],
+        iamInstanceProfile: runnerInstanceProfile.name,
+        cpuOptions: { nestedVirtualization: "enabled" },
+        associatePublicIpAddress: true,
+        userDataBase64: $resolve([api.url, apiKey.result, registry.url]).apply(
+          ([apiUrl, token, registryUrl]) => buildRunnerUserData({ apiUrl, token, registryUrl }),
+        ),
+        rootBlockDevice: { volumeSize: RUNNER.rootDiskGB },
+        tags: { Name: `boxlite-runner-${name}` },
+      }, {
+        ignoreChanges: ["ami", "userDataBase64"],
+        protect: true,
+      });
+      return { name, apiKey, instance };
+    });
+
+    // Register the extra runners with the control plane once the API is healthy.
+    // Idempotent (treats HTTP 409 as success), so redeploys are safe; only re-runs
+    // when the API URL or the runner set changes.
+    if (extraRunners.length > 0) {
+      const runnersPayload = $resolve(extraRunners.map((r) => r.apiKey.result)).apply((keys) =>
+        JSON.stringify(extraRunners.map((r, i) => ({ name: r.name, apiKey: keys[i] }))),
+      );
+      new command.local.Command(
+        "RegisterExtraRunners",
+        {
+          create: "node scripts/register-runners.mjs",
+          update: "node scripts/register-runners.mjs",
+          environment: {
+            API_URL: api.url,
+            ADMIN_API_KEY: adminApiKey.result,
+            REGION_ID: envOr("DEFAULT_REGION_ID", "us"),
+            RUNNERS: runnersPayload,
+          },
+          triggers: [api.url, runnersPayload],
+        },
+        { dependsOn: extraRunners.map((r) => r.instance) },
+      );
+    }
   },
 });