feat(remote): ios remote — minimal browser remote-control (WDA-free live screen)

[danielpaulus]

What it does

Adds ios remote [--port=<port>] [--driver=<driver>] [--devicekit-url=<url>] [--wda-url=<url>] [options] — a tiny, self-contained browser remote-control for a device. It serves an inline HTML/CSS/JS page (no external/CDN assets) that mirrors the screen and drives basic input. It binds 0.0.0.0:<port> (default 8080) so it is reachable over Tailscale.

Usage:

ios remote --udid=<udid> --port=8080
# then open http://<host>:8080/ in a browser

The UI is an <img src="/screen"> filling the page, with a click→tap / drag→swipe handler, a text box for typing, and Home / Lock / Vol+ / Vol- buttons.

The split: WDA-free screen + WDA-free (DeviceKit) input

Two independent halves, and both are now WDA-free:

• Live screen is WDA-FREE. GET /screen reuses the instruments screenshot service (ios/instruments) in a ~12 fps loop and streams JPEG frames as MJPEG (multipart/x-mixed-replace). This needs the tunnel + a mounted developer disk image, but no WebDriverAgent. A small in-package broadcaster fans the latest frame out to all clients and drops frames for slow consumers so a stalled browser can't back-pressure capture (no global state, unlike the existing StartMJPEGStreamingServer).
• Input uses DeviceKit by default (WDA is broken on iOS 26; DeviceKit works). /tap, /swipe, /type, /button shell out to the same ios binary (os.Executable) and reuse the proven ios ui … commands, targeting the device udid and the selected driver: --driver=devicekit (default, --devicekit-url, default http://127.0.0.1:12004 or GO_IOS_DEVICEKIT_URL) or --driver=wda (--wda-url, default http://127.0.0.1:8100 or GO_IOS_WDA_URL). So the screen works with no driver running; input needs a reachable driver. Buttons (home/lock/volume) route through DeviceKit, which supports them.

Coordinate mapping is in one place (fractionToPoints): the browser sends each click as a 0..1 fraction of the displayed image, and the server maps it linearly to the driver's logical points using the size fetched once via ios ui size on the same driver (cached), never a hardcoded/WDA size. parseSize understands DeviceKit's device.info JSON-RPC envelope ({"result":{"screenSize":{width,height},"scale"}}) as well as the WDA window/size and bare shapes. On an iPhone SE 3rd gen DeviceKit reports 375×667 logical points (MJPEG frames are points × scale = 750×1334 px). Keeping the browser in fraction-space means it never has to know the device's pixel/point sizes or CSS letterboxing.

Runner supervision — ios remote self-heals input (NEW)

Input intermittently broke and stayed broken until a human restarted the runner. Root cause: the input path uses a separate long-running ios ui run devicekit process (JSON-RPC at http://127.0.0.1:12004). That runner's on-device XCTest/testmanagerd automation channel intermittently drops — DTX Connection with EOF → conn2 closed unexpectedly error=EOF → the process exits rc=1 (ui run: runner failed: lost connection to testmanagerd). It is not memory/jetsam and not a deterministic load threshold; it's an iOS-side disconnect we can't prevent. Previously ios remote shelled out per action and did not own the runner, so once it died every action returned connection refused forever.

ios remote now supervises the DeviceKit runner (default, DeviceKit driver):

• Spawns ios ui run devicekit --udid=<udid> as a child of the same binary (os.Executable), detecting readiness by polling the DeviceKit health endpoint with a ~60s startup timeout.
• Monitors + auto-respawns: when the child dies it logs the exit (reason/rc) at WARN and respawns with capped exponential backoff (1s→10s), resetting the backoff after a runner stays up >60s — never spin-looping hot.
• Lifecycle state (starting|ready|restarting|down) is exposed at GET /status (and /healthz) and surfaced in the UI ("input runner restarting…").
• 503 during recovery: /tap /swipe /type /button return 503 {"error":"input runner starting, retry shortly","runnerState":"…"} while not ready, instead of failing with connection-refused.
• Clean shutdown: SIGINT/SIGTERM terminates the child (own process group) so no orphaned ios ui run devicekit is left behind.

--no-manage-runner keeps the old behavior (connect to an externally-run runner at --devicekit-url); --runner-driver selects which driver's runner to supervise (only devicekit is spawnable). The screen stream's tunnel self-heal is unchanged. Rich golog lifecycle logging (module go-ios/remote, udid attr): spawned (pid), ready (elapsed), exited (reason/rc), restarting (attempt/backoff). The spawn/health seam is a small interface, so the supervisor state machine is unit-tested without a device (fake death → respawn, backoff reset, shutdown stops child, input 503 when not ready).

Tunnel-info self-refresh on reconnect (hardening)

The screen service used to cache the tunnel address for the process lifetime, so when the device's tunnel changed (CI run ends, replug, tunnel daemon restart) the stream went blank forever with dial … i/o timeout / "reconnect failed". On each screen-service reconnect we now re-fetch fresh tunnel/RSD info via a non-fatal resolver and dial that, mirroring how a fresh ios screenshot re-resolves each call — so the mirror self-recovers instead of dying.

Native iOS-26 touch — follow-up

DeviceKit already gives WDA-free input on iOS 26. A further follow-up is native in-process touch injection (e.g. via CoreDevice / a HID path) so ios remote needs no external UI driver at all.

Options considered

• Screen via WDA MJPEG (ios ui stream mjpeg) — rejected: it needs a driver just to see the screen, defeating the "screen works standalone" goal. The instruments screenshot service is driver-free and already present.
• Re-implement a driver HTTP client in the new package for input — rejected: shelling out to ios ui reuses battle-tested code paths (driver selection, session handling, env/URL resolution) instead of duplicating them; the cost is one subprocess per input event, which is negligible for a human-driven remote.
• Reuse the global-state StartMJPEGStreamingServer/mjpegHandler — rejected: its sync.Map/global conversion-queue design can't be instanced cleanly; the small local broadcaster is instanceable and testable and passes JPEG through untouched.
• Browser sends pixel coords + image dims — rejected in favor of pure fractions: fractions keep all device-size knowledge server-side in a single mapping function.

Tests

go build ./..., go vet ./..., gofmt -l, go test ./... are clean. Unit tests cover driver defaulting (empty → DeviceKit), the --devicekit-url/--wda-url flag selection, coordinate mapping against the driver's logical size (including DeviceKit's 375×667), and ui size parsing for the DeviceKit JSON-RPC envelope + DeviceKit screenSize + WDA envelope + bare + noisy output, plus that / serves the HTML UI with 200 and /button rejects unknown buttons.

🤖 Generated with Claude Code

Deployment status (office01, iOS 26.5, udid 00008110-001C58C00A88401E)

Deployed and live on the office01 CI Mac over Tailscale, driving input via DeviceKit (WDA-free), reachable at http://office01:8090/.

• Screen mirror: WORKS. GET /screen streams live JPEG frames (~28 frames / 5 s captured, ~2.7 MB, valid JPEG SOI markers). Fully WDA-free.
• Input via DeviceKit: WORKS. POST /tap at multiple coordinates, POST /type, and POST /button (lock) all return {"result":{"success":true}} and actually act on the iOS 26.5 device. The server logs driver: devicekit, driverURL: http://127.0.0.1:12004.
• Swipe: blocked by a pre-existing DeviceKit ui swipe bug, not by ios remote. POST /swipe (and a plain ios ui swipe --driver=devicekit) returns Invalid parameters … data missing from DeviceKit's device.io.swipe RPC. Tap/type/button are unaffected; fixing the DeviceKit swipe param schema is separate from this PR.

This supersedes the earlier WDA-only status: WDA v13.1.3 on iOS 26.5 returned Unhandled endpoint for tap/homescreen/keys, so ios remote now defaults to DeviceKit.

Earlier robustness fixes remain in this PR: resolve tunnel/RSD info for ios remote (instruments needs it on iOS 17+); read --port through docopt's repeatable-list shape; send integer tap coordinates (ios ui tap rejects decimals); self-heal the screen stream on reconnect.

Usage:

ios remote --udid=<udid> --port=8090            # DeviceKit input (default)
ios remote --udid=<udid> --driver=wda           # opt back into WDA
# open http://<tailscale-host>:8090/ in a browser