fix(tunnel): TunnelManager self-heals dead tunnels, stops retrying unsupported devices, tested udid filter

[danielpaulus]

Stacked PR — based on #808. This branch includes #808's commit (the shared dial-timeout helper its liveness probe reuses) and should merge after #808; once #808 lands, this diff reduces to the TunnelManager changes only.

Problem

Three long-standing tunnel-agent problems, all in TunnelManager.UpdateTunnels:

1. TunnelManager agent doesn't auto-recover a dead-but-still-present tunnel (reboot test only covers explicit refresh) #765 — no self-heal for dead-but-still-present tunnels. A tunnel that dies without a usbmux disconnect (quick reboot the mux never observes, transport death while the TUN route stays up) remains a stale record forever: the create path skips devices that already have a record, and teardown only fires for devices that vanish from ListDevices(). Every consumer then dials the dead endpoint (135s each before fix(tunnel): bound tunnel/RSD TCP dials with a 15s timeout #808/Kernel tunnel: device-unreachable mid-run causes 135s no-timeout RSD dials → e2e job timeout #764). Observed live on the Linux CI runner across two runs.
2. Using ios <17 and >=17 together is generating a huge amount of logging #523 — sub-iOS17 devices spam warnings forever. With mixed iOS <17 / 17+ fleets, every update cycle logs failed to start tunnel ... unsupported iOS version for each old device, escalating with device count.
3. How to tunnel start for multiple iOS devices? #607 / Is it possible to remove the output of the tunnel start command or to start or stop the tunnel by udid with ios commands? #479 — per-device tunnel management. Users asked for ios tunnel start --udid to manage a single device's tunnel. The TunnelManager.udidFilter plumbing exists (NewTunnelManagerForDevice, wired from cmd_tunnel.go with help text in main.go) but had zero unit-test coverage guarding it.

Root cause

UpdateTunnels had exactly two behaviors: create-if-absent and teardown-if-vanished. There was no liveness notion for existing records, and no classification of permanent vs transient start failures — everything took the retry/backoff path.

Fix

1. Liveness probing (TunnelManager agent doesn't auto-recover a dead-but-still-present tunnel (reboot test only covers explicit refresh) #765). Existing records of still-connected devices are probed behind a new tunnelProber seam, at most once per 30s per device (defaultProbeInterval). The default rsdProber is a 5s-timeout TCP dial of the tunnel's RSD endpoint using fix(tunnel): bound tunnel/RSD TCP dials with a 15s timeout #808's ios.DialTunnelTCPWithTimeout; userspace tunnels are skipped (the localhost forwarder accepts connects regardless of device state, so a dial proves nothing). On probe failure the record is torn down via the per-device stop machinery from Add per-device tunnel stop and refresh #738 and the create loop rebuilds it in the same cycle. Fresh tunnels defer their first probe by a full interval.
2. Permanent-failure classification (Using ios <17 and >=17 together is generating a huge amount of logging #523). manualPairingTunnelStart wraps its version-based failures (iOS <17, and userspace on 17.0–17.3) in a new ErrUnsupportedVersion sentinel. UpdateTunnels logs those once at info and marks the udid permanently skipped for the process lifetime (restart the agent after upgrading a device's OS). Transient errors keep the existing failedDevice exponential backoff untouched.
3. udid filter (How to tunnel start for multiple iOS devices? #607/Is it possible to remove the output of the tunnel start command or to start or stop the tunnel by udid with ios commands? #479). The --udid filter path is now locked in by unit tests: only the matching device is attempted/tunneled, empty filter keeps managing all devices. No behavior change needed — the plumbing and ios tunnel start help text are already on main; this closes the issues with the behavior now guaranteed by tests.

Also adds a productVersion seam (defaults to ios.GetProductVersion) so the start path is unit-testable without a device.

Options considered

• Probe = short-timeout RSD TCP dial (chosen) vs a full RSD/RemoteXPC handshake: the incident signature is a dead route (SYN blackhole), which a plain connect detects; a handshake adds protocol weight and failure modes for little extra signal. The tunnelProber interface leaves room to upgrade later.
• Probe cadence: every-cycle probing (1s) would dial each device every second and stall the loop up to 5s per dead device; a 30s per-device interval bounds overhead while detecting death well under a minute.
• Using ios <17 and >=17 together is generating a huge amount of logging #523 blacklist on any first failure (patch suggested in the issue thread): too aggressive — a transient pairing hiccup would permanently disable a healthy device. Classifying only version errors as permanent keeps backoff for everything genuinely retryable.
• Pruning the unsupported set on disconnect (device might be OS-upgraded): rejected for now — issues ask for silence; process-lifetime skip is the simplest contract and an agent restart re-detects.

Test plan

New device-free tests in ios/tunnel/tunnel_manager_robustness_test.go (fake starter/lister/prober/version seams), each failing without its change:

• TestUpdateTunnelsRebuildsDeadTunnel — device still listed, prober says dead → old record closed exactly once, tunnel restarted, new record served.
• TestUpdateTunnelsKeepsHealthyTunnel — healthy record probed but untouched.
• TestUpdateTunnelsDoesNotProbeDisconnectedDevice — vanished devices stay with the disconnect teardown.
• TestShouldProbeRespectsInterval, TestRsdProberSkipsUserspaceTunnels.
• TestManualPairingTunnelStartClassifiesUnsupportedVersions — iOS 16.x and userspace-17.2 both classify as ErrUnsupportedVersion.
• TestUpdateTunnelsSkipsUnsupportedDevicePermanently — one attempt across three cycles, no backoff entry; TestUpdateTunnelsKeepsBackoffForTransientFailures — transient errors still back off, never marked unsupported.
• TestUpdateTunnelsUdidFilter — filter attempts only the matching device; empty filter manages all.

go build ./..., go test ./... and go test -race ./ios/tunnel/ green; gofmt -l clean. Real-device e2e via /test-devices.

Fixes #765, fixes #523, fixes #607, fixes #479

🤖 Generated with Claude Code

https://claude.ai/code/session_01J8eMENxJ1nec9CeHp4tjWk

[danielpaulus]

/test-devices

[github-actions]

🧪 Running real-device tests on PR #809 — run.

[github-actions]

❌ Real-device tests failed — see run.

[danielpaulus]

Adversarial review + fixes (pushed to this branch as `0352b30`).

Verdict: logic was sound (probe lives outside the mux, map access is consistently mutex-guarded, `%w` classification and `errors.Is` work, teardown/`lastProbe` cleanup was complete, no orphaned utun). Fixed two confirmed defects and one bounded-growth issue in the self-heal path:

1. Probe starvation (HIGH). The liveness-probe loop ran serially inside `UpdateTunnels`; each dead-but-still-routed tunnel blackholes the SYN for the full `probeDialTimeout` (5s), so N dead tunnels could delay tunnel creation for newly connected devices by up to 5s×N. Probes now run concurrently, bounded by `maxConcurrentProbes` (8) via a semaphore + WaitGroup, collected under the mux.

2. False-positive teardown (MEDIUM). A single failed probe immediately tore down a healthy tunnel. Teardown now requires `probeFailureThreshold` (2) consecutive failures, reset on any success (new `probeFailures` map, mutex-guarded), so a momentary route hiccup/load spike can't destroy a healthy tunnel. Trade-off: a persistently dead tunnel is now rebuilt in ~2 probe intervals (~60s) instead of ~30s — documented on `defaultProbeInterval`.

3. Bounded-growth / stale-after-upgrade (MEDIUM). `unsupportedDevices` (and the new `probeFailures`) are now pruned for disconnected devices in the disconnect teardown, so they stay bounded by current device count, and a device that was unsupported → upgraded to a tunnel-capable iOS → reconnected is retried instead of skipped for the process lifetime.

New tests: two-failure threshold, transient-failure absorption (counter reset), unsupported-on-disconnect pruning + reconnect-retry. Made `fakeProber` thread-safe (probes are concurrent now).

Dismissed: the exact-`17.4.0` userspace-TUN boundary (`GreaterThan(17.4.0)`) is pre-existing and unchanged by this PR (only the error string was reworded). No transient-version misclassification risk: `GetProductVersion` either returns the real version or errors (→ transient-failure backoff, not the permanent set).

Verification (both branches): `go build ./...`, `go test ./...`, `go test -race ./ios/tunnel/` all green; `gofmt -l` clean.

Note: #808 (base of this stack) was reviewed and confirmed clean — no changes needed there.