restapi: full CLI parity + production readiness

[danielpaulus]

Brings the REST API to feature parity with the ios CLI and makes it production-grade. Auth model from #792 (GO_IOS_API_KEY / --disable-auth) is preserved. 56 new routes across 12 commits; go build/vet/test ./restapi/... green (incl. -race).

Production readiness

• Explicit http.Server with graceful shutdown (SIGINT/SIGTERM), stream-safe timeouts (ReadHeaderTimeout/IdleTimeout/MaxHeaderBytes, deliberately no WriteTimeout so streaming endpoints survive), optional TLS (--tls-cert/--tls-key), configurable bind (--addr).
• Unauthenticated /healthz + /readyz; swagger UI gated behind auth when a token is set.
• MustMarshal no longer panics (error envelope); RespondError helper for a consistent {"error":...} shape.
• New ios/golog-based structured logging (module=go-ios/restapi + udid + job attrs).

Endpoints (mirroring the exact library call each CLI command uses)

• Device info: devicename, date, battery, diagnostics, mobilegestalt, processes, lockdown
• Device mgmt: reboot, shutdown, erase (confirm-gated), devmode, lang, memlimitoff
• Files/crash: files ls/pull/push (stream-based → traversal-safe), crashes ls/rm
• Media: wallpaper, icon-layout, pasteboard
• Profiles/image: profile add/remove, image list/unmount
• Settings: assistivetouch, timeformat, wifi
• Monitoring: sysmontap (streaming)
• MDM (supervised): security-info, fetch-unlock-token, clear-passcode, clear-screen-time-password
• HTTP proxy: set/remove
• Tunnel (agent-level, /api/v1): GET /tunnels, DELETE /tunnels/:udid, POST /tunnels/:udid/refresh, POST /tunnel-agent/shutdown
• Async jobs: POST /jobs/runtest|runwda|forward, GET /jobs, GET /jobs/:id, GET /jobs/:id/logs (per-job streamed logs), DELETE /jobs/:id — long-running ops as background jobs with isolated, streamable logs; terminal state is immutable so a stop isn't relabeled a failure.

Tests

• Job manager + per-job log sink unit-tested (lifecycle, stop-is-terminal, per-device isolation, log stream/close) incl. -race.
• Production-readiness units (health, RespondError, no-panic marshal, flag parsing).
• Route-registration smoke test builds the full tree so gin route conflicts fail loudly.

Remaining / intentionally out

• ui (tap/swipe/button/type/app/…): the CLI's ui is an HTTP client proxying to WDA (:8100) / DeviceKit (:12004), not a go-ios library — see the open design note below. In the meantime UI automation is reachable through the API via POST /jobs/runwda + POST /jobs/forward (host→8100) then talking to WDA directly.
• runxctest (xctestrun-file upload) — planned follow-up on the job subsystem.
• pcap — deferred until ios/pcap exposes a packet-callback streaming API (it currently writes a local file).
• CLI-only: interactive debug (lldb) and dproxy; tunnel start (privileged daemon); sign/prepare create-cert (host-local, no device).

Follow-ups before merge

• Handler tests (httptest + mocked device) for the device-touching endpoints.
• Triage the two CI checks (CodeQL + the Linux e2e flake).

🤖 Generated with Claude Code

[stoktamisoglu]

Hi @danielpaulus , @shamanec , I couldn't find a better place to ask the question so I write here as comment to the latest pull request :).

Do you have any plan for the next release date even roughly? Thanks.

[danielpaulus]

Review round: adversarial security + correctness pass (pushed cbde822)

Reviewed all 56 endpoints for the four priority classes (files traversal, proxy SSRF, auth gaps, jobs races/resource exhaustion), got a second opinion from Codex, verified each claim, and fixed the confirmed defects with tests. go build/vet/test ./restapi/... green incl. -race; gofmt clean.

CI triage

• CodeQL (failing): not this PR. The only blocking annotation is ncm/ncm.go:185 go/allocation-size-overflow — a pre-existing alert (created 2024-06-11, open on main). This PR does not touch ncm/; zero alerts in any restapi/ file. CodeQL is mis-attributing a stale-baseline alert to the diff. Recommend dismissing/annotating alert Deviceinfo doesn't work #9 (maintainer action) — no code change is warranted here.
• macOS e2e (failing): confirmed infra flake, unrelated. TestTunnelLoad/.../syslog-b failed with ios [syslog]: no streamed output within 10s while the concurrent syslog-a streamed 11 MB fine — two concurrent syslog relays on one device, one starved. Lives in test/e2e/tunnel/perf_test.go; nothing to do with restapi/. (Linux e2e is green.)

Confirmed defects fixed

• [Med] Unbounded in-memory uploads — readFormFile and the raw-body reads in SetPasteboard/AddProfile used io.ReadAll with no cap; an authenticated client could OOM the daemon. Now routed through readAllLimited (256 MiB). PushFile already streams (Content-Length gated) and was fine.
• [Med] Lost log line in GET /jobs/:id/logs — snapshot() then subscribe() raced; a line written in between was dropped from the stream. Added jobLog.snapshotAndSubscribe() (backlog + subscription under one lock).
• [Med] Unbounded job-registry growth — terminal jobs (and their buffered logs) lived forever. DELETE /jobs/:id on a terminal job now purges it (remove, terminal-only); running jobs are still stopped, never silently dropped.

Reviewed and dismissed (not defects)

• Files traversal — claim holds. Pull streams to c.Writer, push streams from c.Request.Body; the only host-side use of the caller path is path.Base(remote) in Content-Disposition (test added). remote is device-side (iOS sandbox enforced), matching the CLI.
• Proxy "SSRF" — PUT /httpproxy sets the device's proxy config; no server-side fetch, so not SSRF.
• Auth coverage — every /api/v1 route is behind BearerAuth (health//swagger intentionally outside/gated). Added a tree-walk test asserting all 80 routes 401 unauthenticated.
• Jobs terminal-state immutability & double-close — finish is a no-op once non-running, jobLog.close is idempotent; a stop-then-late-finish stays stopped. Verified under -race.

Streaming framing inventory (for the separate SSE workstream — not changed here)

endpoint	new in #817	framing
/syslog	no	undelimited concatenated JSON
/listen	no	undelimited concatenated JSON
/notifications	no	NDJSON (\n)
/ostrace	no	NDJSON (\n)
/sysmontap	yes	NDJSON (\n)
/jobs/:id/logs	yes	raw text lines (lines carry \n)

Both new streaming endpoints are already newline-delimited. The two undelimited legacy streams (/syslog, /listen) are pre-existing and untouched by this PR; left for the SSE-alignment workstream to avoid scope creep here. None are true text/event-stream SSE yet.

Tests added (restapi/api/handlers_more_test.go)

Auth (accept/reject + all-routes-401), files (base-name sanitisation, ls/pull/push validation, push-411), upload limits, proxy/MDM multipart validation, jobs HTTP lifecycle, per-device isolation (cross-udid 404 + no-stop), remove-terminal-only, atomic snapshot+subscribe.

Merge-readiness

Code is merge-ready. Both red checks are non-blockers external to this PR (stale CodeQL baseline + a device-side syslog-concurrency e2e flake). Suggested pre-merge: dismiss CodeQL alert #9 on ncm/ncm.go and re-run the macOS e2e to confirm the flake clears.

[danielpaulus]

SDK-contract conformance pass

Made the REST daemon conform to the TypeSpec-authored OpenAPI contract the SDKs are generated from (spec/openapi/openapi.yaml x-sse-events + docs/DESIGN.md). API is not final, so no back-compat aliases were kept.

Real SSE framing on all six streaming endpoints

Replaced the previous NDJSON / concatenated-JSON writes with proper SSE frames (event: <name>\ndata: <compact-json>\n\n) plus a heartbeat frame on idle. A shared streamSSE helper (new restapi/api/sse.go) drives the frame loop, emits heartbeats via a ticker (reset on each real event), and flushes after every write. The text/event-stream + no-cache + chunked headers from the streaming middleware are unchanged.

Endpoint	event: name	payload model
/device/{udid}/syslog	syslog	SyslogMessage
/device/{udid}/notifications	appstate	AppStateNotification
/device/{udid}/ostrace	ostrace	OsTraceEntry
/device/{udid}/listen	attachdetach	AttachDetachEvent
/device/{udid}/sysmontap	sample	CpuUsageSample
/device/{udid}/jobs/{id}/logs	log	JobLogLine (backlog replayed as log frames, then live)
all	heartbeat	{}

Payload structs use the spec's camelCase field names. AttachDetachEvent.properties is emitted via a local DeviceProperties struct with camelCase JSON tags (the underlying ios.DeviceProperties has none and would marshal PascalCase).

Other spec fixes

• longtitude → longitude in SetLocation: query read, empty-check, error + success messages, and swagger @Param/@Description. No alias — the old misspelled param no longer satisfies the required field.
• Screenshot content-type now image/png (dropped the application/octet-stream override + redundant header line).
• Streaming error paths now use RespondError (GenericResponse envelope) instead of ad-hoc gin.H.

Tests (device-free)

restapi/api/sse_test.go + additions to device_endpoints_test.go:

• exact wire framing of writeSSEFrame (event/data/blank-line, compact JSON) and the heartbeat frame
• streamSSE emits typed frames and terminates on end-of-stream (via a fake next, no c.Stream/CloseNotifier dependency)
• heartbeat is emitted on idle (test-only interval override) then real events resume
• payload mappers (toAppStateNotification, toOsTraceEntry, toAttachDetachEvent) incl. camelCase JSON
• setLocation reads longitude (422 on missing; old longtitude rejected)

go build ./... && go vet ./restapi/... && go test ./restapi/... && go test -race ./restapi/... all green; gofmt -l clean.

Divergences deliberately left

• AppStateNotification key mapping is best-effort. The instruments channel yields an untyped map[string]interface{} whose exact keys for bundle id / state aren't captured in-repo; the mapper tries the likely key names and falls back to empty strings. Verifying/locking the real key names needs a device capture — flagging rather than guessing further (server-side, testable once captured).
• Screenshot 200 image/png path is only exercisable with a real device (the service connects on call), so the unit test asserts the longitude/error side; the content-type change itself is a one-line c.Data fix covered by the e2e suite.