feat(annotations): comment picker + clipboard markdown bundle for agents

[malopezr7]

Summary

Adds an opt-in Comment annotations mode to the preview UI. Click the picker icon, tap an element on the simulator, write a comment, repeat. When you're done, copy a markdown bundle to the clipboard with one click and paste it into Claude Code, Cursor, Codex — any agent that lives next to your editor.

The feature is gated behind a new "Comments" entry in the Tools panel, so existing users see no UI change until they opt in.

• Picker uses the live AX snapshot to draw a bbox sized to the element you're hovering and to auto-fill accessibilityLabel / role / type on the saved annotation.
• Crops are JPEG-encoded client-side from the visible MJPEG <img>, served back over HTTP so any agent with access to the preview port can pull the exact pixels of what the user pointed at.
• Storage is JSONL append-only at $TMPDIR/serve-sim/annotations/{udid}/, cleared on demand from the dropdown.

Demo

Default toolbar — feature is off, nothing changes	Tools panel — opt-in via the Comments / Enable toggle
Toolbar after Enable — picker (crosshair) + comments (chat bubble) join the toolbar	Picker hover — bbox sized to the AX element, badge shows <label> <role>
Composer — inline input anchored to the click; Enter submits, Esc cancels	Comments dropdown — pending list with delete, clear-all, and Copy

The crop saved to disk for the example above (256 px on the long side, JPEG q70):

Output for agents

Copy writes a single markdown document to the clipboard:

# Feedback from simulator session
Device: iPhone 17 Pro (UDID: 8B3BDAE3-83C2-4BC2-B4D1-F5D72A41250B)
Captured: 1 annotation · 2026-05-05T22:55:39.026Z

## [1] make this CTA bigger and more contrasted
- Element: button "Personalizar la página principal"
- Location: (200, 352)
- Screenshot: http://localhost:3200/api/annotations/02cae5aaf6ec/crop?device=8B3BDAE3-83C2-4BC2-B4D1-F5D72A41250B

The screenshot URLs hit the new /api/annotations/{id}/crop endpoint. An agent that runs locally next to the dev server can fetch the JPEG directly — no need to put base64 image data in the prompt. When the agent doesn't have HTTP access, the same bytes are also on disk under the device's annotations dir.

The on-disk record (one line of annotations.jsonl) carries everything the picker captured:

{
  "id": "02cae5aaf6ec",
  "createdAt": "2026-05-05T22:55:39.026Z",
  "device": { "udid": "8B3BDAE3-…", "name": "iPhone 17 Pro" },
  "point": { "x": 200, "y": 352 },
  "region": { "x": 36, "y": 334.33, "w": 330, "h": 34.33 },
  "screenshotPath": "crops/anno-02cae5aaf6ec.jpg",
  "context": {
    "accessibilityLabel": "Personalizar la página principal",
    "accessibilityRole": "button",
    "accessibilityType": "Button",
    "accessibilityId": "0.0.2"
  },
  "comment": "make this CTA bigger and more contrasted",
  "status": "pending"
}

How it works

Storage layer (src/annotations.ts, src/annotations-shared.ts)

• Append-only JSONL at $TMPDIR/serve-sim/annotations/{udid}/annotations.jsonl.
• Crops at crops/anno-{id}.jpg, optional full frames at frames/.
• Pure functions: listAnnotations, createAnnotation, updateAnnotation, deleteAnnotation, deleteAllAnnotations, resolveAnnotationAsset.
• Path-traversal guard on the asset resolver since IDs are reflected in HTTP responses.

HTTP API (mounted by middleware.ts and dev.ts)

GET    /api/annotations           list (filter via ?device=)
POST   /api/annotations           create from { point, region?, comment, cropDataUri, context? }
DELETE /api/annotations           clear all for device
GET    /api/annotations/:id       fetch one
PATCH  /api/annotations/:id       update comment / status
DELETE /api/annotations/:id       delete one
GET    /api/annotations/:id/crop  serve the JPEG crop
GET    /api/annotations/:id/frame serve the full-frame JPEG (if captured)

The Connect-style middleware and the Bun dev server share the storage layer; only the request/response plumbing differs.

Picker UI (src/client/annotations/index.tsx)

• Opt-in via Tools panel. Disabled by default; the topbar buttons, picker overlay, and dropdown only mount when annotationsEnabled is true. Toggling on auto-closes the panel so the toolbar change is visible. Toggling off resets all transient picker state.
• Hover hit-test runs against the existing AX SSE (/ax). The picker subscribes to that endpoint when active, regardless of whether the AX overlay is visible, so picking with the AX overlay off still gives you element-shaped bboxes.
• Click capture draws the visible MJPEG <img> onto a canvas, crops to the element bbox in physical pixels, scales the longest side down to 256 px, and posts a JPEG data URI to POST /api/annotations.
• AXe missing is a soft-degrade. Without axe on PATH (brew install cameroncooke/axe/axe) the picker still captures point + crop, just without an element-fitting bbox. The Tools panel surfaces an inline amber callout linking to the install command.
• Keyboard fix: serve-sim's HID-keyboard forwarder now bails when an INPUT / TEXTAREA / contentEditable is focused, so typing in the composer doesn't leak into iOS.

Future work (not in this PR)

The picker is currently AX-driven (Tier 3 + Accessibility tree). The same overlay can be powered by richer sources without changing the wire format — Annotation.context was designed with componentName, sourceFile, sourceLine, etc. fields ready to fill:

• React Native DevTools backend. react-devtools-core is embedded in any RN dev build; we can act as the headless client (the same mechanism react-native-debugger and Flipper use) and resolve the Fiber under the cursor to component name + file:line. That's the next PR — same UX, richer context.
• react-native-grab integration. When the host app already runs grab (Fabric ShadowTree, native IDs), we can prefer its pipeline for native-side metadata. Optional.

This PR is intentionally agnostic — no DevTools wiring, no react-native-grab dependency — so it ships value for any sim, RN or not, today.

Test plan

• bun install --frozen-lockfile
• bun run --filter serve-sim build
• bun test ./packages/serve-sim/src/__tests__
• bun test ./packages/serve-sim-client/src/__tests__/
• Open serve-sim against a booted iOS simulator → toolbar is unchanged.
• Open the Tools panel → see a new COMMENTS / Enable row.
• Click Enable → panel closes, picker (crosshair) and comments (chat bubble) appear in the toolbar.
• With axe on PATH: hover the simulator with picker active → bbox tracks the element under the cursor; badge shows <label> <role>.
• Without axe: open Tools while Comments is enabled → amber callout points at the install command.
• Click → composer appears, autofocused. Type, press Enter → annotation persists, badge counter increments, picker stays active for the next pick.
• Press Esc while typing → composer cancels.
• Open the Comments dropdown → list shows pending annotations with relative timestamps.
• Hover an item → delete chip appears.
• Click an item → bbox flashes over the simulator for ~1 s.
• Copy button → markdown bundle on the clipboard.
• Trash icon → first click confirms, second click clears.
• curl http://localhost:3200/api/annotations returns the persisted list.
• curl http://localhost:3200/api/annotations/<id>/crop returns a JPEG.
• Multiple devices: annotations are scoped per udid on disk and over the API.

[malopezr7]

Added a follow-up commit (21b1dd6) that wipes a device's annotations dir when serve-sim --kill runs — same lifecycle as the existing helper state file. Single-device kill purges $TMPDIR/serve-sim/annotations/{udid}/; --kill with no args purges the whole annotations root. Manual clear (per-item delete, dropdown trash-all, DELETE /api/annotations) is unchanged; this just stops the disk from growing across kill/restart cycles.