Merge pull request #3 from timzhong1024/feat/relative-coordinates

Add relative coordinates and improve modal/window handling

Author: timzhong1024

.github/workflows/build-cli.yml

@@ -53,7 +53,7 @@ jobs:
           echo "ARCHIVE_BASENAME=${ARCHIVE_BASENAME}" >> "${GITHUB_ENV}"
 
       - name: Upload packaged CLI
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v6
         with:
           name: ${{ env.ARCHIVE_BASENAME }}
           path: |

.gitignore

@@ -1,9 +1,14 @@
 .DS_Store
 /.build
+/.build-npm-arm64
+/.build-npm-x86_64
 /Packages
 xcuserdata/
 DerivedData/
 .swiftpm/configuration/registries.json
 .swiftpm/xcode/package.xcworkspace/contents.xcworkspacedata
 .netrc
 .vscode/launch.json
+/npm/bin/macos-cua
+/npm/LICENSE
+/npm/*.tgz

README.md

@@ -2,99 +2,129 @@
 
 `macos-cua` is a macOS-only low-level computer-use runtime for agents.
 
-It is designed around three defaults:
+Its default path is simple:
 
-- Coordinates default to the frontmost-window coordinate space.
-- The frontmost window is first-class.
-- Human-readable stdout is the default; add `--json` for structured output.
+- coordinates are absolute
+- coordinates are window-first
+- stdout is human-readable by default
 
-## Permissions
+## Install
 
-`macos-cua` relies on standard macOS permissions:
+Build locally:
 
-- `Accessibility`: required for synthetic mouse, keyboard, and window actions.
-- `Screen Recording`: required for screenshots.
-
-Use `macos-cua doctor` to inspect current readiness.
+```bash
+swift build
+```
 
-Use `macos-cua onboard` to trigger the native prompts, open the relevant System Settings panes, and guide a human through granting both permissions. In a tty session it waits by default; in non-tty mode it triggers the flow and returns immediately unless you pass `--wait`. When Screen Recording appears to have been granted but the process has not yet been restarted, `onboard` surfaces a targeted restart hint rather than a generic enable instruction. Add `--json` for structured output including per-permission `granted`, `waited`, and `likelyNeedsRestart` fields.
+Run with either:
 
-## Commands
+```bash
+swift run macos-cua <command>
+```
 
-```text
-macos-cua [--json] <command> [args...]
+or:
 
-onboard [--wait|--no-wait] [--timeout <seconds>] [--no-request] [--no-open]
-doctor
-state
-record enable|disable|status
-screenshot [--screen] [--region x y w h] <path.png>
-move <x> <y> [--screen] [--fast|--precise]
-click <x> <y> [left|right|middle] [--screen] [--fast|--precise]
-double-click <x> <y> [left|right|middle] [--screen] [--fast|--precise]
-scroll <dx> <dy>
-keypress <key[+key...]>
-type [--fast] <text>
-wait <ms>
-clipboard get|set|copy|paste
-app list|frontmost|activate
-window frontmost|list|activate|minimize|maximize|close
+```bash
+./.build/debug/macos-cua <command>
 ```
 
-## Examples
+## Authorize
+
+`macos-cua` needs two standard macOS permissions:
+
+- `Accessibility` for mouse, keyboard, and window actions
+- `Screen Recording` for screenshots
+
+First check readiness:
 
 ```bash
 swift run macos-cua doctor
+```
+
+Then start the built-in onboarding flow:
+
+```bash
 swift run macos-cua onboard
+```
+
+If you are running in a terminal and want it to wait for you:
+
+```bash
 swift run macos-cua onboard --wait --timeout 180
-swift run macos-cua --json onboard --no-wait
-swift run macos-cua record enable
-swift run macos-cua --json state
+```
+
+## Happy Path
+
+Daily use should start with absolute coordinates.
+
+1. Inspect the current coordinate space:
+
+```bash
+swift run macos-cua state
+```
+
+2. Capture the frontmost window:
+
+```bash
 swift run macos-cua screenshot /tmp/frontmost.png
-swift run macos-cua screenshot --screen /tmp/screen.png
+```
+
+3. Or Capture a region in the active coordinate space:
+
+```bash
 swift run macos-cua screenshot --region 100 100 300 200 /tmp/region.png
+```
+
+4. Move and click with absolute coordinates:
+
+```bash
 swift run macos-cua move 800 400 --precise
-swift run macos-cua move 800 400 --screen --precise
 swift run macos-cua click 800 400 --fast
+```
+
+5. Use screen-global coordinates only when needed:
+
+```bash
+swift run macos-cua screenshot --screen /tmp/screen.png
+swift run macos-cua move 800 400 --screen --precise
 swift run macos-cua click 800 400 --screen --fast
-swift run macos-cua keypress cmd+n
-swift run macos-cua type "hello from macos-cua"
-swift run macos-cua clipboard set "hello"
-swift run macos-cua clipboard paste
-swift run macos-cua app activate Code
-swift run macos-cua window list
 ```
 
-## Notes
-
-- Default `screenshot`, `move`, `click`, and `double-click` use frontmost-window coordinates.
-- `--screen` switches `screenshot`, `move`, `click`, and `double-click` to screen-global coordinates.
-- If no usable frontmost window is available, default coordinate-taking commands fall back to screen coordinates and report that fallback in output.
-- `window list` is AX-first when Accessibility is available, then falls back to CoreGraphics window discovery.
-- `window list`, `window frontmost`, and `state.frontmostWindow.bounds` remain screen-global diagnostics; they are not window-local action coordinates.
-- Missing permission errors point back to `macos-cua onboard` so agent and human flows land on the same recovery path.
-- Browser DOM/ref actions are intentionally out of scope for this repo.
-- `record enable` starts a persistent session under `~/Library/Application Support/macos-cua/records/`; each subsequent command appends an action log entry, a full-screen timeline screenshot, failure-only snapshots, and a replayable `replay.sh` trace until `record disable`.
-- A shareable VS Code debug example lives at `.vscode/launch.example.json`; local `.vscode/launch.json` stays ignored.
-- GitHub Actions can be triggered manually to build release CLI archives for both `arm64` and `x86_64` macOS runners.
-- Pointer movement anti-bot research notes live in [`docs/research/movement-anti-bot.md`](docs/research/movement-anti-bot.md).
-- `move`, `click`, and `double-click` use humanized pointer motion profiles; default is `--fast`, with `--precise` available for tighter target acquisition.
-
-## State Output
-
-- `state.defaultCoordinateSpace` reports whether default coordinate-taking commands currently resolve to `window` or `screen`.
-- `state.defaultCoordinateFallback` is `true` when default commands had to fall back from window coordinates to screen coordinates.
-- `state.pointerScreen` is the current pointer in screen-global coordinates.
-- `state.pointerWindow` is the current pointer relative to the frontmost window when available, otherwise `null`.
-- `state.frontmostWindow.bounds` stays in screen-global coordinates to make window-local to screen-global translation explicit.
-
-## Screenshot Resolution
-
-- Screenshot output is normalized to logical action-space dimensions, not Retina/native pixel dimensions.
-- This keeps screenshot coordinates aligned with the active coordinate space for `move` and `click` without requiring callers to divide by `scale`.
-- `actionSpace.width` and `actionSpace.height` still describe the main-screen logical action space.
-- For `screenshot --screen`, `image.width` and `image.height` match the main-screen action space.
-- For default window screenshots, `image.width` and `image.height` match the frontmost window size and the returned `bounds` are window-local.
-- For default region screenshots, `bounds` are interpreted in the active coordinate space and align with the returned raster.
-- Window screenshots are captured without the macOS drop shadow so the image edges line up with the reported window bounds.
-- Native pixel fidelity is intentionally discarded during screenshot export to preserve direct coordinate compatibility for agent actions.
+## Dense UI Fallback
+
+When a page is visually dense and the target is a small icon, URL, or toolbar
+item, treat `screenshot --region` as the fallback inspection step instead of
+guessing a final click from the full screenshot alone.
+
+Recommended pattern:
+
+1. Capture the full frontmost window for global context.
+2. Take a second local crop tightly around the likely target area.
+3. Re-read the local crop, then issue the final click.
+
+Example:
+
+```bash
+swift run macos-cua screenshot /tmp/frontmost.png
+swift run macos-cua screenshot --region 720 88 220 96 /tmp/toolbar-crop.png
+swift run macos-cua click 812 132 --fast
+```
+
+## Model Resolution
+
+Use absolute coordinates when your screenshot can be consumed at full useful resolution by the model. If the image must be resized or compressed before inference, switch to the relative-mode workflow in the docs.
+
+As of 2026-04-18, public vendor docs indicate the following:
+
+| Model | Documented vision resolution support |
+| --- | --- |
+| [`gpt-5.4`](https://developers.openai.com/api/docs/models/gpt-5.4) | Up to 6000 px max dimension with `detail: "original"`; 2048 px max dimension with `detail: "high"`. |
+| [`gpt-5.4-mini`](https://developers.openai.com/api/docs/models/gpt-5.4-mini) | Up to 2048 px max dimension with `detail: "high"`. |
+| [Claude Opus 4.7](https://platform.claude.com/docs/en/build-with-claude/vision) | Up to 2576 px on the long edge. |
+| [Claude Sonnet 4.5 / 4.6 and other current Claude vision models](https://platform.claude.com/docs/en/build-with-claude/vision) | Up to 1568 px on the long edge. See the [relative-mode workflow](docs/README.md#relative-mode-and-resized-images). |
+
+Older or weaker vision models are not recommended for coordinate-driven desktop control.
+
+## More
+
+Advanced workflows and reference material live under [docs/README.md](docs/README.md).