Skip to content

Commit 3f07e70

Browse files
chore: add device selection tools (#7)
<!-- Thanks for your contribution! Take a moment to answer these questions so that reviewers have the information they need to properly understand your changes: * What is the current state of things and why does it need to change? * What is the solution your changes offer and how does it work? Are there any issues or other links reviewers should consult to understand this pull request better? For instance: * Fixes #12345 * See: #67890 -->
1 parent dee87fd commit 3f07e70

20 files changed

Lines changed: 2090 additions & 225 deletions

README.md

Lines changed: 61 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# @metamask/device-mcp
22

3-
MCP server for mobile device interaction — iOS (IDB), Android (ADB), and remote devices (Appium/BrowserStack).
3+
MCP server for mobile device interaction — iOS (simctl + IDB), Android (ADB), and remote devices (Appium/BrowserStack).
44

55
Provides device interaction tools for LLM agents to inspect UI state, interact with elements, capture evidence, and control app lifecycle. Works standalone for debugging or as part of the [self-healing test infrastructure](https://github.com/MetaMask/metamask-mobile) for MetaMask Mobile.
66

@@ -15,8 +15,8 @@ Provides device interaction tools for LLM agents to inspect UI state, interact w
1515
## Requirements
1616

1717
- **Node.js** `^20 || ^22 || >=24`
18-
- **iOS local**: [IDB](https://fbidb.io/) (`brew tap facebook/fb && brew install idb-companion && pip3 install fb-idb`)
19-
- **Android local**: ADB (Android SDK platform-tools)
18+
- **iOS local**: Xcode Command Line Tools (for `xcrun simctl`) + [IDB](https://fbidb.io/) for UI interaction (`brew tap facebook/fb && brew install idb-companion && pip3 install fb-idb`)
19+
- **Android local**: ADB (Android SDK platform-tools) — auto-discovered from `$ANDROID_HOME`, `$ANDROID_SDK_ROOT`, or `~/Library/Android/sdk`
2020
- **Remote/BrowserStack**: No local tools needed — connects via Appium W3C WebDriver HTTP
2121

2222
## Installation
@@ -41,15 +41,32 @@ device-mcp
4141

4242
# Target a specific device
4343
DEVICE_ID=<udid-or-serial> device-mcp
44+
45+
# Target a specific platform (useful in CI with one device per platform)
46+
DEVICE_PLATFORM=ios device-mcp
47+
DEVICE_PLATFORM=android device-mcp
4448
```
4549

4650
### Backend Selection
4751

4852
The server selects a backend in this order:
4953

5054
1. **`.device-session` file** — if present in the working directory, connects via Appium (local or BrowserStack)
51-
2. **`DEVICE_ID` env var** — format determines platform (UUID = iOS, serial = Android)
52-
3. **Auto-detect** — checks for booted iOS simulator (IDB), then connected Android device (ADB)
55+
2. **`DEVICE_ID` + `DEVICE_PLATFORM`** — direct connect, no auto-detection
56+
3. **`DEVICE_ID` only** — platform inferred from format (UUID = iOS, serial/emulator-\* = Android)
57+
4. **`DEVICE_PLATFORM` only** — auto-detect first device of that platform
58+
5. **Nothing set, 1 device** — auto-connect
59+
6. **Nothing set, multiple devices** — returns device list, agent asks user to pick via `device_select_device`
60+
61+
### Multi-Device Selection
62+
63+
When multiple devices are connected and no `DEVICE_ID` is set, the server enters an "awaiting selection" state. Any tool call returns the list of available devices. Use `device_list_devices` to enumerate them and `device_select_device` to choose one.
64+
65+
### Device Discovery
66+
67+
- **iOS simulators** are discovered via `xcrun simctl list devices booted --json` — no IDB needed for discovery
68+
- **Android devices** are discovered via `adb devices` — the server probes `$ANDROID_HOME/platform-tools/adb`, `$ANDROID_SDK_ROOT/platform-tools/adb`, and `~/Library/Android/sdk/platform-tools/adb` when `adb` is not on `$PATH`
69+
- **IDB** is resolved from `$PATH`, `/usr/local/bin`, `/opt/homebrew/bin`, and `~/Library/Python/*/bin` (pip user installs)
5370

5471
### BrowserStack / Appium
5572

@@ -102,6 +119,13 @@ The `.device-session` file is typically written by the test runner when it creat
102119

103120
## Tools
104121

122+
### Device Management
123+
124+
| Tool | Description |
125+
| ---------------------- | ------------------------------------------------------------------ |
126+
| `device_list_devices` | List all connected devices and simulators/emulators. |
127+
| `device_select_device` | Select a device for this session. Use after `device_list_devices`. |
128+
105129
### Inspection
106130

107131
| Tool | Description |
@@ -142,24 +166,26 @@ Elements are identified by accessibility attributes — not internal refs. Match
142166

143167
### Backend Implementation
144168

145-
| Tool | iOS (IDB) | Android (ADB) | Appium (W3C WebDriver) |
146-
| ------------------------- | ----------------------- | -------------------- | ----------------------------- |
147-
| `device_snapshot` | `idb ui describe-all` | `uiautomator dump` | `mobile: source` |
148-
| `device_screenshot` | `idb screenshot` | `screencap` + `pull` | `mobile: getScreenshot` |
149-
| `device_info` | `idb describe` | `getprop` | session capabilities |
150-
| `device_tap_element` | find + `idb ui tap` | find + `input tap` | find + W3C Actions |
151-
| `device_tap_coordinates` | `idb ui tap x y` | `input tap x y` | W3C Actions |
152-
| `device_type` | `idb ui text` | `input text` | `findElement` + `sendKeys` |
153-
| `device_swipe` | `idb ui swipe` | `input swipe` | W3C Actions |
154-
| `device_long_press` | `idb ui tap --duration` | `input swipe` (hold) | W3C Actions (pause) |
155-
| `device_wait_for` | poll snapshot | poll snapshot | poll snapshot |
156-
| `device_app_state` | `idb list-apps` | `dumpsys activity` | `mobile: queryAppState` |
157-
| `device_open_app` | `idb launch` | `monkey -p` | `mobile: activateApp` |
158-
| `device_close_app` | `idb terminate` | `am force-stop` | `mobile: terminateApp` |
159-
| `device_press_button` | `idb ui key` | `input keyevent` | `mobile: pressButton/Key` |
160-
| `device_dismiss_keyboard` | `idb ui key RETURN` | `input keyevent 111` | `mobile: hideKeyboard` |
161-
| `device_dismiss_alert` | find button + tap | find button + tap | `mobile: accept/dismissAlert` |
162-
| `device_logs` | `idb log` | `logcat` | `mobile: getLog` |
169+
| Tool | iOS (IDB) | Android (ADB) | Appium (W3C WebDriver) |
170+
| ------------------------- | ------------------------------------ | -------------------- | ----------------------------- |
171+
| `device_snapshot` | `idb ui describe-all` | `uiautomator dump` | `mobile: source` |
172+
| `device_screenshot` | `idb screenshot` | `screencap` + `pull` | `mobile: getScreenshot` |
173+
| `device_info` | `idb describe` | `getprop` | session capabilities |
174+
| `device_tap_element` | find + `idb ui tap` | find + `input tap` | find + W3C Actions |
175+
| `device_tap_coordinates` | `idb ui tap x y` | `input tap x y` | W3C Actions |
176+
| `device_type` | `idb ui text` | `input text` | `findElement` + `sendKeys` |
177+
| `device_swipe` | `idb ui swipe` | `input swipe` | W3C Actions |
178+
| `device_long_press` | `idb ui tap --duration` | `input swipe` (hold) | W3C Actions (pause) |
179+
| `device_wait_for` | poll snapshot | poll snapshot | poll snapshot |
180+
| `device_list_devices` | `xcrun simctl list` | `adb devices` | N/A |
181+
| `device_select_device` | select by UDID | select by serial | N/A |
182+
| `device_app_state` | `idb list-apps` / `simctl listapps` | `dumpsys activity` | `mobile: queryAppState` |
183+
| `device_open_app` | `idb launch` / `simctl launch` | `monkey -p` | `mobile: activateApp` |
184+
| `device_close_app` | `idb terminate` / `simctl terminate` | `am force-stop` | `mobile: terminateApp` |
185+
| `device_press_button` | `idb ui key` | `input keyevent` | `mobile: pressButton/Key` |
186+
| `device_dismiss_keyboard` | `idb ui key RETURN` | `input keyevent 111` | `mobile: hideKeyboard` |
187+
| `device_dismiss_alert` | find button + tap | find button + tap | `mobile: accept/dismissAlert` |
188+
| `device_logs` | `idb log` | `logcat` | `mobile: getLog` |
163189

164190
## MCP Client Configuration
165191

@@ -172,15 +198,14 @@ Add to `~/.config/opencode/opencode.json`:
172198
"mcp": {
173199
"device": {
174200
"type": "local",
175-
"command": ["node", "/path/to/metamask-device-mcp/dist/index.js"],
176-
"environment": {
177-
"PATH": "/path/to/idb/bin:/usr/local/bin:/usr/bin:/bin"
178-
}
201+
"command": ["npx", "-y", "@metamask/device-mcp"]
179202
}
180203
}
181204
}
182205
```
183206

207+
IDB and ADB are auto-discovered from standard install locations. No `PATH` override needed unless tools are installed in custom directories.
208+
184209
### Cursor
185210

186211
Add to `.cursor/mcp.json` in your project root:
@@ -235,27 +260,30 @@ Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
235260
@metamask/device-mcp
236261
├── src/
237262
│ ├── index.ts # Entry point — lazy backend, stdio MCP server
238-
│ ├── server.ts # MCP server — registers 16 tools
263+
│ ├── server.ts # MCP server — registers 25 tools
239264
│ ├── backends/
240-
│ │ ├── types.ts # DeviceBackend interface (16 operations)
241-
│ │ ├── idb-backend.ts # iOS local — IDB commands + JSON parser
265+
│ │ ├── types.ts # DeviceBackend interface
266+
│ │ ├── idb-backend.ts # iOS local — IDB commands + simctl fallback
242267
│ │ ├── adb-backend.ts # Android local — ADB commands + XML parser
243268
│ │ ├── appium-backend.ts # Remote — Appium/BrowserStack via W3C WebDriver
244269
│ │ ├── webdriver-client.ts # Minimal W3C WebDriver HTTP client (fetch)
245270
│ │ ├── session-file.ts # .device-session file reader
246271
│ │ └── index.ts # createBackend() + createLazyBackend() factory
247-
│ ├── tools/ # One file per MCP tool (16 tools)
272+
│ ├── tools/ # One file per MCP tool (25 tools)
273+
│ │ ├── list-devices.ts # device_list_devices — enumerate connected devices
274+
│ │ ├── select-device.ts # device_select_device — choose device for session
275+
│ │ └── ... # snapshot, tap, type, swipe, etc.
248276
│ └── utils/
249277
│ ├── exec.ts # Shell execution wrapper
250-
│ ├── platform.ts # Platform auto-detection
278+
│ ├── platform.ts # Device discovery (simctl, adb), path resolution
251279
│ └── element.ts # Element search, matching, formatting
252280
```
253281

254282
## Development
255283

256284
```bash
257285
yarn build # Compile TypeScript
258-
yarn test # Run tests (69 tests)
286+
yarn test # Run tests
259287
yarn lint # Lint everything (ESLint + Prettier + changelog)
260288
yarn lint:fix # Auto-fix lint issues
261289
yarn dev # Watch mode compilation

SKILL.md

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# @metamask/device-mcp — Agent Reference
22

3-
You interact with a mobile device (iOS simulator, Android emulator, or BrowserStack remote device) through the `device-mcp` MCP server. It exposes 16 tools for inspecting UI state, interacting with elements, capturing evidence, and controlling app lifecycle.
3+
You interact with a mobile device (iOS simulator, Android emulator, or BrowserStack remote device) through the `device-mcp` MCP server. It exposes 25 tools for device management, inspecting UI state, interacting with elements, capturing evidence, and controlling app lifecycle.
44

55
## When to Use This
66

@@ -10,6 +10,17 @@ You interact with a mobile device (iOS simulator, Android emulator, or BrowserSt
1010
- **Verifying UI changes** — check that a screen looks right after a code change
1111
- **Self-healing test recovery** — find alternative paths when an element is missing
1212

13+
## Getting Started — Device Selection
14+
15+
When multiple devices are connected, the server won't auto-pick one. Use the device management tools first:
16+
17+
```
18+
device_list_devices # See all connected simulators/emulators/devices
19+
device_select_device # Pick the one you want to use
20+
```
21+
22+
If only one device is connected, it's auto-selected. If `DEVICE_ID` is set, this step is skipped.
23+
1324
## Core Loop
1425

1526
```
@@ -29,6 +40,30 @@ device_snapshot # 3. See the result — NEVER assume screen state
2940

3041
## Tools
3142

43+
### device_list_devices (read-only)
44+
45+
List all connected devices and simulators/emulators. Returns platform and device ID for each.
46+
47+
**When:** Multiple devices are connected and you need to pick one. Checking what's available before starting.
48+
49+
**Output format:**
50+
51+
```
52+
Platform Device ID
53+
ios FACF3006-1FF8-482A-B6EF-58995E1DF1CB
54+
android emulator-5554
55+
```
56+
57+
### device_select_device
58+
59+
Select a device to use for this session. Call `device_list_devices` first to see available devices.
60+
61+
```json
62+
{ "deviceId": "FACF3006-1FF8-482A-B6EF-58995E1DF1CB" }
63+
```
64+
65+
**When:** After `device_list_devices` shows multiple devices. The agent should present the list to the user and let them choose.
66+
3267
### device_snapshot (read-only)
3368

3469
Capture the full UI accessibility hierarchy. Returns every visible element with type, label, identifier, value, frame coordinates, and enabled state.
@@ -204,6 +239,15 @@ Capture recent device logs with optional text filter.
204239

205240
## Common Patterns
206241

242+
### Select a device when multiple are connected
243+
244+
```
245+
device_list_devices # see all devices
246+
# present list to user, get their choice
247+
device_select_device { "deviceId": "FACF3006-..." } # select device
248+
device_snapshot # start working
249+
```
250+
207251
### Navigate to a screen and verify
208252

209253
```
@@ -299,6 +343,10 @@ xcrun simctl boot "iPhone 16"
299343
emulator -avd Pixel_7_API_34
300344
```
301345

346+
iOS simulators are discovered via `xcrun simctl` (ships with Xcode). ADB is auto-discovered from `$ANDROID_HOME`, `$ANDROID_SDK_ROOT`, or `~/Library/Android/sdk`. IDB is auto-discovered from `$PATH`, `/usr/local/bin`, `/opt/homebrew/bin`, or `~/Library/Python/*/bin`.
347+
348+
No `PATH` configuration is needed in the MCP client — tools are found automatically.
349+
302350
### Remote (BrowserStack/Appium)
303351

304352
Create a `.device-session` file — see the README for the full schema. The test runner typically writes this when starting the Appium session.

0 commit comments

Comments
 (0)