Skip to content

Commit 9dcb4d9

Browse files
Michael McNeesclaude
authored and
Michael McNees
committed
docs: add browser automation v0.6.0 design spec (#32, #33)
Covers declarative browser actions (navigate, click, fill, extract, screenshot, wait, evaluate) with stateless session handoff, and custom Playwright images as an independent opt-in track. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 59648c9 commit 9dcb4d9

1 file changed

Lines changed: 274 additions & 0 deletions

File tree

‎docs/superpowers/specs/2026-06-01-browser-automation-v060-design.md‎

Lines changed: 274 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,274 @@
1+
# Browser Automation v0.6.0 Design
2+
3+
**Date:** 2026-06-01
4+
**Issues:** #32 (custom Playwright images), #33 (declarative browser actions)
5+
**Milestone:** v0.6.0
6+
7+
---
8+
9+
## Overview
10+
11+
Two independent tracks ship under the browser automation umbrella:
12+
13+
- **#33 — Declarative browser actions:** Seven new connectors (`browser/navigate`, `browser/click`, `browser/fill`, `browser/extract`, `browser/screenshot`, `browser/wait`, `browser/evaluate`) built on top of the existing `browser/run` connector. Session state is carried between steps as a serializable blob threaded via CEL expressions.
14+
- **#32 — Custom Playwright images:** Opt-in Docker images (`ghcr.io/dvflw/mantle-playwright-node` and `ghcr.io/dvflw/mantle-playwright-python`) that pre-install the Playwright package, eliminating the bootstrap `npm install` / `pip install` step for `browser/run` power users.
15+
16+
The tracks are independent: declarative actions always use the official Microsoft Playwright images. Custom images remain opt-in for `browser/run`.
17+
18+
---
19+
20+
## Track 1: Declarative Browser Actions (#33)
21+
22+
### Architecture
23+
24+
All new code lives in `packages/engine/internal/connector/browser_declarative.go`. The existing `browser.go` and `BrowserRunConnector` are untouched.
25+
26+
The new file introduces:
27+
28+
1. **`BrowserSession` struct** — the serializable handoff type:
29+
```go
30+
type BrowserSession struct {
31+
Cookies []map[string]any `json:"cookies"`
32+
LocalStorage map[string]map[string]string `json:"local_storage"`
33+
URL string `json:"url"`
34+
}
35+
```
36+
37+
2. **`buildDeclarativeScript(actionSnippet string, session *BrowserSession, timeoutMs int) string`** — shared JS generator. Produces a complete, self-contained Playwright script:
38+
- Launch Chromium (headless)
39+
- Create browser context
40+
- If `session` is non-nil: restore cookies and localStorage
41+
- Set `page.setDefaultTimeout(timeoutMs)`
42+
- Navigate to `session.URL` if session is being restored
43+
- Execute the action-specific snippet (injected verbatim)
44+
- Serialize cookies + localStorage + current URL into `session_state`
45+
- `console.log(JSON.stringify({ session_state, data }))` then exit 0
46+
- On any unhandled error: print to stderr, exit 1
47+
48+
3. **Seven connector structs** — each implements `Execute(ctx, params)` by:
49+
- Parsing and validating its own params
50+
- Calling `buildDeclarativeScript` with its action snippet
51+
- Delegating execution to `DockerRunConnector` (same path as `BrowserRunConnector`)
52+
- Parsing stdout JSON and returning `session_state` + action-specific fields
53+
54+
4. **Registration** — all seven connectors registered in `connector.go` alongside `browser/run`:
55+
```
56+
"browser/navigate" → BrowserNavigateConnector
57+
"browser/click" → BrowserClickConnector
58+
"browser/fill" → BrowserFillConnector
59+
"browser/extract" → BrowserExtractConnector
60+
"browser/screenshot" → BrowserScreenshotConnector
61+
"browser/wait" → BrowserWaitConnector
62+
"browser/evaluate" → BrowserEvaluateConnector
63+
```
64+
65+
Language support is JavaScript-only for this iteration. Python support is deferred.
66+
67+
### Session State
68+
69+
```json
70+
{
71+
"cookies": [
72+
{ "name": "session", "value": "abc123", "domain": "example.com", ... }
73+
],
74+
"local_storage": {
75+
"https://example.com": { "token": "xyz", "user_id": "42" }
76+
},
77+
"url": "https://example.com/dashboard"
78+
}
79+
```
80+
81+
When `session_state` input is absent or empty, the script skips restoration and starts a fresh browser context. This makes every connector usable as a standalone step.
82+
83+
### Connector Reference
84+
85+
All connectors share three common optional params:
86+
87+
| Param | Type | Default | Description |
88+
|---|---|---|---|
89+
| `session_state` | object | — | Handoff blob from a prior browser step |
90+
| `timeout_ms` | int | 30000 | Playwright default timeout (ms) |
91+
| `memory` | string | `"1g"` | Container memory limit |
92+
93+
#### `browser/navigate`
94+
95+
| Param | Type | Required | Description |
96+
|---|---|---|---|
97+
| `url` | string | yes | URL to navigate to |
98+
| `wait_until` | string | no | `"load"` (default) \| `"networkidle"` \| `"domcontentloaded"` |
99+
100+
Outputs: `session_state`, `title` (string — page title after navigation)
101+
102+
#### `browser/click`
103+
104+
| Param | Type | Required | Description |
105+
|---|---|---|---|
106+
| `selector` | string | yes | CSS selector of element to click |
107+
| `wait_for` | string | no | CSS selector to wait for after click |
108+
109+
Outputs: `session_state`
110+
111+
#### `browser/fill`
112+
113+
| Param | Type | Required | Description |
114+
|---|---|---|---|
115+
| `fields` | map[string]string | yes | CSS selector → value pairs |
116+
117+
Outputs: `session_state`
118+
119+
#### `browser/extract`
120+
121+
| Param | Type | Required | Description |
122+
|---|---|---|---|
123+
| `selectors` | map[string]string | yes | Name → CSS selector pairs |
124+
| `attribute` | string | no | Extract attribute value instead of text (e.g. `"href"`) |
125+
126+
Outputs: `session_state`, `data` (map[string]string — name → extracted text/attribute)
127+
128+
#### `browser/screenshot`
129+
130+
| Param | Type | Required | Description |
131+
|---|---|---|---|
132+
| `full_page` | bool | no | Capture full scrollable page (default false) |
133+
| `path` | string | no | Filename in output (default `"screenshot.png"`) |
134+
135+
Outputs: `session_state`, `base64` (string — base64-encoded PNG), `path` (string)
136+
137+
#### `browser/wait`
138+
139+
Exactly one of `selector`, `url_pattern`, or `duration_ms` must be provided.
140+
141+
| Param | Type | Description |
142+
|---|---|---|
143+
| `selector` | string | Wait for CSS selector to be visible |
144+
| `url_pattern` | string | Wait for page URL to match glob pattern |
145+
| `duration_ms` | int | Wait unconditionally for N ms (distinct from the shared `timeout_ms` Playwright timeout) |
146+
147+
Outputs: `session_state`
148+
149+
#### `browser/evaluate`
150+
151+
| Param | Type | Required | Description |
152+
|---|---|---|---|
153+
| `expression` | string | yes | JS expression evaluated in page context |
154+
155+
Outputs: `session_state`, `result` (any — return value of the expression)
156+
157+
### Error Handling
158+
159+
**Script contract:** Every script prints exactly one JSON line to stdout on success and exits 0. On failure it writes to stderr and exits non-zero. The connector maps non-zero exits to Go errors with stderr included in the message.
160+
161+
**Selector/navigation timeouts:** Playwright's `TimeoutError` propagates as a non-zero exit. The step fails like any other connector failure — users handle it with `retry` or `continue_on_error` in the workflow YAML.
162+
163+
**Session corruption:** If the `session_state` input is present but unparseable (malformed JSON), the connector returns an error before spawning a container.
164+
165+
### Example Workflow Fragment
166+
167+
```yaml
168+
steps:
169+
- name: go_to_login
170+
action: browser/navigate
171+
params:
172+
url: https://example.com/login
173+
174+
- name: submit_credentials
175+
action: browser/fill
176+
params:
177+
session_state: "{{ steps.go_to_login.outputs.session_state }}"
178+
fields:
179+
"#email": "{{ inputs.email }}"
180+
"#password": "{{ secrets.password }}"
181+
182+
- name: click_submit
183+
action: browser/click
184+
params:
185+
session_state: "{{ steps.submit_credentials.outputs.session_state }}"
186+
selector: "button[type=submit]"
187+
wait_for: ".dashboard"
188+
189+
- name: extract_profile
190+
action: browser/extract
191+
params:
192+
session_state: "{{ steps.click_submit.outputs.session_state }}"
193+
selectors:
194+
username: ".profile-name"
195+
plan: ".subscription-tier"
196+
```
197+
198+
### Testing
199+
200+
**Unit tests** (`browser_declarative_test.go`):
201+
- `buildDeclarativeScript` outputs correct Playwright calls per action
202+
- Session restore code present when `session_state` provided, absent when not
203+
- Malformed params return errors before container spawn
204+
- `browser/wait` rejects invocations where zero or multiple of its exclusive params are set
205+
206+
**Integration tests** (using `testcontainers` + `httptest.NewServer`):
207+
- `navigate` → `extract` session chain: login page, fill form, extract dashboard value
208+
- `fill` + `click` submits a form correctly
209+
- `screenshot` produces non-empty base64
210+
- `evaluate` returns a computed value from page context
211+
- `wait` times out and surfaces error on missing selector
212+
213+
---
214+
215+
## Track 2: Custom Playwright Images (#32)
216+
217+
### Overview
218+
219+
Two Docker images that pre-install the `playwright` npm/pip package on top of the official Microsoft images, eliminating the `npm install --no-save playwright@X.Y.Z` bootstrap step that `browser/run` currently performs on every invocation.
220+
221+
These images are opt-in: users point `browser/run` at them via the `image` param. Declarative actions (#33) continue using the official images.
222+
223+
### Directory Structure
224+
225+
```
226+
packages/playwright/
227+
PLAYWRIGHT_VERSION # single line: "1.52.0"
228+
Dockerfile.node # FROM mcr.microsoft.com/playwright:vX.Y.Z-noble + npm install
229+
Dockerfile.python # FROM mcr.microsoft.com/playwright/python:vX.Y.Z-noble + pip install
230+
```
231+
232+
### Dockerfiles
233+
234+
**Dockerfile.node:**
235+
```dockerfile
236+
ARG PLAYWRIGHT_VERSION
237+
FROM mcr.microsoft.com/playwright:v${PLAYWRIGHT_VERSION}-noble
238+
RUN npm install --global --no-save playwright@${PLAYWRIGHT_VERSION}
239+
```
240+
241+
**Dockerfile.python:**
242+
```dockerfile
243+
ARG PLAYWRIGHT_VERSION
244+
FROM mcr.microsoft.com/playwright/python:v${PLAYWRIGHT_VERSION}-noble
245+
RUN pip install --quiet playwright==${PLAYWRIGHT_VERSION}
246+
```
247+
248+
### Image Tags
249+
250+
- `ghcr.io/dvflw/mantle-playwright-node:<version>` (e.g. `1.52.0`)
251+
- `ghcr.io/dvflw/mantle-playwright-python:<version>`
252+
- `ghcr.io/dvflw/mantle-playwright-node:latest` (only on stable releases, not pre-release)
253+
254+
### CI Workflow
255+
256+
New workflow `build-playwright-images.yml`:
257+
- **Trigger:** push to `main` when `packages/playwright/PLAYWRIGHT_VERSION` changes, or `workflow_dispatch` for manual rebuilds
258+
- **Steps:** checkout → QEMU → buildx → GHCR login → build + push both images (multi-arch: amd64, arm64)
259+
- **Smoke test:** `docker run --rm ghcr.io/dvflw/mantle-playwright-node:<version> node -e "require('playwright'); console.log('ok')"`
260+
261+
### Testing
262+
263+
A `docker build` smoke test in CI verifies both images build successfully. No additional runtime tests — declarative action integration tests cover runtime behavior via official images.
264+
265+
---
266+
267+
## Out of Scope for v0.6.0
268+
269+
- Python language support for declarative actions
270+
- Browser/tab management (multiple pages per session)
271+
- File upload/download via browser actions
272+
- Proxy configuration
273+
- Custom browser launch args
274+
- Authenticated GHCR pulls for private registries in `browser/run`

0 commit comments

Comments
 (0)