Skip to content

Commit ca21b82

Browse files
suchintansuchintan
authored
Sync rustwright-cloud changes to rustwright (#44) (#68)
Skyvern-AI/rustwright-cloud#44 --------- Co-authored-by: suchintan <3853670+suchintan@users.noreply.github.com>
1 parent 1ccb610 commit ca21b82

23 files changed

Lines changed: 2163 additions & 0 deletions

‎benchmarks/form_fill/.gitattributes‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
assets/*.pdf binary

‎benchmarks/form_fill/.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
out/
2+
.venv/
3+
venv/
4+
__pycache__/
5+
*.py[cod]
6+
.env
7+
artifacts/

‎benchmarks/form_fill/Dockerfile.record‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
ARG RUSTWRIGHT_BASE_IMAGE=rustwright-form-fill-base:latest
2+
FROM ${RUSTWRIGHT_BASE_IMAGE}
3+
4+
USER root
5+
6+
RUN apt-get update \
7+
&& apt-get install -y --no-install-recommends \
8+
x11-utils \
9+
xvfb \
10+
&& rm -rf /var/lib/apt/lists/* \
11+
&& python -m pip install --no-cache-dir "matplotlib>=3.9,<4"
12+
13+
# Keep benchmark-only edits on a cheap layer while still inheriting the
14+
# repository image's built Rustwright package and browser installation.
15+
COPY benchmarks/form_fill /workspace/benchmarks/form_fill

‎benchmarks/form_fill/README.md‎

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
# Form-fill benchmark
2+
3+
> **Responsible use is a requirement.** This workload fills a real job
4+
> application with dummy data and never submits it by design. Only target a
5+
> posting you have a legitimate reason and permission to test, respect the
6+
> site's terms of service and rate limits, and never remove or weaken the
7+
> no-submit guardrail.
8+
9+
This demo runs one backend-switchable Python workload through reference
10+
Playwright and Rustwright. It scrolls to each configured field, highlights the
11+
control, enters dummy applicant data, captures a screenshot, validates the
12+
retained value, and writes `timeline.json` plus `timings.json`. The target is
13+
always supplied through the required `BENCH_JOB_URL` environment variable.
14+
15+
The checked-in `field_map.example.json` is a declarative map written for one
16+
specific Greenhouse posting. Its custom question selectors will not be
17+
portable to every posting. Copy and adapt the map for the authorized target
18+
instead of adding a target URL to the workload. Keep submit controls out of the
19+
field list. Controls whose options require live search requests are marked
20+
`network_dependent` and skipped because the actual fill phase runs offline.
21+
Omit that flag only for controls whose options are already available offline.
22+
23+
## Safety guardrail
24+
25+
Before filling anything, the workload finds the configured submit controls,
26+
intercepts both submit events and programmatic form submission before page
27+
scripts run, disables service workers, and installs a browser-context route
28+
that aborts every state-changing HTTP method during startup. Browser-level
29+
WebSocket routing intercepts new sockets without connecting them to a network
30+
peer, while pre-document guards disable page and worker constructors for
31+
WebSocket, WebTransport, EventSource, WebRTC, and dedicated/shared workers. As
32+
soon as the form reaches its ready selector, the
33+
browser context is forced offline for the entire fill and validation phase. It
34+
then disables the visible submit controls.
35+
At the end it verifies that the page did not navigate, no submission was
36+
attempted, all submit controls remain guarded, and no submission-confirmation
37+
text appeared. The workload contains no action that clicks a submit control.
38+
Reports include only the number of network requests blocked by the startup
39+
interlock, never their URLs.
40+
41+
File fixtures are passed to the browser with `set_input_files`. If the site
42+
tries to upload a selected file immediately, the network interlock blocks that
43+
request; the benchmark never waits for or asserts a server-side upload. Some
44+
sites clear their file widget after that blocked request, which is allowed only
45+
for file-field highlight cleanup and is visible in the captured screenshot.
46+
47+
These checks are defense in depth, not permission to target arbitrary sites.
48+
49+
## What is measured
50+
51+
Each Docker run emits two 10 Hz memory series:
52+
53+
- `stack_pss.csv` sums proportional set size (PSS) for the workload Python
54+
process and its descendant driver/browser processes. This avoids
55+
double-counting shared pages and is categorized as the benchmark stack.
56+
- `cgroup.csv` and `cgroup_memory_peak_bytes.txt` cover the whole capped
57+
container, including harness and recording overhead. The kernel peak can
58+
capture spikes between sampled points.
59+
60+
`epochs.json` aligns sampler, workload, and optional ffmpeg timestamps.
61+
`timeline.json` separates launch, browser/network navigation, scripted pauses,
62+
and actions. Rendered charts shade browser-time and scripted-pause bands so
63+
they are not mistaken for library execution time.
64+
65+
Remote CDP runs only measure the local client/driver container. The remote
66+
browser is outside both local PSS and cgroup scope, so remote memory is not an
67+
end-to-end browser-memory comparison.
68+
69+
## Fairness protocol
70+
71+
For an A/B comparison:
72+
73+
1. Build one repository image and derive the recording image from it.
74+
2. Run the exact same `fill_form.py` and field map for both backends.
75+
3. Keep the container memory, swap, CPU, shared-memory cap, viewport, target,
76+
pauses, and run order fixed.
77+
4. Set `BENCH_CHROMIUM_EXECUTABLE` to one executable for both backends. The
78+
Docker harness defaults both to the Rustwright image's Chromium symlink.
79+
5. Run variants sequentially, repeat enough times, and report failures as
80+
failures rather than dropping them from the sample.
81+
6. Treat local results as diagnostics. Per the repository's
82+
[`BENCHMARK.md`](../../BENCHMARK.md), durable claims must be reproduced in
83+
capped, sharded Docker workloads on the Testbox path with provenance.
84+
85+
Recording adds Xvfb and ffmpeg overhead. Compare recorded runs with recorded
86+
runs, and non-recorded runs with non-recorded runs.
87+
88+
## Build the Docker images
89+
90+
From the repository root:
91+
92+
```bash
93+
benchmarks/form_fill/harness/build_images.sh
94+
```
95+
96+
The first build uses the repository's root `Dockerfile` and tags its result as
97+
`rustwright-form-fill-base:latest`. `Dockerfile.record` then uses that image as
98+
its `FROM` base and adds Xvfb plus the plotting dependency. Override the tags
99+
with `FORM_FILL_BASE_IMAGE` and `FORM_FILL_RECORD_IMAGE` if needed.
100+
101+
## Run the capped local Docker comparison
102+
103+
```bash
104+
export BENCH_JOB_URL="https://job-board.example/jobs/authorized-test-target"
105+
benchmarks/form_fill/harness/run_pair.sh
106+
```
107+
108+
Artifacts are written under the ignored `benchmarks/form_fill/out/` directory.
109+
Defaults are an 8 GiB memory/swap cap, 4 CPUs, and 1 GiB shared memory; use
110+
`BENCH_MEMORY_LIMIT`, `BENCH_CPUS`, and `BENCH_SHM_SIZE` to change them for all
111+
variants. Use `BENCH_FIELD_CONFIG_HOST=/path/to/field-map.json` for an adapted
112+
map. `BENCH_PAUSE_SCALE=0` is useful for a quick smoke test, but must be held
113+
constant across comparisons.
114+
115+
To run one variant:
116+
117+
```bash
118+
benchmarks/form_fill/harness/run_one.sh rustwright rustwright-smoke
119+
```
120+
121+
## Record and render
122+
123+
```bash
124+
benchmarks/form_fill/harness/record_one.sh playwright playwright-record
125+
benchmarks/form_fill/harness/record_one.sh rustwright rustwright-record
126+
benchmarks/form_fill/harness/render.sh
127+
```
128+
129+
The renderer writes two synchronized animated PSS videos, a PSS/cgroup
130+
comparison chart, and demo-grade statistics under `out/rendered/`. Videos,
131+
screenshots, CSVs, logs, and generated reports are intentionally ignored.
132+
133+
## Run directly on the host
134+
135+
Create an isolated environment, install this checkout and the reference
136+
Playwright package, install a compatible Chromium, then run:
137+
138+
```bash
139+
python -m venv benchmarks/form_fill/.venv
140+
source benchmarks/form_fill/.venv/bin/activate
141+
python -m pip install -e . "playwright==1.59.0"
142+
python -m playwright install chromium
143+
144+
BACKEND=rustwright \
145+
BENCH_JOB_URL="https://job-board.example/jobs/authorized-test-target" \
146+
python benchmarks/form_fill/fill_form.py
147+
148+
BACKEND=playwright \
149+
BENCH_JOB_URL="https://job-board.example/jobs/authorized-test-target" \
150+
python benchmarks/form_fill/fill_form.py
151+
```
152+
153+
Set `BENCH_CHROMIUM_EXECUTABLE` to the same browser binary for both commands.
154+
Host runs are convenient for development but are not durable benchmark
155+
evidence.
156+
157+
## Connect to a remote browser over CDP
158+
159+
Any provider-neutral Chrome DevTools Protocol WebSocket URL works; the code
160+
sends no provider-specific headers and makes no provider API calls:
161+
162+
```bash
163+
export BENCH_JOB_URL="https://job-board.example/jobs/authorized-test-target"
164+
export CDP_URL="wss://cdp-provider.example/session"
165+
benchmarks/form_fill/harness/run_remote.sh rustwright rustwright-remote
166+
```
167+
168+
For a direct host run, `fill_form_remote.py` is an explicit wrapper around the
169+
same workload and requires both `BENCH_JOB_URL` and `CDP_URL`.
170+
171+
The endpoint must permit creation of a dedicated browser context. The workload
172+
fails closed if it cannot create one; it never reuses, takes offline, or closes
173+
a provider-owned context.
174+
175+
Remote uploads are skipped by default because provider file-transfer behavior
176+
varies. Set `BENCH_SKIP_UPLOADS=0` when the endpoint supports local file upload.
177+
A Skyvern browser session is one public way to obtain a CDP URL, but session
178+
creation and credentials are deliberately outside this benchmark.

‎benchmarks/form_fill/RESULTS.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# Form-fill benchmark — recorded demo results
2+
3+
Demo-grade results from single run pairs of this harness, recorded 2026-07-14.
4+
Per [`BENCHMARK.md`](../../BENCHMARK.md), these are **illustrative demo numbers,
5+
not durable benchmark claims** — durable claims should come from capped,
6+
repeated testbox runs. They are recorded here so that published figures (README
7+
media, demo GIFs) have a citable, reproducible source. Raw data:
8+
[`results/`](results/).
9+
10+
## Protocol
11+
12+
- One Python script, byte-identical for both backends; only the import differs
13+
(`BACKEND=playwright|rustwright`). See [`fill_form.py`](fill_form.py).
14+
- Workload: a public Greenhouse job-application form (22 fields: text, combobox,
15+
EEOC dropdowns, resume + cover-letter PDF uploads), filled with dummy data at
16+
a 400×600 viewport with per-field highlight/pan choreography. Never submitted
17+
(hard guardrail).
18+
- Same Chromium 1217 binary for both backends. Reference Playwright 1.59.0
19+
(pinned). Rustwright: a 0.1.0-alpha development build baked into a local
20+
Docker image (`rustwright-verify`, image ID `9123a56066a7`, built
21+
2026-06-13; recording derivative `rustwright-bench-record`, image ID
22+
`dabfd27d62a9`). The exact source commit of that build was not recorded —
23+
a provenance gap that is part of why these numbers are demo-grade. The
24+
build predates the Chromium launch-flag alignment and `Locator.fill`
25+
changes now on `main`; re-running against a current build is the
26+
recommended way to obtain citable numbers.
27+
- Containers: one per backend, sequential, `--memory=8g --memory-swap=8g
28+
--cpus=4`, headed under Xvfb with ffmpeg screen capture.
29+
- Memory sampled at 10 Hz: cgroup v2 plus per-process PSS with
30+
python/driver/chromium attribution (`harness/sample_stack_memory.py`).
31+
- Scripted demo pauses (~11.7 s per run) are identical constants in both runs
32+
and are excluded from "actions" time.
33+
34+
## Local recorded pair (`results/stats_local_recorded.json`)
35+
36+
| Metric | Playwright | Rustwright | Δ |
37+
|---|---:|---:|---:|
38+
| Wall time | 22.53 s | 18.72 s | −16.9% |
39+
| Actions (library-controlled) | 8.59 s | 5.95 s | −30.8% |
40+
| Browser launch | 1.24 s | 0.30 s | −75.9% |
41+
| Tool-stack peak memory (PSS: python + driver + chromium) | 662.5 MiB | 646.5 MiB | −2.4% |
42+
| Client-library share at stack peak (PSS: python + driver) | 130.0 MiB | 37.8 MiB | −71.0% |
43+
| …of which driver (Node) | 102.3 MiB | 0 | — |
44+
45+
Client-library values are the python + driver components at the stack-peak
46+
sample in `results/stats_local_recorded.json`. Together with the remote
47+
pair below (133.5 vs 40.6 MiB, −69.6%, measured directly), they are the
48+
source of the "~71% less client memory" figure used in demo media. The
49+
full-stack numbers are close because both backends drive the same
50+
Chromium; rustwright's chromium tree measured heavier in this pair due to
51+
launch-flag differences since aligned with Playwright's defaults.
52+
53+
## Remote CDP pair (`results/stats_remote_cdp.json`)
54+
55+
Same workload via `connect_over_cdp` to a fresh remote browser session per run
56+
(WAN), so container memory contains only the client stack. File-upload fields
57+
were skipped in both runs (remote `DOM.setFileInputFiles` requires
58+
browser-host paths); 20 fields filled per run.
59+
60+
| Metric | Playwright | Rustwright | Δ |
61+
|---|---:|---:|---:|
62+
| Wall time | 117.2 s | 96.3 s | −17.8% |
63+
| Actions | 102.4 s | 82.0 s | −19.9% |
64+
| Client memory peak (PSS) | 133.5 MiB | 40.6 MiB | −69.6% |
65+
| Connect | 1.01 s | 1.24 s | +22% |
66+
67+
These are single-pair observations over a WAN and network conditions were
68+
not controlled; they should not be read as a general protocol-efficiency
69+
result. Repeated runs under controlled latency would be needed to
70+
establish that.
71+
72+
## Reproduce
73+
74+
```bash
75+
# a Greenhouse-style posting you are authorized to test against:
76+
export BENCH_JOB_URL="https://job-board.example/jobs/authorized-test-target"
77+
78+
# local recorded pair (docker, headed under Xvfb):
79+
benchmarks/form_fill/harness/record_one.sh playwright playwright-record
80+
benchmarks/form_fill/harness/record_one.sh rustwright rustwright-record
81+
82+
# remote pair (also requires CDP_URL per run, see README "Remote mode"):
83+
benchmarks/form_fill/harness/run_remote.sh playwright playwright-remote
84+
benchmarks/form_fill/harness/run_remote.sh rustwright rustwright-remote
85+
```
86+
87+
The recorded figures above came from a Greenhouse posting with 22 fields;
88+
results depend on the chosen posting's field mix (see
89+
[`field_map.example.json`](field_map.example.json)).
90+
91+
See [`README.md`](README.md) for prerequisites and the responsible-use note.

‎benchmarks/form_fill/assets/cover-letter.pdf‎

631 Bytes
Binary file not shown.

‎benchmarks/form_fill/assets/resume.pdf‎

625 Bytes
Binary file not shown.

0 commit comments

Comments
 (0)