Skip to content

Commit 528eff0

Browse files
andreakirokalil0321lucas-nottergarcialeo-notte
committed
initial commit
Co-Authored-By: kalil0321 <kalil0321@users.noreply.github.com> Co-Authored-By: Lucas Giordano <lucas@notte.cc> Co-Authored-By: Rafael <raf@kernel.sh> Co-Authored-By: Leo <leo@notte.cc> Co-Authored-By: junhsss <junhsssr@gmail.com>
0 parents  commit 528eff0

199 files changed

Lines changed: 165601 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env.example‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Copy to `.env` and set keys only for the providers you run (`npm run bench -- --provider=...`).
2+
# See README.md for flags and defaults.
3+
4+
# Browserbase
5+
BROWSERBASE_API_KEY=
6+
BROWSERBASE_PROJECT_ID=
7+
8+
# Steel
9+
STEEL_API_KEY=
10+
11+
# Kernel
12+
KERNEL_API_KEY=
13+
14+
# Notte
15+
NOTTE_API_KEY=
16+
17+
# Hyperbrowser
18+
HYPERBROWSER_API_KEY=
19+
20+
# Anchor Browser
21+
ANCHORBROWSER_API_KEY=
22+
23+
# Browser Use (cloud)
24+
BROWSER_USE_API_KEY=
25+
26+
# --- Optional CLI defaults (same as flags on `npm run bench` / `npm run dev`) ---
27+
# BENCHMARK=hello-browser
28+
# PROVIDER=browserbase
29+
# RUNS=1000
30+
# URL=https://google.com/
31+
# OUTPUT=
32+
# RATE=
33+
# CONCURRENCY=1

‎.github/workflows/ci.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
name: CI
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches:
7+
- main
8+
9+
permissions:
10+
contents: read
11+
12+
jobs:
13+
root:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
17+
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
18+
with:
19+
node-version: 20
20+
cache: npm
21+
cache-dependency-path: package-lock.json
22+
- run: npm ci
23+
- run: npm run build
24+
- run: npm audit --package-lock-only
25+
26+
web:
27+
runs-on: ubuntu-latest
28+
steps:
29+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
30+
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
31+
with:
32+
node-version: 20
33+
cache: npm
34+
cache-dependency-path: web/package-lock.json
35+
- run: npm ci
36+
working-directory: web
37+
- run: npm run build
38+
working-directory: web
39+
- run: npm audit --package-lock-only
40+
working-directory: web

‎.github/workflows/secrets.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
name: Secret Scan
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches:
7+
- main
8+
9+
permissions:
10+
contents: read
11+
12+
jobs:
13+
scan:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
17+
- name: Scan tracked files for common secrets
18+
run: |
19+
set -euo pipefail
20+
21+
pattern='(ghp_[A-Za-z0-9]{36}|github_pat_[A-Za-z0-9_]{20,}|AKIA[0-9A-Z]{16}|-----BEGIN (RSA|EC|OPENSSH|DSA|PGP) PRIVATE KEY-----|xox[baprs]-[A-Za-z0-9-]{10,}|sk_live_[A-Za-z0-9]{16,}|sk-[A-Za-z0-9]{48,}|(OPENAI_API_KEY|ANTHROPIC_API_KEY)[[:space:]]*[:=][[:space:]]*[^[:space:]]+)'
22+
23+
if git ls-files -z | xargs -0 grep -nIE "$pattern"; then
24+
echo "Potential secret material detected"
25+
exit 1
26+
fi
27+
28+
echo "No known secret patterns detected"

‎.gitignore‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
node_modules/
2+
dist/
3+
!results/.gitkeep
4+
recordings/
5+
!web-app/web/public/recordings/
6+
.env
7+
*.local
8+
*.db
9+
*.css
10+
!web-app/**/*.css
11+
.vercel
12+
.next/
13+
TODO.md

‎Dockerfile‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
FROM node:20-slim
2+
3+
WORKDIR /app
4+
5+
# Install dependencies
6+
COPY package.json package-lock.json ./
7+
RUN npm ci --omit=dev && npm install typescript
8+
9+
# Copy source and build
10+
COPY tsconfig.json .env.example ./
11+
COPY src/ src/
12+
COPY queries/ queries/
13+
RUN npx tsc -p tsconfig.json && npm uninstall typescript
14+
15+
# Default: run hello-browser benchmark with 100 runs
16+
ENV BENCHMARK=hello-browser
17+
ENV RUNS=100
18+
ENV PROVIDER=""
19+
ENV CONCURRENCY=1
20+
21+
CMD node dist/cli.js \
22+
--benchmark=$BENCHMARK \
23+
--provider=$PROVIDER \
24+
--runs=$RUNS \
25+
--concurrency=$CONCURRENCY

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Browser Arena contributors
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
# Browser Arena
2+
3+
Open benchmarks for cloud browser providers. Two benchmarks test different aspects of browser performance:
4+
5+
- **Hello Browser** — minimal session lifecycle (create, connect, navigate, release)
6+
- **v0** — realistic workflow (extract data, crawl pages, fill forms)
7+
8+
Providers: **Browserbase**, **Steel**, **Kernel**, **Notte**, **Hyperbrowser**, **Anchor Browser**, **Browser Use**.
9+
10+
## Benchmarks
11+
12+
### Hello Browser
13+
14+
Measures the minimal end-to-end lifecycle for a remote Chrome session:
15+
16+
1. **Create** a session (control plane)
17+
2. **Connect** Playwright over CDP (data plane)
18+
3. **Navigate** to a URL (wait for `domcontentloaded`)
19+
4. **Release** the session (control plane)
20+
21+
Supports concurrency testing (`--concurrency=16`) to measure provider performance under parallel load.
22+
23+
### v0
24+
25+
Simulates a realistic workflow across 16 Wikipedia pages and a form submission:
26+
27+
1. **Extract** — Navigate to a seed article, scroll to trigger lazy content, extract structured data (title, summary, infobox, headings, links, word count, image count)
28+
2. **Crawl** — Two-level deep crawl: follow 5 links from the seed, then 2 links from each of those (15 additional pages), scrolling and extracting on every page
29+
3. **Form** — Navigate to httpbin, fill a form with the collected research data, submit, and verify the response
30+
31+
Data flows between phases: links extracted in Phase 1 drive Phase 2 navigation, and all extracted data feeds into Phase 3's form fill.
32+
33+
## Methodology
34+
35+
### Sample size
36+
37+
- **Hello Browser**: 1,000 runs per provider (sequential), 100 batches of 16 (concurrent)
38+
- **v0**: configurable, default 1,000
39+
40+
### Benchmark runner & RTT
41+
42+
Two AWS EC2 VMs (t3.micro) run the benchmarks. TCP+TLS RTT measured with `curl time_appconnect`, median of 10.
43+
44+
| Provider | CDP host | RTT | Runner region |
45+
|---|---|---|---|
46+
| Notte | `api.notte.cc` | 12ms | us-west-1 |
47+
| Steel | `connect.steel.dev` | 14ms | us-east-1 |
48+
| Kernel (Headless) | `api.onkernel.com` | 14ms | us-east-1 |
49+
| Kernel (Headful) | `api.onkernel.com` | 14ms | us-east-1 |
50+
| Hyperbrowser | `connect-us-east-1.hyperbrowser.ai` | 9ms | us-east-1 |
51+
| Browser Use | `cdp1.browser-use.com` | 29ms | us-east-1 |
52+
| Anchor Browser | `connect.anchorbrowser.io` | 38ms | us-east-1 |
53+
| Browserbase | `connect.usw2.browserbase.com` | 62ms | us-west-1 |
54+
55+
### Fairness
56+
57+
- **Warm-up runs** before measurement (10 for hello-browser, 3 for v0)
58+
- **Same URL** across all providers (`google.com` for hello-browser, Wikipedia AI article for v0)
59+
- **No provider-specific tuning** — default settings only
60+
- **Rate limit handling**: automatic 30s backoff on 429 errors
61+
62+
Note: `create` and `release` timings reflect API design choices (sync vs async cleanup) more than browser performance. The `connect` + `goto` timings best represent actual browser speed.
63+
64+
### Value Score
65+
66+
Each provider gets a single score from 0 to 1 combining three dimensions: **reliability**, **latency**, and **cost**. By default all three are weighted equally (33/33/33). Visitors can shift the balance with presets like "Speed first" or "Budget first."
67+
68+
Each dimension is normalized to a 0–1 scale using fixed anchors that define "perfect" (1.0) and "unacceptable" (0.0):
69+
70+
| Dimension | 0.0 (unacceptable) | 1.0 (perfect) | Why |
71+
|---|---|---|---|
72+
| Reliability | 90% | 100% | Below 90% is unusable in production |
73+
| Latency | 10,000 ms | 0 ms | 10s is the practical timeout threshold |
74+
| Cost | $0.20/hr | $0.00/hr | ~2× the most expensive current provider |
75+
76+
The final score is: `score = w_latency × norm_latency + w_reliability × norm_reliability + w_cost × norm_cost`
77+
78+
#### Why fixed anchors instead of relative normalization?
79+
80+
Two common alternatives have problems:
81+
82+
- **Ratio-to-best** (`best / yours`) produces different effective scales per dimension. When latency has a 6.8× spread but cost only 2.4×, cost silently dominates even at "equal" weights.
83+
- **Min-max on observed data** (`(yours - worst) / (best - worst)`) zeroes out the worst provider in each dimension. A provider at 98.3% reliability maps to 0.00 if everyone else is at 100% — disproportionate to a 1.7% gap.
84+
85+
Fixed anchors avoid both issues: the scales are genuinely comparable, no one gets zeroed out for small gaps, and adding or removing a provider doesn't shift anyone else's score.
86+
87+
## Quickstart
88+
89+
### Requirements
90+
91+
- Node.js >= 18
92+
93+
### Install
94+
95+
```bash
96+
git clone https://github.com/nottelabs/browserarena
97+
cd browserarena
98+
npm install
99+
```
100+
101+
### Configure
102+
103+
```bash
104+
cp .env.example .env
105+
```
106+
107+
Add API keys for the providers you want to test:
108+
109+
| Variable | Provider |
110+
|---|---|
111+
| `BROWSERBASE_API_KEY`, `BROWSERBASE_PROJECT_ID` | Browserbase |
112+
| `STEEL_API_KEY` | Steel |
113+
| `KERNEL_API_KEY` | Kernel |
114+
| `NOTTE_API_KEY` | Notte |
115+
| `HYPERBROWSER_API_KEY` | Hyperbrowser |
116+
| `ANCHORBROWSER_API_KEY` | Anchor Browser |
117+
| `BROWSER_USE_API_KEY` | Browser Use |
118+
119+
### Run
120+
121+
```bash
122+
# Hello Browser — single provider, 100 runs
123+
npm run bench -- --provider=notte --runs=100
124+
125+
# Hello Browser — concurrent (16 parallel sessions)
126+
npm run bench -- --provider=notte --concurrency=16 --runs=100
127+
128+
# v0 — data extraction and filling worflow benchmark
129+
npm run bench -- --benchmark=v0 --provider=notte --runs=10
130+
131+
# All providers, default settings
132+
npm run bench
133+
```
134+
135+
### View results
136+
137+
Requires [DuckDB](https://duckdb.org/docs/installation/).
138+
139+
```bash
140+
# Quick summary (median per provider, sorted by total latency)
141+
duckdb -c ".read queries/hello-browser/simple.sql"
142+
143+
# Full breakdown (median, p90, p95, min, max per stage)
144+
duckdb -c ".read queries/hello-browser/full.sql"
145+
```
146+
147+
## CLI options
148+
149+
| Flag | Default | Description |
150+
|---|---|---|
151+
| `--provider` | all providers | Comma-separated names |
152+
| `--benchmark` | `hello-browser` | `hello-browser` or `v0` |
153+
| `--runs` | `1000` | Measured iterations per provider |
154+
| `--concurrency` | `1` | Parallel sessions per iteration |
155+
| `--url` | auto | Target URL (hello-browser: `google.com`, v0: Wikipedia AI article) |
156+
| `--rate` | unlimited | Max sessions per minute |
157+
| `--out` | auto | Output path |
158+
159+
All flags can also be set via environment variables (`BENCHMARK`, `PROVIDER`, `RUNS`, `CONCURRENCY`, `URL`, `OUTPUT`, `RATE`).
160+
161+
## What gets measured
162+
163+
### Hello Browser
164+
165+
| Field | Description |
166+
|---|---|
167+
| `session_creation_ms` | Control plane latency to create a session |
168+
| `session_connect_ms` | Playwright `connectOverCDP` handshake |
169+
| `page_goto_ms` | `page.goto(url, { waitUntil: "domcontentloaded" })` |
170+
| `session_release_ms` | Control plane latency to release the session |
171+
| `concurrency` | Number of parallel sessions in this batch |
172+
173+
Output: `results/hello-browser/{provider}/{date}/c{N}/results.jsonl`
174+
175+
### v0
176+
177+
| Field | Description |
178+
|---|---|
179+
| `session_creation_ms` | Control plane latency to create a session |
180+
| `session_connect_ms` | Playwright `connectOverCDP` handshake |
181+
| `extract_ms` | Phase 1: navigate + scroll + extract structured data |
182+
| `crawl_ms` | Phase 2: two-level crawl across 15 pages |
183+
| `form_ms` | Phase 3: form fill + submit + verify |
184+
| `total_agent_ms` | Total time for all three phases |
185+
| `session_release_ms` | Control plane latency to release the session |
186+
| `cost_usd` | Estimated cost for this session |
187+
188+
Output: `results/v0/{provider}/{date}/results.jsonl`
189+
190+
## Deploy
191+
192+
[![Deploy on Railway](https://railway.com/button.svg)](https://railway.com/deploy/UNedGj?referralCode=YUwE3Q&utm_medium=integration&utm_source=template&utm_campaign=generic)
193+
194+
Set your provider API keys as environment variables in the Railway dashboard.
195+
196+
## License
197+
198+
MIT

‎SECURITY.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Security Policy
2+
3+
## Reporting a vulnerability
4+
5+
Please do not open public GitHub issues for suspected security vulnerabilities.
6+
7+
- Email: `junhsssr@gmail.com`
8+
- Subject line: `browserarena security report`
9+
- Include reproduction steps, affected files or endpoints, impact, and any suggested remediation.
10+
11+
If GitHub private vulnerability reporting is enabled for this repository, you may use that instead.
12+
13+
## What to report
14+
15+
Please report issues such as:
16+
17+
- exposed credentials, tokens, or private keys
18+
- workflow or CI/CD permission problems
19+
- vulnerabilities in the public `web/` app or API routes
20+
- benchmark artifacts exposing non-public data unexpectedly
21+
- dependency vulnerabilities with practical impact on this repository
22+
23+
## Response expectations
24+
25+
- Initial triage response: within 5 business days
26+
- Status update after triage: within 10 business days
27+
- Fix timing depends on severity, exploitability, and release coordination
28+
29+
We appreciate responsible disclosure and will work to acknowledge and remediate valid reports promptly.

0 commit comments

Comments
 (0)