Feat/104 support browserbase

.gitignore

@@ -1,4 +1,6 @@
 models/models.yaml
+.env.local
+.clawbench-secrets/
 test-output-*/
 test-output/
 **/__pycache__/
@@ -16,4 +18,4 @@ build/
 .pytest_cache/
 run-logs/
 harbor-datasets/
-harbor-jobs/
+harbor-jobs/

CHANGELOG.md

@@ -6,6 +6,16 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](https://semver.org/).
 
+## [Unreleased]
+
+### Added
+
+- Supported Browserbase as a remote browser runtime for less resource consumption and better scalability.
+
+### Fixed
+
+- Fixed Hermes startup with custom OpenAI-compatible model endpoints.
+
 ## [0.8.0] - 2026-08-01
 ### Added
 - Added support for remote browsers with CDP connection.

README.md

@@ -365,11 +365,31 @@ Once the container starts, the script prints a **noVNC URL** (e.g. `http://local
 
 Results land in `./test-output/<model>/<harness>-<case>-<model>-<timestamp>/` with the full five-layer recording. The default harness is `openclaw`; pass `--harness opencode` to use [opencode](https://opencode.ai), `--harness claude-code` to use [Claude Code](https://docs.anthropic.com/en/docs/claude-code), `--harness claude-code-chrome-extension` to use Claude Code + the [Claude in Chrome](https://code.claude.com/docs/en/chrome) extension (Microsoft Edge + local bridge, bypass stack so any LiteLLM-routed provider works), `--harness codex` to use [OpenAI Codex CLI](https://github.com/openai/codex), `--harness claw-code` to use [claw-code](https://github.com/ultraworkers/claw-code), `--harness browser-use` to use [browser-use](https://github.com/browser-use/browser-use) (Python framework, routed via LiteLLM), `--harness hermes` to use [Hermes Agent](https://github.com/NousResearch/hermes-agent) with native browser tools attached to ClawBench Chrome via CDP, or `--harness pi` to use [Pi](https://pi.dev/) with pinned [pi-browser-harness](https://pi.dev/packages/pi-browser-harness) browser tools attached to the same ClawBench Chrome CDP endpoint.
 
+To use a managed Browserbase browser, put the key in `.env.local` and select the
+runtime on a single or batch run:
+
+```dotenv
+BROWSERBASE_API_KEY=bb_...
+```
+
+```bash
+uv run clawbench-run test-cases/v1/<case> your-model \
+  --browser-runtime browserbase
+
+uv run clawbench-batch --models your-model --all-cases \
+  --browser-runtime browserbase
+```
+
+Browserbase runs reuse the same CDP action capture, screenshots, HTTP logging,
+and request interception as local runs. The provider records the video; the
+Browserbase Session Inspector URL is stored as `browser_runtime.recording_url`
+in `run-meta.json`, so a local `recording.mp4` is not required. Provider options such as region or proxy can be passed as JSON, for example `--browser-runtime-options '{"region":"us-west-2","proxies":true}'`.
+
 **(c) Evaluate a model across a whole corpus** — one command runs every task in a suite:
 ```bash
 clawbench-batch --models your-model --cases-suite v2 --all-cases
 ```
-`your-model` is a key you configured in step 1; `--cases-suite v2` runs the full V2 corpus (swap in `v1-lite` for the 20-task subset). Add `--max-concurrent N` to run tasks in parallel (default 2) and `--harness <name>` to pick an agent (default `openclaw`). Each task is intercepted and scored by the `deepseek-v4-pro` judge you set up in step 1 — pass `--no-judge` to skip scoring. A `batch-summary.json` plus per-run recordings land under `./test-output/`. From a source checkout, prefix the command with `uv run`. See [Reproduce the leaderboard](#-reproduce-the-leaderboard) for the end-to-end scoring workflow.
+`your-model` is a key you configured in step 1; `--cases-suite v2` runs the full V2 corpus (swap in `v1-lite` for the 20-task subset). Add `--max-concurrent N` to run tasks in parallel (default 2 locally and 1 with Browserbase) and `--harness <name>` to pick an agent (default `openclaw`). Each task is intercepted and scored by the `deepseek-v4-pro` judge you set up in step 1 — pass `--no-judge` to skip scoring. A `batch-summary.json` plus per-run recordings land under `./test-output/`. From a source checkout, prefix the command with `uv run`. See [Reproduce the leaderboard](#-reproduce-the-leaderboard) for the end-to-end scoring workflow.
 
 **(d) Drive the browser yourself via noVNC** — produces a human reference run:
 ```bash
@@ -741,6 +761,9 @@ ClawBench's niche: **live consumer websites, everyday tasks, end-to-end recordin
 # Single run:
 uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4-6
 
+# Single run with a Browserbase browser (key from .env.local):
+uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4-6 --browser-runtime browserbase
+
 # Human mode (you control the browser via noVNC):
 uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats --human
 
@@ -803,8 +826,8 @@ Each session records five layers of synchronized data under `/data/`:
 
 | Layer              | File                   | Description                                                     |
 | ------------------ | ---------------------- | --------------------------------------------------------------- |
-| Session replay     | `recording.mp4`        | Full session video (H.264, 15fps)                               |
-| Action screenshots | `screenshots/*.png`    | Timestamped PNG per browser action                              |
+| Session replay     | `recording.mp4` or `run-meta.json` recording URL | Local H.264 video or Browserbase Session Inspector replay |
+| Action screenshots | `screenshots/*.png`    | Throttled timestamped PNGs captured after browser actions       |
 | Browser actions    | `actions.jsonl`        | Every DOM event (click, keydown, input, pageLoad, scroll, etc.) |
 | HTTP traffic       | `requests.jsonl`       | Every HTTP request with headers, body, and query params         |
 | Agent messages     | `agent-messages.jsonl` | Full agent conversation transcript (thinking, text, tool calls) |

docs/README.zh-CN.md

@@ -624,6 +624,9 @@ ClawBench 定位:**真实消费级网站、日常任务、端到端录制**。
 # 单次运行:
 uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4-6
 
+# 使用 Browserbase 云浏览器进行单次运行（密钥读取自 .env.local）:
+uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4-6 --browser-runtime browserbase
+
 # 人工模式 (通过 noVNC 控制浏览器):
 uv run clawbench-run test-cases/v1/001-daily-life-food-uber-eats --human
 
@@ -652,6 +655,19 @@ uvx --from harbor==0.15.0 harbor run -p ./harbor-datasets/clawbench-v2 -a "<agen
 #     -a hermes -m deepseek/deepseek-v4-flash
 ```
 
+使用 Browserbase 前，请把密钥放在不会提交到 Git 的 `.env.local` 中：
+
+```dotenv
+BROWSERBASE_API_KEY=bb_...
+```
+
+Browserbase 运行会复用本地运行相同的 CDP 动作捕获、动作截图、HTTP
+日志和请求拦截逻辑。视频由 Browserbase 保存；Session Inspector 地址写入
+`run-meta.json` 的 `browser_runtime.recording_url`，因此不会生成本地
+`recording.mp4`。批量运行默认并发数为 1；可用
+`--browser-runtime-options '{"region":"us-west-2","proxies":true}'`
+传入区域或代理等选项。
+
 V1 任务位于 [`test-cases/v1/`](../test-cases/v1/)（153 个任务）。V2 任务位于 `test-cases/v2/`（130 个任务），Lite 位于 `test-cases/v1-lite/`（20 个任务）。所有 suite 都使用 [`test-cases/task.schema.json`](../test-cases/task.schema.json)。测试用例编写细节见 [CONTRIBUTING.md](../CONTRIBUTING.md)；输出结构与评测流程见 [eval/README.md](../eval/README.md)。
 
 <br/>
@@ -683,8 +699,8 @@ V1 任务位于 [`test-cases/v1/`](../test-cases/v1/)（153 个任务）。V2 
 
 | 层 | 文件 | 描述 |
 |-------|------|-------------|
-| 会话回放 | `recording.mp4` | 完整的会话视频 (H.264, 15fps) |
-| 动作截图 | `screenshots/*.png` | 每个浏览器动作的带时间戳 PNG |
+| 会话回放 | `recording.mp4` 或 `run-meta.json` 中的录制 URL | 本地 H.264 视频，或 Browserbase Session Inspector 回放 |
+| 动作截图 | `screenshots/*.png` | 浏览器动作后经限流捕获的带时间戳 PNG |
 | 浏览器动作 | `actions.jsonl` | 每个 DOM 事件 (click, keydown, input, pageLoad, scroll 等) |
 | HTTP 流量 | `requests.jsonl` | 每个 HTTP 请求,包含 headers、body 和查询参数 |
 | 智能体消息 | `agent-messages.jsonl` | 完整的智能体对话记录 (思考、文本、工具调用) |

src/README.md

@@ -196,12 +196,22 @@ A stagger delay is applied between job starts since during container startup it
 | `--cases-dir PATH`        | Custom case directory                                     | none                                            |
 | `--all-cases`             | Use all task directories in the selected suite/dir        | false                                            |
 | `--case-range START-END`  | Filter by numeric case ID prefix                          | none                                             |
-| `--max-concurrent N`      | Max parallel jobs; a recommended value is 1/3 - 1/2 n CPU | 2                                                |
+| `--max-concurrent N`      | Max parallel jobs; Browserbase defaults to 1               | 2 locally, 1 with Browserbase                    |
 | `--output-dir PATH`       | Base output directory                                     | `test-output`                                    |
 | `--stagger-delay SECONDS` | Minimum gap between consecutive container starts          | 15                                               |
 | `--dry-run`               | Print job matrix without running                          | false                                            |
 | `--no-upload`             | Skip HuggingFace upload for all runs                      | false                                            |
 | `--harness NAME`          | Harness image to use                                      | `openclaw`                                       |
+| `--browser-runtime NAME`  | Browser runtime (`local`, `remote-cdp`, or `browserbase`)  | `local`                                          |
+| `--browser-cdp-url URL`   | CDP endpoint used with `--browser-runtime remote-cdp`      | none                                             |
+| `--browser-runtime-options JSON` | Provider options such as Browserbase region or proxies | none                                          |
+
+Browserbase reads `BROWSERBASE_API_KEY` from `.env.local` or the process
+environment. Its provider-hosted Session Inspector recording URL is stored in
+`browser_runtime.recording_url` in `run-meta.json`; local MP4 recording is
+thus omitted to save space and bandwidth. The signed provider CDP URL is mounted into the runtime
+container through a temporary read-only secret file and redacted from saved
+metadata.
 
 Signal handling:
 

src/clawbench/runner/batch.py

@@ -242,6 +242,9 @@ async def run_job(
     batch_start: float,
     no_upload: bool = False,
     harness: str | None = None,
+    browser_runtime: str | None = None,
+    browser_cdp_url: str | None = None,
+    browser_runtime_options: str | None = None,
     judge: str | None = None,
     no_judge: bool = False,
 ) -> None:
@@ -284,6 +287,15 @@ async def run_job(
                     cmd_parts.append("--no-upload")
                 if harness:
                     cmd_parts += ["--harness", harness]
+                if browser_runtime:
+                    cmd_parts += ["--browser-runtime", browser_runtime]
+                if browser_cdp_url:
+                    cmd_parts += ["--browser-cdp-url", browser_cdp_url]
+                if browser_runtime_options:
+                    cmd_parts += [
+                        "--browser-runtime-options",
+                        browser_runtime_options,
+                    ]
                 if no_judge:
                     cmd_parts.append("--no-judge")
                 elif judge:
@@ -370,7 +382,12 @@ def print_progress(jobs: list[Job], start: float) -> None:
 # ---------------------------------------------------------------------------
 
 
-def print_summary(jobs: list[Job], elapsed: float, max_concurrent: int) -> None:
+def print_summary(
+    jobs: list[Job],
+    elapsed: float,
+    max_concurrent: int,
+    browser_runtime: str = "local",
+) -> None:
     print(f"\n{'=' * 60}")
     print("BATCH SUMMARY")
     print(f"{'=' * 60}")
@@ -401,9 +418,12 @@ def print_summary(jobs: list[Job], elapsed: float, max_concurrent: int) -> None:
     # copy-paste to debug with real-time noVNC.
     bad = [j for j in jobs if j.status in ("failed", "error")]
     if bad:
-        print("\nTo debug a failed case with live noVNC, re-run it as a single run:")
+        print("\nTo debug a failed case, re-run it as a single run:")
         for j in bad[:10]:
-            print(f"  uv run clawbench-run {j.case_dir} {j.model}")
+            print(
+                f"  uv run clawbench-run {j.case_dir} {j.model} "
+                f"--browser-runtime {browser_runtime}"
+            )
         if len(bad) > 10:
             print(f"  ... and {len(bad) - 10} more")
 
@@ -433,11 +453,18 @@ def print_run_stats(base_output: Path) -> None:
                 model = meta.get("model", model_dir.name)
                 intercepted = meta.get("intercepted", False)
                 duration = meta.get("duration_seconds", 0)
+                browser_runtime = meta.get("browser_runtime")
+                provider_recording = bool(
+                    isinstance(browser_runtime, dict)
+                    and browser_runtime.get("recording_mode") == "provider"
+                    and browser_runtime.get("recording_url")
+                )
             else:
                 case = run_dir.name
                 model = model_dir.name
                 intercepted = False
                 duration = 0
+                provider_recording = False
 
             # Count actions
             actions_file = data / "actions.jsonl"
@@ -462,6 +489,7 @@ def print_run_stats(base_output: Path) -> None:
                     "actions": actions,
                     "screenshots": screenshots,
                     "recording_mb": rec_mb,
+                    "provider_recording": provider_recording,
                     "duration": duration,
                     "intercepted": intercepted,
                 }
@@ -486,13 +514,16 @@ def print_run_stats(base_output: Path) -> None:
         abnormal = (
             r["actions"] == 0
             or r["screenshots"] == 0
-            or r["recording_mb"] < 0.5
+            or (not r["provider_recording"] and r["recording_mb"] < 0.5)
             or r["duration"] < 30
         )
+        recording = (
+            "provider" if r["provider_recording"] else f"{r['recording_mb']:.1f} MB"
+        )
         line = (
             f"{case:<{case_w}}  {r['model']:<{model_w}}  "
             f"{r['actions']:>7}  {r['screenshots']:>11}  "
-            f"{r['recording_mb']:>7.1f} MB  "
+            f"{recording:>10}  "
             f"{fmt_duration(r['duration']):>8}  {result}"
         )
         if abnormal:
@@ -506,7 +537,7 @@ def print_run_stats(base_output: Path) -> None:
         for r in rows
         if r["actions"] == 0
         or r["screenshots"] == 0
-        or r["recording_mb"] < 0.5
+        or (not r["provider_recording"] and r["recording_mb"] < 0.5)
         or r["duration"] < 30
     )
     print(f"\n{total_pass}/{len(rows)} intercepted", end="")
@@ -522,13 +553,15 @@ def write_summary_json(
     elapsed: float,
     max_concurrent: int,
     started_at: str,
+    browser_runtime: str = "local",
 ) -> None:
     now = datetime.now(timezone.utc).isoformat()
     data = {
         "started_at": started_at,
         "finished_at": now,
         "elapsed_seconds": round(elapsed),
         "max_concurrent": max_concurrent,
+        "browser_runtime": browser_runtime,
         "jobs": [
             {
                 "model": j.model,
@@ -555,6 +588,18 @@ async def async_main(args: argparse.Namespace) -> int:
     global shutdown_event
     shutdown_event = asyncio.Event()
     running_procs.clear()
+    browser_runtime = getattr(args, "browser_runtime", None) or "local"
+    if getattr(args, "max_concurrent", None) is None:
+        args.max_concurrent = 1 if browser_runtime == "browserbase" else 2
+    if (
+        browser_runtime == "browserbase"
+        and args.harness == "claude-code-chrome-extension"
+    ):
+        print(
+            "ERROR: browserbase runtime does not support the "
+            "claude-code-chrome-extension harness"
+        )
+        return 1
 
     models = discover_models(args.models, args.all_models)
     cases = discover_cases(
@@ -578,6 +623,7 @@ async def async_main(args: argparse.Namespace) -> int:
     print(
         f"Job matrix: {len(models)} model(s) x {len(cases)} case(s) = {len(jobs)} job(s)"
     )
+    print(f"Browser runtime: {browser_runtime} (max_concurrent={args.max_concurrent})")
     for j in jobs:
         print(f"  {j.case_name} x {j.model}")
 
@@ -681,6 +727,9 @@ async def _noop() -> None:
                 batch_start,
                 no_upload=args.no_upload,
                 harness=args.harness,
+                browser_runtime=browser_runtime,
+                browser_cdp_url=getattr(args, "browser_cdp_url", None),
+                browser_runtime_options=getattr(args, "browser_runtime_options", None),
                 judge=args.judge,
                 no_judge=args.no_judge,
             )
@@ -700,9 +749,16 @@ async def _noop() -> None:
     loop.remove_signal_handler(signal.SIGTERM)
 
     elapsed = time.monotonic() - batch_start
-    print_summary(jobs, elapsed, args.max_concurrent)
+    print_summary(jobs, elapsed, args.max_concurrent, browser_runtime)
     print_run_stats(base_output)
-    write_summary_json(jobs, base_output, elapsed, args.max_concurrent, started_at)
+    write_summary_json(
+        jobs,
+        base_output,
+        elapsed,
+        args.max_concurrent,
+        started_at,
+        browser_runtime,
+    )
     print(f"\nSummary written to {base_output / 'batch-summary.json'}")
 
     # Upload batch summary to HuggingFace
@@ -762,7 +818,10 @@ def main() -> None:
     )
     p.add_argument("--case-range", default=None, help="Numeric ID range, e.g. 1-50")
     p.add_argument(
-        "--max-concurrent", type=int, default=2, help="Max parallel jobs (default: 2)"
+        "--max-concurrent",
+        type=int,
+        default=None,
+        help="Max parallel jobs (default: 1 for browserbase, otherwise 2)",
     )
     p.add_argument("--output-dir", default="test-output", help="Base output directory")
     p.add_argument(
@@ -797,6 +856,27 @@ def main() -> None:
         default=DEFAULT_HARNESS,
         help=f"Coding-agent harness (default: {DEFAULT_HARNESS})",
     )
+    from clawbench.runner.run_support.browser_runtime import BROWSER_RUNTIME_CHOICES
+
+    p.add_argument(
+        "--browser-runtime",
+        choices=BROWSER_RUNTIME_CHOICES,
+        default=None,
+        help=(
+            "Browser runtime provider: local, remote-cdp, steel, or browserbase "
+            "(default: local)"
+        ),
+    )
+    p.add_argument(
+        "--browser-cdp-url",
+        default=None,
+        help="CDP endpoint for --browser-runtime remote-cdp",
+    )
+    p.add_argument(
+        "--browser-runtime-options",
+        default=None,
+        help="JSON object with provider-specific browser runtime options",
+    )
     p.add_argument(
         "--judge",
         default="deepseek-v4-pro",