TIGER-AI-Lab · Perry2004 · Apr 16, 2026 · Apr 16, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -4,7 +4,7 @@ This file is for coding agents (Claude Code, Cursor, Copilot, etc.) to understan
 
 ## What This Is
 
-ClawBench is a benchmarking framework for evaluating AI web agents on 153 real-world online tasks spanning 144 live websites and 15 life categories. Each task runs in an isolated Docker container with Chromium, a recording Chrome extension, and an AI agent harness (`openclaw` or `opencode`, selectable via `--harness`). The framework captures five layers of data: session replay (MP4), action screenshots, HTTP traffic, browser actions, and agent messages.
+ClawBench is a benchmarking framework for evaluating AI web agents on 153 real-world online tasks spanning 144 live websites and 15 life categories. Each task runs in an isolated Docker container with Chromium, a recording Chrome extension, and an AI agent harness (`openclaw`, `opencode`, or `claude-code`, selectable via `--harness`). The framework captures five layers of data: session replay (MP4), action screenshots, HTTP traffic, browser actions, and agent messages.
 
 ## Project Structure
 
@@ -14,11 +14,14 @@ ClawBench/
   Dockerfile.base                 # Base image (Chromium + extension + Node + uv, harness-agnostic)
   Dockerfile.openclaw             # Layer that adds the openclaw CLI on top of base
   Dockerfile.opencode             # Layer that adds opencode + @playwright/mcp on top of base
+  Dockerfile.claude-code          # Layer that adds Claude Code + @playwright/mcp + LiteLLM on top of base
   entrypoint.sh                   # Shared infra (Xvfb, Chrome, noVNC, human mode); execs /run-harness.sh in agent mode
   setup-openclaw.sh               # Generates ~/.openclaw/openclaw.json from env vars (called from run-openclaw.sh)
   setup-opencode.sh               # Generates ~/.config/opencode/opencode.json from env vars (called from run-opencode.sh)
+  setup-claude-code.sh            # Configures API keys + LiteLLM proxy from env vars (called from run-claude-code.sh)
   run-openclaw.sh                 # Per-harness agent runner; copied into clawbench-openclaw image as /run-harness.sh
   run-opencode.sh                 # Per-harness agent runner; copied into clawbench-opencode image as /run-harness.sh
+  run-claude-code.sh              # Per-harness agent runner; copied into clawbench-claude-code image as /run-harness.sh
   .env.example                    # Template for PurelyMail credentials
   models/
     models.yaml                   # Model API configs (gitignored -- copy from example)
@@ -72,6 +75,9 @@ cp models/models.example.yaml models/models.yaml
 # Single run:
 uv run --project test-driver test-driver/run.py test-cases/<case-dir> <model-name> --harness openclaw
 
+# Single run with Claude Code harness:
+uv run --project test-driver test-driver/run.py test-cases/<case-dir> <model-name> --harness claude-code
+
 # Batch run (model x case cross-product):
 uv run --project test-driver test-driver/batch.py \
   --models <model-name> --all-cases --max-concurrent 3 --harness openclaw
@@ -111,7 +117,8 @@ test-output/<model>/<harness>-<case>-<model>-<timestamp>/
     requests.jsonl          # HTTP request log
     agent-messages.jsonl    # Agent conversation transcript (shape varies by harness:
                             #   openclaw → session-schema events; opencode → AI-SDK
-                            #   `step_start`/`tool_use`/`text`/`reasoning`/`step_finish` events)
+                            #   `step_start`/`tool_use`/`text`/`reasoning`/`step_finish` events;
+                            #   claude-code → stream-json events with `system`/`assistant`/`user`/`result` types)
     screenshots/            # Timestamped PNGs
     recording.mp4           # Full session video
     interception.json       # Interception result

diff --git a/Dockerfile.claude-code b/Dockerfile.claude-code
@@ -0,0 +1,16 @@
+FROM clawbench-base
+
+# Pin Claude Code and Playwright MCP versions for reproducible builds —
+# matches the openclaw-side `openclaw@2026.3.13` pin pattern.
+RUN PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm install -g \
+    @anthropic-ai/claude-code@2.1.110 \
+    @playwright/mcp@0.0.70
+
+# LiteLLM proxy translates Anthropic-format requests to other API formats
+# (OpenAI, Google, etc.) so Claude Code can work with any api_type.
+# Only started at runtime when api_type != anthropic-messages.
+RUN pip install --no-cache-dir 'litellm[proxy]'
+
+COPY setup-claude-code.sh /setup-claude-code.sh
+COPY run-claude-code.sh /run-harness.sh
+RUN chmod +x /setup-claude-code.sh /run-harness.sh
diff --git a/README.md b/README.md
@@ -158,7 +158,7 @@ clawbench run 001-daily-life-food-uber-eats claude-sonnet-4-6
 ```
 Once the container starts, the script prints a **noVNC URL** (e.g. `http://localhost:6080/vnc.html`) — open it in your browser to watch the agent operate in real-time. If port 6080 is already in use, an alternative port is chosen automatically.
 
-Results land in `./claw-output/<model>/<harness>-<case>-<model>-<timestamp>/` with the full five-layer recording. The default harness is `openclaw`; pass `--harness opencode` to use [opencode](https://opencode.ai) (driven via the [Playwright MCP server](https://github.com/microsoft/playwright-mcp)) instead.
+Results land in `./claw-output/<model>/<harness>-<case>-<model>-<timestamp>/` with the full five-layer recording. The default harness is `openclaw`; pass `--harness opencode` to use [opencode](https://opencode.ai) or `--harness claude-code` to use [Claude Code](https://docs.anthropic.com/en/docs/claude-code) (both driven via the [Playwright MCP server](https://github.com/microsoft/playwright-mcp)).
 
 **(c) Drive the browser yourself via noVNC** — produces a human reference run:
 ```bash
@@ -625,4 +625,4 @@ Open to contributions — new test cases, bug fixes, or evaluation submissions f
 
 Apache 2.0 -- see [LICENSE](LICENSE).
 
-Built with [OpenClaw](https://github.com/openclaw/openclaw) and [opencode](https://opencode.ai) (selectable harnesses), [Microsoft Playwright MCP](https://github.com/microsoft/playwright-mcp) (browser control bridge for the opencode harness), [noVNC](https://github.com/novnc/noVNC) (MPL 2.0), and [websockify](https://github.com/novnc/websockify) (LGPL 3.0).
+Built with [OpenClaw](https://github.com/openclaw/openclaw), [opencode](https://opencode.ai), and [Claude Code](https://docs.anthropic.com/en/docs/claude-code) (selectable harnesses), [Microsoft Playwright MCP](https://github.com/microsoft/playwright-mcp) (browser control bridge for the opencode and claude-code harnesses), [LiteLLM](https://github.com/BerriAI/litellm) (API translation proxy for the claude-code harness), [noVNC](https://github.com/novnc/noVNC) (MPL 2.0), and [websockify](https://github.com/novnc/websockify) (LGPL 3.0).
diff --git a/README.zh-CN.md b/README.zh-CN.md
@@ -158,7 +158,7 @@ clawbench run 001-daily-life-food-uber-eats claude-sonnet-4-6
 ```
 容器启动后,脚本会打印一个 **noVNC URL**（如 `http://localhost:6080/vnc.html`）—— 在浏览器中打开即可实时观看 agent 操作。如果 6080 端口被占用,会自动选一个空闲端口。
 
-结果落在 `./claw-output/<model>/<harness>-<case>-<model>-<timestamp>/`,包含完整的五层录制。默认 harness 是 `openclaw`；想用 [opencode](https://opencode.ai)（通过 [Playwright MCP](https://github.com/microsoft/playwright-mcp) 驱动浏览器）就加 `--harness opencode`。
+结果落在 `./claw-output/<model>/<harness>-<case>-<model>-<timestamp>/`,包含完整的五层录制。默认 harness 是 `openclaw`；想用 [opencode](https://opencode.ai) 就加 `--harness opencode`，想用 [Claude Code](https://docs.anthropic.com/en/docs/claude-code) 就加 `--harness claude-code`（两者均通过 [Playwright MCP](https://github.com/microsoft/playwright-mcp) 驱动浏览器）。
 
 **(c) 通过 noVNC 手动控制浏览器** —— 产出人工参考轨迹:
 ```bash
@@ -624,4 +624,4 @@ uv run --project test-driver test-driver/batch.py --all-models --case-range 1-50
 
 Apache 2.0 -- 详见 [LICENSE](LICENSE)。
 
-基于以下开源项目构建: [OpenClaw](https://github.com/openclaw/openclaw) 和 [opencode](https://opencode.ai)（可选的 harness）, [Microsoft Playwright MCP](https://github.com/microsoft/playwright-mcp)（opencode harness 的浏览器控制桥）, [noVNC](https://github.com/novnc/noVNC) (MPL 2.0), [websockify](https://github.com/novnc/websockify) (LGPL 3.0)。
+基于以下开源项目构建: [OpenClaw](https://github.com/openclaw/openclaw)、[opencode](https://opencode.ai) 和 [Claude Code](https://docs.anthropic.com/en/docs/claude-code)（可选的 harness）, [Microsoft Playwright MCP](https://github.com/microsoft/playwright-mcp)（opencode 和 claude-code harness 的浏览器控制桥）, [LiteLLM](https://github.com/BerriAI/litellm)（claude-code harness 的 API 转换代理）, [noVNC](https://github.com/novnc/noVNC) (MPL 2.0), [websockify](https://github.com/novnc/websockify) (LGPL 3.0)。
diff --git a/run-claude-code.sh b/run-claude-code.sh
@@ -0,0 +1,144 @@
+#!/bin/bash
+set -e
+
+# Run-time harness script for the Claude Code agent.
+/setup-claude-code.sh
+
+# Source the env vars written by setup (ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL).
+source /tmp/claude-code-env.sh
+
+# Start LiteLLM translation proxy if needed (non-Anthropic api_types).
+# The proxy accepts Anthropic-format requests and translates them to the
+# target format (OpenAI, Google, etc.).
+PROXY_PID=""
+if [ -f /tmp/litellm-config.yaml ]; then
+  echo "Starting API translation proxy (litellm)..."
+  litellm --config /tmp/litellm-config.yaml --port 4000 \
+    > /data/proxy.log 2>&1 &
+  PROXY_PID=$!
+  for i in $(seq 1 30); do
+    if curl -sf http://localhost:4000/health/liveliness > /dev/null 2>&1; then
+      echo "API proxy ready"
+      break
+    fi
+    if [ "$i" -eq 30 ]; then
+      echo "API proxy not ready after 30s — check /data/proxy.log"
+      echo "proxy_failed" > /data/.stop-reason
+      exit 1
+    fi
+    sleep 1
+  done
+fi
+
+# Copy /my-info/ into the workspace so the agent can access it via ./my-info/
+WORKSPACE=/root/workspace
+mkdir -p "$WORKSPACE"
+if [ -d /my-info ]; then
+  cp -r /my-info "$WORKSPACE/my-info"
+  echo "Copied /my-info to $WORKSPACE/my-info"
+fi
+
+# Wait for Chrome CDP to be ready
+echo "Waiting for Chrome CDP..."
+for i in $(seq 1 30); do
+  if curl -sf http://127.0.0.1:9222/json/version > /dev/null 2>&1; then
+    echo "Chrome CDP ready"
+    break
+  fi
+  if [ "$i" -eq 30 ]; then
+    echo "Chrome CDP not ready after 30s, aborting"
+    echo "chrome_cdp_timeout" > /data/.stop-reason
+    exit 1
+  fi
+  sleep 1
+done
+
+# Build the claude command.
+# -p (print mode): non-interactive, runs the prompt to completion.
+# --output-format stream-json: streams one JSON object per line to stdout.
+# --dangerously-skip-permissions: skip permission prompts (container is the sandbox).
+# IS_SANDBOX=1 bypasses the root-user check for --dangerously-skip-permissions.
+export IS_SANDBOX=1
+cd "$WORKSPACE"
+echo "Starting Claude Code agent (model=${MODEL_NAME})..."
+CLAUDE_ARGS=(-p --verbose --output-format stream-json --model "$MODEL_NAME" --dangerously-skip-permissions)
+# --thinking requires a mode: enabled | adaptive | disabled.
+case "${THINKING_LEVEL:-off}" in
+  ""|off)  ;;
+  *)       CLAUDE_ARGS+=(--thinking enabled) ;;
+esac
+# --mcp-config is variadic; use -- to stop it consuming the prompt.
+CLAUDE_ARGS+=(--mcp-config /tmp/claude-mcp.json -- "$INSTRUCTION")
+claude "${CLAUDE_ARGS[@]}" \
+  > /data/agent-messages.jsonl 2> /data/agent.log &
+AGENT_PID=$!
+sleep 3
+
+# Watchdog: detect agent no action for 300s
+IDLE_THRESHOLD=300
+MAX_WAIT=${TIME_LIMIT_S:-1800}
+ELAPSED=0
+LAST_SIZE=0
+IDLE=0
+STOP_REASON=""
+
+while kill -0 $AGENT_PID 2>/dev/null && [ "$ELAPSED" -lt "$MAX_WAIT" ]; do
+  sleep 5
+  ELAPSED=$((ELAPSED + 5))
+
+  # Check if server requested stop (eval interceptor matched)
+  if [ -f /data/.stop-requested ]; then
+    echo "Stop requested by server (eval matched), killing agent."
+    STOP_REASON="eval_matched"
+    break
+  fi
+
+  CURRENT_SIZE=$(wc -c < /data/actions.jsonl 2>/dev/null || echo 0)
+
+  if [ "$CURRENT_SIZE" -gt 0 ] && [ "$CURRENT_SIZE" -eq "$LAST_SIZE" ]; then
+    IDLE=$((IDLE + 5))
+    if [ "$IDLE" -ge "$IDLE_THRESHOLD" ]; then
+      echo "Agent idle for ${IDLE_THRESHOLD}s, assuming done."
+      STOP_REASON="agent_idle"
+      break
+    fi
+  else
+    IDLE=0
+  fi
+  LAST_SIZE=$CURRENT_SIZE
+done
+
+# Determine stop reason if not set (loop exited without breaking)
+if [ -z "$STOP_REASON" ]; then
+  if ! kill -0 $AGENT_PID 2>/dev/null; then
+    STOP_REASON="agent_exited"
+  else
+    echo "Time limit (${MAX_WAIT}s) exceeded, killing agent."
+    STOP_REASON="time_limit_exceeded"
+  fi
+fi
+
+echo "$STOP_REASON" > /data/.stop-reason
+
+# Kill Claude Code, MCP, and proxy processes
+kill $AGENT_PID 2>/dev/null || true
+[ -n "$PROXY_PID" ] && kill $PROXY_PID 2>/dev/null || true
+pkill -f "@anthropic-ai/claude-code" 2>/dev/null || true
+pkill -f "@playwright/mcp" 2>/dev/null || true
+pkill -f "litellm" 2>/dev/null || true
+sleep 2
+
+curl -sf -X POST http://localhost:7878/api/stop || true
+
+# Clean up internal marker (created by /api/stop)
+rm -f /data/.stop-requested
+
+# Grace period: keep recording for 15s after agent is killed to capture end result
+echo "Agent finished, recording grace period (15s)..."
+sleep 15
+
+# Stop recording
+echo "Stopping recording..."
+curl -sf -X POST http://localhost:7878/api/stop-recording || true
+sleep 2
+echo "Done."
diff --git a/setup-claude-code.sh b/setup-claude-code.sh
@@ -0,0 +1,143 @@
+#!/bin/bash
+set -e
+
+# All config comes from env vars set by the test driver (sourced from models.yaml).
+# BASE_URL and MODEL_NAME are required.
+if [ -z "$BASE_URL" ] || [ -z "$MODEL_NAME" ]; then
+  echo "ERROR: BASE_URL and MODEL_NAME must be set"
+  exit 1
+fi
+
+if [ -n "$TEMPERATURE" ]; then
+  echo "WARN: Claude Code CLI does not expose a --temperature flag; TEMPERATURE='$TEMPERATURE' will be ignored."
+fi
+if [ -n "$MAX_TOKENS" ]; then
+  echo "WARN: Claude Code CLI does not expose a --max-tokens flag; MAX_TOKENS='$MAX_TOKENS' will be ignored."
+fi
+
+# Extract API key, write env-var exports, and optionally configure the
+# LiteLLM translation proxy for non-Anthropic api_types.
+python3 - <<'PYEOF'
+import json, os
+from pathlib import Path
+
+base_url = os.environ["BASE_URL"]
+model_name = os.environ["MODEL_NAME"]
+api_type = os.environ.get("API_TYPE", "anthropic-messages")
+
+# Pick a single API key (first from API_KEYS list, else API_KEY).
+keys_json = os.environ.get("API_KEYS", "")
+single_key = os.environ.get("API_KEY", "")
+key = ""
+if keys_json:
+    try:
+        parsed = json.loads(keys_json)
+        if parsed:
+            key = parsed[0]
+            if len(parsed) > 1:
+                print(f"WARN: Claude Code does not rotate keys — using first of {len(parsed)}")
+    except json.JSONDecodeError:
+        pass
+if not key and single_key:
+    key = single_key
+if not key:
+    raise SystemExit("ERROR: no API key provided (API_KEYS or API_KEY)")
+
+# ── Decide whether to use the LiteLLM translation proxy ──────────────
+# Claude Code speaks only the Anthropic messages API. For other api_types
+# we start a LiteLLM proxy that accepts Anthropic-format requests on a
+# local port and translates them to the target format.
+needs_proxy = api_type != "anthropic-messages"
+
+if needs_proxy:
+    import urllib.request
+    import yaml  # installed as a litellm dependency
+
+    is_openrouter = "openrouter.ai" in base_url
+
+    if is_openrouter:
+        # LiteLLM's openrouter/ provider properly translates reasoning
+        # content between Anthropic and OpenAI formats. It needs the full
+        # OpenRouter model ID (e.g. "qwen/qwen3.5-397b-a17b"), so resolve
+        # it from the models endpoint.
+        full_model_id = model_name
+        try:
+            req = urllib.request.Request(
+                f"{base_url}/models",
+                headers={"Authorization": f"Bearer {key}"},
+            )
+            resp = json.loads(urllib.request.urlopen(req, timeout=10).read())
+            for m in resp.get("data", []):
+                if m["id"].endswith(f"/{model_name}") or m["id"] == model_name:
+                    full_model_id = m["id"]
+                    break
+        except Exception as e:
+            print(f"WARN: could not resolve OpenRouter model ID: {e}")
+
+        litellm_model = f"openrouter/{full_model_id}"
+        litellm_params = {
+            "model": litellm_model,
+            "api_key": key,
+        }
+    else:
+        # Generic OpenAI-compatible endpoint.
+        _PREFIX_MAP = {
+            "openai-completions": "openai",
+            "openai-responses":   "openai",
+            "google-generative-ai": "gemini",
+        }
+        prefix = _PREFIX_MAP.get(api_type, "openai")
+        litellm_model = f"{prefix}/{model_name}"
+        litellm_params = {
+            "model": litellm_model,
+            "api_key": key,
+            "api_base": base_url,
+        }
+
+    proxy_config = {
+        "model_list": [{
+            "model_name": model_name,
+            "litellm_params": litellm_params,
+        }],
+        # drop_params: silently drop Anthropic-specific params that can't
+        # be translated (e.g. thinking budget) instead of erroring.
+        "litellm_settings": {"drop_params": True},
+    }
+
+    proxy_path = Path("/tmp/litellm-config.yaml")
+    proxy_path.write_text(yaml.dump(proxy_config, default_flow_style=False))
+    os.chmod(proxy_path, 0o600)
+
+    # Claude Code will talk to the local proxy instead of the real endpoint.
+    # The proxy key is arbitrary — LiteLLM proxy doesn't validate it.
+    anthropic_base_url = "http://localhost:4000"
+    anthropic_api_key = "sk-proxy-placeholder"
+    print(f"API proxy enabled: {api_type} → litellm ({litellm_model})")
+else:
+    anthropic_base_url = base_url
+    anthropic_api_key = key
+
+# Write a sourceable env file for the run script.
+env_path = Path("/tmp/claude-code-env.sh")
+env_lines = [
+    f'export ANTHROPIC_API_KEY="{anthropic_api_key}"',
+    f'export ANTHROPIC_BASE_URL="{anthropic_base_url}"',
+]
+env_path.write_text("\n".join(env_lines) + "\n")
+os.chmod(env_path, 0o600)
+
+# Write MCP config file for --mcp-config flag.
+mcp_config = {
+    "mcpServers": {
+        "playwright": {
+            "command": "npx",
+            "args": ["@playwright/mcp", "--cdp-endpoint", "http://127.0.0.1:9222"],
+        },
+    },
+}
+mcp_path = Path("/tmp/claude-mcp.json")
+mcp_path.write_text(json.dumps(mcp_config, indent=2))
+os.chmod(mcp_path, 0o644)
+
+print(f"Claude Code config: model={model_name}, base_url={anthropic_base_url}")
+PYEOF