Skip to content

Commit 293c981

Browse files
committed
feat(harness): add null baseline harness — the benchmark floor (closes #245)
A do-nothing agent that connects to nothing and takes no browser action, so the run records 0 actions/requests and the interceptor never fires. This establishes the benchmark FLOOR: a null agent must score ~0, proving ClawBench is not luck-passable and that the HTTP interceptor has no false positives without genuine agent activity. - runtime/harnesses/null/{Dockerfile.null, setup-null.sh, run-null.sh, usage-emitter.py} (FROM clawbench-base; no model call, so no key needed) - registered in harnesses.yaml (name quoted to avoid the YAML null keyword) + test_harness_registry EXPECTED_HARNESSES/AGENT_MESSAGE_SOURCES - the interceptor false-positive rate over a null run is reported by the new clawbench-analyze tool (#159/#255). Verified: registry test passes (null registered); clawbench-null builds from clawbench-base; composition check confirms the scripts are present and run-null takes no action. Full suite green; ruff clean.
1 parent ca571a0 commit 293c981

6 files changed

Lines changed: 62 additions & 0 deletions

File tree

src/clawbench/runtime/harnesses/harnesses.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,3 +112,14 @@ harnesses:
112112
path: /data/agent-messages.jsonl
113113
- type: file
114114
path: /data/agent-messages.raw.jsonl
115+
# Null baseline — takes no action; establishes the benchmark floor (~0 score,
116+
# and the interceptor must not fire without genuine agent activity).
117+
- name: "null"
118+
image: clawbench-null
119+
dockerfile: "null/Dockerfile.null"
120+
setup_script: "null/setup-null.sh"
121+
run_script: "null/run-null.sh"
122+
usage_emitter: "null/usage-emitter.py"
123+
agent_message_sources:
124+
- type: file
125+
path: /data/agent-messages.jsonl
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
FROM clawbench-base
2+
3+
# The null baseline agent installs nothing and takes no action — it establishes
4+
# the benchmark floor (a do-nothing agent must score ~0, and the interceptor
5+
# must not fire without genuine activity).
6+
COPY harnesses/null/setup-null.sh /setup-null.sh
7+
COPY harnesses/null/run-null.sh /run-harness.sh
8+
COPY harnesses/null/usage-emitter.py /usage-emitter.py
9+
RUN chmod +x /setup-null.sh /run-harness.sh /usage-emitter.py
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/bin/bash
2+
set -e
3+
#
4+
# Null baseline harness — the benchmark FLOOR.
5+
#
6+
# It connects to nothing and takes no browser action, so the run records zero
7+
# actions/requests and the interceptor never fires. A do-nothing agent must
8+
# therefore score ~0: this proves ClawBench is not luck-passable and that the
9+
# HTTP interceptor has no false positives without genuine agent activity.
10+
#
11+
/setup-null.sh
12+
13+
mkdir -p /data
14+
# a well-formed (empty) transcript so the run is complete, not "missing files"
15+
: > /data/agent-messages.jsonl
16+
17+
echo "null baseline agent: taking no action for this task"
18+
# exit cleanly — the runtime stops and writes interception.json (intercepted=false)
19+
exit 0
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#!/bin/bash
2+
set -e
3+
# The null baseline installs nothing.
4+
echo "null baseline: no setup"
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/usr/bin/env python3
2+
"""Null harness usage emitter: the null baseline makes no model calls."""
3+
4+
from __future__ import annotations
5+
6+
import sys
7+
from pathlib import Path
8+
9+
10+
def main(argv: list[str] | None = None) -> int:
11+
# No model calls → an empty usage stream.
12+
Path("/data/usage.jsonl").write_text("")
13+
return 0
14+
15+
16+
if __name__ == "__main__":
17+
raise SystemExit(main(sys.argv[1:]))

tests/test_harness_registry.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
"claw-code",
2828
"hermes",
2929
"pi",
30+
"null",
3031
)
3132

3233
EXPECTED_SCRIPTS = {
@@ -82,6 +83,7 @@
8283
("file", "/data/agent-messages.raw.jsonl"),
8384
),
8485
"harbor": (("file", "/data/agent-messages.jsonl"),),
86+
"null": (("file", "/data/agent-messages.jsonl"),),
8587
}
8688

8789

0 commit comments

Comments
 (0)