You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Parent: #1481 (Eval harness foundation — decomposed per owner request)
This is Step 3-4 of the eval harness (null-agent + baseline run). Depends on #1514 (Step 1: task set) and #1515 (Step 2: runner).
Scope (this issue — ≤200 lines)
Implement the null-agent (an agent that declines every task) and run it against the task set to produce floor baseline scores in scores.jsonl. This establishes the adversarial floor that #1267 requires.
Specific changes
Null-agent: a minimal agent config that declines every task (system prompt: "Decline all tasks. Respond with 'I cannot do this.'"). No tools, no reasoning — pure refusal.
{"task_id": "...", "agent": "null", "tool_calls": 0, "final_answer": "I cannot do this.", "score": 0, "timestamp": "..."}
Baseline scoring: a simple scoring function that checks whether the agent made any tool calls toward the expected tools and whether the result matches the expected pattern. Null-agent should score 0 on all tasks.
Parent: #1481 (Eval harness foundation — decomposed per owner request)
This is Step 3-4 of the eval harness (null-agent + baseline run). Depends on #1514 (Step 1: task set) and #1515 (Step 2: runner).
Scope (this issue — ≤200 lines)
Implement the null-agent (an agent that declines every task) and run it against the task set to produce floor baseline scores in
scores.jsonl. This establishes the adversarial floor that #1267 requires.Specific changes
Null-agent: a minimal agent config that declines every task (system prompt: "Decline all tasks. Respond with 'I cannot do this.'"). No tools, no reasoning — pure refusal.
run_null_agent_baseline(task_set, output_path): runs the null-agent against all tasks in the task set using the runner from [Eval Harness] Step 2: Agent runner — execute task, capture full trajectory #1515, captures trajectories, and writes results toscores.jsonl.scores.jsonl format: one JSON object per line:
{"task_id": "...", "agent": "null", "tool_calls": 0, "final_answer": "I cannot do this.", "score": 0, "timestamp": "..."}Baseline scoring: a simple scoring function that checks whether the agent made any tool calls toward the expected tools and whether the result matches the expected pattern. Null-agent should score 0 on all tasks.
What this issue does NOT do
Success criteria
run_null_agent_baseline()runs null-agent against all tasksscores.jsonloutput with correct formatDependencies
Unblocks