Skip to content

[Eval Harness] Step 3-4: Null-agent baseline + run baseline, output scores.jsonl #1516

Description

@Lexus2016

Parent: #1481 (Eval harness foundation — decomposed per owner request)

This is Step 3-4 of the eval harness (null-agent + baseline run). Depends on #1514 (Step 1: task set) and #1515 (Step 2: runner).

Scope (this issue — ≤200 lines)

Implement the null-agent (an agent that declines every task) and run it against the task set to produce floor baseline scores in scores.jsonl. This establishes the adversarial floor that #1267 requires.

Specific changes

  1. Null-agent: a minimal agent config that declines every task (system prompt: "Decline all tasks. Respond with 'I cannot do this.'"). No tools, no reasoning — pure refusal.

  2. run_null_agent_baseline(task_set, output_path): runs the null-agent against all tasks in the task set using the runner from [Eval Harness] Step 2: Agent runner — execute task, capture full trajectory #1515, captures trajectories, and writes results to scores.jsonl.

  3. scores.jsonl format: one JSON object per line:

    {"task_id": "...", "agent": "null", "tool_calls": 0, "final_answer": "I cannot do this.", "score": 0, "timestamp": "..."}
  4. Baseline scoring: a simple scoring function that checks whether the agent made any tool calls toward the expected tools and whether the result matches the expected pattern. Null-agent should score 0 on all tasks.

What this issue does NOT do

Success criteria

  • Null-agent config implemented (declines all tasks)
  • run_null_agent_baseline() runs null-agent against all tasks
  • scores.jsonl output with correct format
  • Simple scoring function (0 for null-agent on all tasks)
  • Unit test: null-agent produces 0 scores on all tasks
  • Unit test: scores.jsonl is valid JSONL
  • Fits ≤200-line self-merge cap

Dependencies

Unblocks

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions