[Eval Harness] Step 1: Fixed versioned task set (5-10 tasks from evolution cycles)

[Lexus2016]

Parent: #1481 (Eval harness foundation — decomposed per owner request)

This is Step 1 of the eval harness, decomposed from #1481 into independently-mergeable ≤200-line slices. Unblocks #1267 (null-agent floor test) and #1302 (replay grading).

Scope (this issue — ≤200 lines)

Define a fixed, versioned task set — a gold-standard eval dataset that the harness runner (Step 2) and null-agent (Step 3) will consume. This is the data layer; no runner code yet.

Specific changes

1. Task set schema: a simple dataclass or TypedDict defining a task:

   • id: unique task identifier
   • prompt: the task instruction text
   • expected_tools: list of tool calls the agent should make (for trajectory grading)
   • expected_result_pattern: optional regex/substring for result validation
   • category: task category (code, search, file-ops, etc.)
   • difficulty: easy/medium/hard

2. Task definitions: 5-10 tasks sourced from existing evolution cycle tasks (real work the agent has done), covering different tool categories. Stored as a Python module or JSON file.

3. Versioning: the task set has a version field so eval runs can pin to a specific set.

4. Loader function: load_task_set(version: str = "1.0") -> list[EvalTask]

What this issue does NOT do

• Does NOT build the runner (Step 2 / [Eval Harness] Step 2: Agent runner — execute task, capture full trajectory #1515)
• Does NOT implement the null-agent (Step 3 / [Eval Harness] Step 3-4: Null-agent baseline + run baseline, output scores.jsonl #1516)
• Does NOT run any baseline scoring (Step 4)

Success criteria

• EvalTask schema defined with all fields
• 5-10 tasks defined, covering ≥3 tool categories
• Task set versioned (v1.0)
• load_task_set() loader implemented
• Unit test: loader returns correct tasks
• Unit test: all tasks have required fields
• Fits ≤200-line self-merge cap

Existing code checked

• scripts/evolution_evaluator.py — deterministic scoring rubric (text grading only, not a task set)
• tests/agent/test_agent_judge.py — Agent-as-a-Judge eval (different scope)
• tests/agent/test_plan_mode_eval.py — plan-mode eval harness (different scope, model comparison)
• No existing eval task set in the codebase (verified by grep)

[Lexus2016]

Blocked by code-review (PR #1520 closed): the Step-1 data layer shipped with zero live call sites — scripts/eval_tasks.py is imported only by its own test, so it is currently dead code and was not mergeable.

The decomposition into a pure-data Step 1 is a reasonable design, but a module with no consumer cannot land alone. To re-open a PR, do ONE of:

1. Smoke-wire it into an existing eval path so the loader is actually exercised. The closest existing surface is scripts/evolution_evaluator.py (referenced in the issue as already-checked existing code). Have it import and call load_task_set() / latest_version() at module load or in a no-op path so the schema is live and import-failures surface in CI. Even a single assert task_count() >= 5 sanity check wired into the existing evaluator's import is enough to make it non-dead.

2. Merge Step 1 + the Step-2 runner skeleton into one PR ([Eval Harness] Step 2: Agent runner — execute task, capture full trajectory #1515), so the data layer ships with its first real consumer. This is the cleaner decomposition: the data layer and its consumer land together and neither is dead.

Definition of done for re-review: a non-test, non-scripts/eval_tasks.py module must import and call at least one public symbol (load_task_set, EvalTask, latest_version, etc.). Confirm with:

grep -rn 'eval_tasks\|load_task_set\|EvalTask' . --include='*.py' | grep -viE 'tests/|scripts/eval_tasks.py'

must return ≥1 real call site.

Re-queued as needs-work — the next implementation run will pick this up.