Skip to content

feat(eval): task-set schema and loader — Step 1 (#1514) - #1520

Closed
Lexus2016 wants to merge 5 commits into
mainfrom
evolution/issue-1514-eval-harness-step1
Closed

feat(eval): task-set schema and loader — Step 1 (#1514)#1520
Lexus2016 wants to merge 5 commits into
mainfrom
evolution/issue-1514-eval-harness-step1

Conversation

@Lexus2016

Copy link
Copy Markdown
Owner

Summary

Implements Step 1 of the eval harness (issue #1514, parent #1481).

Changes

  • EvalTask dataclass — frozen, with fields: id, prompt, expected_tools, expected_result_pattern (optional regex), category, difficulty, version, notes
  • v1.0 task set — 8 tasks across 4 categories: code (2), search (2), file_ops (2), reasoning (2)
  • Versioning_TASK_SETS registry, list_versions(), latest_version()
  • load_task_set(version) — loader returning tasks for a version (defaults to latest); raises ValueError for unknown versions
  • Helper functionstask_count(), categories(), difficulties()

Design

Test plan

  • 24 unit tests (tests/scripts/test_eval_tasks.py) — all pass
  • Tests cover: schema fields/defaults, compiled_pattern(), versioning, loader, v1.0 requirements (5-10 tasks, ≥3 categories, unique IDs, version consistency)
  • No external dependencies — import-safe and fast

Requirements met (from issue #1514)

  • ✅ EvalTask schema with id, prompt, expected_tools, expected_result_pattern, category, difficulty
  • ✅ 5-10 tasks (8 tasks)
  • ✅ ≥3 tool categories (4 categories: code, search, file_ops, reasoning)
  • ✅ Versioning (v1.0, extensible registry)
  • load_task_set() loader

Closes #1514

Add EvalTask dataclass (id, prompt, expected_tools, expected_result_pattern,
category, difficulty, version), a curated v1.0 task set with 8 tasks across
4 categories (code, search, file_ops, reasoning), versioning support, and a
load_task_set() loader with helper functions.

Closes #1514
@Lexus2016

Copy link
Copy Markdown
Owner Author

Blocked at code-review (autonomous evolution-integration): DEAD CODE.

scripts/eval_tasks.py is imported ONLY by tests/scripts/test_eval_tasks.py — there are zero call sites in the live tree. grep -rn 'eval_tasks\|load_task_set\|EvalTask' . --include='*.py' (excluding the module + its test) returns nothing. This matches the dead-code pattern the integration gate rejects (a green-CI module nothing imports).

The idea (issue #1514) is sound — this is Step 1 of a decomposed roadmap and a data-layer-only PR is a legitimate decomposition choice. But a module with no consumer is not mergeable on its own: it cannot be exercised and provides no behavior. Sending the issue back for rework with a concrete integration requirement — see the comment on #1514.

Closing this PR (its branch is a dead end); the next implementation run will rebuild against the rework brief.

@Lexus2016 Lexus2016 closed this Jul 31, 2026
@Lexus2016
Lexus2016 deleted the evolution/issue-1514-eval-harness-step1 branch July 31, 2026 21:07
Lexus2016 added a commit that referenced this pull request Aug 1, 2026
…loses #1514) (#1549)

Step 1 of the eval harness: a fixed, versioned task set (EvalTask schema +
load_task_set() loader + 5 real evolution-cycle tasks across 4 categories).

Per the rework brief (PR #1520 was closed as dead code), the data layer is
smoke-wired into scripts/evolution_evaluator.py at import so its loader is
exercised in CI and import failures surface — the module ships with a live
consumer rather than standing alone.

Co-authored-by: Hermes Evolution <evolution@hermes.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Eval Harness] Step 1: Fixed versioned task set (5-10 tasks from evolution cycles)

1 participant