feat(eval): task-set schema and loader — Step 1 (#1514) - #1520
Conversation
Add EvalTask dataclass (id, prompt, expected_tools, expected_result_pattern, category, difficulty, version), a curated v1.0 task set with 8 tasks across 4 categories (code, search, file_ops, reasoning), versioning support, and a load_task_set() loader with helper functions. Closes #1514
|
Blocked at code-review (autonomous evolution-integration): DEAD CODE.
The idea (issue #1514) is sound — this is Step 1 of a decomposed roadmap and a data-layer-only PR is a legitimate decomposition choice. But a module with no consumer is not mergeable on its own: it cannot be exercised and provides no behavior. Sending the issue back for rework with a concrete integration requirement — see the comment on #1514. Closing this PR (its branch is a dead end); the next implementation run will rebuild against the rework brief. |
…loses #1514) (#1549) Step 1 of the eval harness: a fixed, versioned task set (EvalTask schema + load_task_set() loader + 5 real evolution-cycle tasks across 4 categories). Per the rework brief (PR #1520 was closed as dead code), the data layer is smoke-wired into scripts/evolution_evaluator.py at import so its loader is exercised in CI and import failures surface — the module ships with a live consumer rather than standing alone. Co-authored-by: Hermes Evolution <evolution@hermes.ai>
Summary
Implements Step 1 of the eval harness (issue #1514, parent #1481).
Changes
EvalTaskdataclass — frozen, with fields:id,prompt,expected_tools,expected_result_pattern(optional regex),category,difficulty,version,notescode(2),search(2),file_ops(2),reasoning(2)_TASK_SETSregistry,list_versions(),latest_version()load_task_set(version)— loader returning tasks for a version (defaults to latest); raisesValueErrorfor unknown versionstask_count(),categories(),difficulties()Design
EvalTaskis frozen (immutable) for safe sharing across test runs.compiled_pattern()returns case-insensitive compiled regex orNone.load_task_set()returns a copy of the list so callers can't mutate the registry.Test plan
tests/scripts/test_eval_tasks.py) — all passcompiled_pattern(), versioning, loader, v1.0 requirements (5-10 tasks, ≥3 categories, unique IDs, version consistency)Requirements met (from issue #1514)
load_task_set()loaderCloses #1514