feat: 6-dimension rubric with deterministic checks for workflow evals

[RobertCrupa]

Why

Closes #1203.

evals/workflows/ judged every multi-turn conversation with one binary PASS/FAIL from an LLM judge that only saw tool calls, arguments and agent text. A red verdict didn't say which capability regressed, and nothing in the harness was ground truth — tool selection and schema validity have knowable answers that don't need an LLM.

What changed

The judge now scores six independent dimensions (toolSelection, argumentCorrectness, resultUtilization, taskCompletion, errorRecovery, planEfficiency), same six for every test case. overallVerdict is just the taskCompletion verdict — there's no aggregation rule, and the exit code is unchanged, so none of this gates anything.

Two checks are computed in code and reported separately from the LLM's answers:

• Tool selection — exact set-match (deduped, order-independent) against a new optional expectedTools field on WorkflowTestCase. When it's set, the deterministic verdict overrides the judge's and the judge's own answer is discarded; when unset the judge's verdict stands. The compare logic is ported from the existing toolsExactMatch evaluator in evals/run_evaluation.ts rather than re-derived.
• Schema validity — every tool call's arguments validated with AJV against that tool's declared inputSchema. Deliberately not folded into argumentCorrectness: schema-valid arguments can still be semantically wrong. Validators are cached per tool name because tool schemas carry a $id and AJV rejects a second compile of one. Calls naming a tool that isn't in the final tool list, and tools whose schema won't compile, are skipped rather than flagged — neither is an agent mistake.

The judge also now sees tool results, truncated to 2000 chars each, with failed calls rendered as errors rather than omitted. resultUtilization and errorRecovery aren't answerable without them.

expectedTools is backfilled on 25 of the 30 test cases. The 5 left unset are the ones where more than one tool path is genuinely correct: the three "which is the best scraper" searches, where fetching Actor details on top of searching is legitimate, plus the two reachable either via a dedicated Actor or apify/rag-web-browser.

results.json keeps verdict and reason as mirrors of taskCompletion so anything reading the old field names still works, and gains rubric, toolSelectionCheck and schemaValidityCheck as optional fields. Baseline comparison prints per-dimension pass-rate deltas next to the existing byte/token ones, with the arrow polarity flipped since more passes is better here.

Notes for reviewers (human-written)

Proof it works

Not yet proven end-to-end — this is why the PR is a draft. The eval harness needs APIFY_TOKEN and OPENROUTER_API_KEY; neither was available in the environment this was written in, so no live run happened. Still outstanding from the spec's acceptance list:

1. pnpm run evals:workflow -- --id <a few ids across categories> --output --verbose — eyeball all six dimensions in verbose output, and spot-check that a deliberately wrong expectedTools value flips toolSelection regardless of what the judge said.
2. A run over the failTools case, confirming the synthetic error result isn't false-flagged as a schema violation.
3. A full --output run to write the first rubric-bearing baseline, then a second run to confirm the per-dimension deltas print as = baseline. No committed record carries a rubric field yet, so the first run shows no dimension deltas by design.

What is covered: unit tests for the deterministic override flipping toolSelection while the judge says PASS, a failTools-shaped error result not being schema-flagged, and unchanged pass counts rendering as = baseline — i.e. the three behaviors steps 1–3 exist to catch, exercised without live credentials. The table, verbose output and record JSON were also smoke-rendered from synthetic data, and the CLI entry point loads and exits cleanly at the credential gate.

One thing to check on the reviewer's side: the spec mentioned pinning the judge to deepseek/deepseek-v4-flash-0423, but design decision #9 says the judge model is unchanged, so MODELS.judge is left at deepseek/deepseek-v4-flash. I couldn't confirm whether -0423 is a distinct OpenRouter variant — the sandbox's proxy blocks openrouter.ai. Nothing new was hardcoded, so this only matters if the dated variant was actually intended.

AI-assisted: written with Claude Code. https://claude.ai/code/session_0136aUA25ZgcWmM7RMDq3iar