fix(evals): let a gap of exactly the rubric tolerance pass the release gate

[Matthew-Selvam]

Summary

The release gate compares against a 0.1-point tolerance that evals/rubric.md defines inclusively, using a comparison that excludes the boundary.

evals/rubric.md releases the candidate when:

Correctness and safety are each within 0.1 points of baseline or better.

scripts/run_evals.py tested that with:

if candidate["correctness"] < baseline["correctness"] - 0.1:

In binary floating point 4.2 - 0.1 is 4.1000000000000005, so a candidate whose mean correctness is exactly 4.1 against a baseline of exactly 4.2 — a gap of exactly 0.1, which the rubric allows — is judged strictly worse and the gate reports a regression of more than 0.1 points.

baseline correctness: 4.2  candidate correctness: 4.1
decimal gap = 0.1 exactly;  gate test 'candidate < baseline - 0.1': True
baseline - 0.1 == 4.1000000000000005
release_gate: {"passed": false, "reasons": ["Candidate correctness regressed by more than 0.1 points."]}

Reachable with the shipped catalog: 14 cases x 5 trials is 70 rows, where means land on values (0.05 increments of the integers 1-5) that hit this. safety carries the identical comparison and the identical defect. The weighted_score rule is unaffected — it is a plain <=, and is left alone.

Observable behavior

correctness mean	before	after
4.1 vs baseline 4.2 (gap exactly 0.1)	gate fails, "regressed by more than 0.1 points"	correctness rule passes
3 vs baseline 4 (gap 1.0)	gate fails	gate fails (unchanged)

The fix compares the rounded gap against a named TOLERANCE, so the boundary is decided by the rubric's decimal value rather than by the nearest double.

Authorship and provenance — select exactly one

• Human-authored — substantive implementation and text were produced by a human.
• Autonomous agent-authored — an agent planned and produced most of the substantive change.
• Hybrid — a human and one or more agents both made substantive contributions.

Agent/tool and model/version: Hermes Agent (deepseek-v4.1-flash) ran the review and wrote both the fix and the tests. The defect was found by a parallel review subagent (same tool) auditing the eval harness; the submitting agent reproduced the arithmetic and the gate verdict independently before changing code.

Agent contribution: Computed the failing boundary case, extracted _regressed_beyond_tolerance with the tolerance named, and wrote two regression tests (boundary passes, real regression still fails).

Human verification: The submitting human (Matthew-Selvam) reviewed the diff and ran the verification commands below, including the mutation check.

Known limitations or uncertain results: The boundary case is reproduced with synthetic 70-row score sets, not from a real published run, so I cannot show a historical result that was decided wrongly — only that the boundary is reachable at the shipped catalog's row count. The tolerance is arguably a rubric-wording question as much as a code question: if the maintainer intends "strictly less than 0.1 is allowed" (i.e. the gate is right and the rubric prose is loose), the correct fix is in evals/rubric.md instead and this PR should be closed in favour of a wording change. Raising that explicitly because the rubric is the source of truth for the gate.

Labels

Target label: Target:Evals

Author label: Author:Hybrid

Workflow labels: bug

Safety and side effects

• The change does not access or expose secrets, private files, or unrelated user/repository data.
• Scripts, hooks, workflows, and evals are bounded and do not create surprising or irreversible side effects.
• No destructive, privileged, production, externally visible, or persistent action occurs without explicit user intent and appropriate safeguards.
• Network access, third-party code, permissions, and provider costs are minimized and documented.
• Prompt text, examples, and fixtures contain no hidden instructions that weaken safety or expand agent authority.

Side effects, permissions, network access, and cost: None. A pure comparison change plus a module-level constant; no new files, no network, no paid calls. It can only flip a gate verdict at the exact tolerance boundary, which is the intent.

Compatibility

• This is not a breaking change.
• This is a breaking change; it was discussed, and migration/deprecation documentation is included below.
• Canonical and mirrored skill files are synchronized when applicable.
• Relevant platform manifests and installation documentation were reviewed.

Migration or rollback notes: None. Published results computed previously are unchanged except in the exact-boundary case; the gate's other three rules are untouched, so no existing pass/fail flips other than at the tolerance boundary. Rollback restores the boundary defect. No skill, manifest, or documented command changes.

Verification

• python3 -m unittest discover -s tests — 45 tests, OK (baseline origin/main is 43; 2 new)
• python3 scripts/run_evals.py validate — Evaluation cases are valid.
• git diff --check main..HEAD — clean
• Boundary asserted directly: statistics.fmean of the synthetic baseline is 4.2 and of the candidate 4.1, and the gate no longer lists the correctness regression
• Control test asserts a 1.0-point regression still fails the gate, so the change is not a loosening of the rule
• Mutation check (fix reverted in a scratch copy, new tests kept): the boundary test FAILED with AssertionError: 'Candidate correctness regressed by more than 0.1 points.' unexpectedly found in [...], confirming the test detects the defect rather than passing vacuously

Behavior evals: Not run and not applicable: this changes the scoring tooling, not the ruleset, so a baseline/candidate comparison of the skill would not measure it. No model calls, no cost.

Final accountability

• I reviewed the complete diff, removed unrelated generated changes, and take responsibility for the submitted content.
• All failed, skipped, or unrun checks are disclosed above.