feat: CrossJudgeExperiment for cross-version judge stability analysis#24
Merged
Merged
Conversation
Provides CrossJudgeExperiment, CrossJudgeResults, and compare_judges to measure how judge-model version affects severity ratings and score distributions on identical subject responses. - CrossJudgeExperiment: orchestrates AuditExperiment runs under multiple judge models; namespaces save_dir per judge to prevent file collision and enable independent resume across judges - CrossJudgeResults: per-judge RepeatedExperimentResults with severity_shifts() for detecting cross-judge modal disagreements and score_summary() - compare_judges(): post-hoc utility for comparing two RepeatedExperimentResults - _safe_judge_label(): derives filesystem-safe labels (strips anthropic:/claude-) - 23 tests covering save-dir isolation, n_repetitions, resume, shift detection, compare_judges utility, label derivation, and input validation - examples/cross_judge_example.py: working nav_aap usage example Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Owner
|
Thanks a lot for the contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
CrossJudgeExperiment, a wrapper aroundAuditExperimentthat orchestrates evaluations across multiple judge models and produces cross-judge stability reports. Also addsCrossJudgeResultsand a standalonecompare_judges()utility.Motivation
Direct follow-up to #23. The judge-stability study published at https://github.com/avalyset/nordeval-skills/tree/main/studies/judge-stability-opus-4.7-4.8 found that identical Haiku 4.5 responses on the AAP Klagefrist scenario (legal deadline misinformation, folketrygdloven §21-12 — errors here cause real rights forfeiture) shifted from modal HIGH under Opus 4.7 to modal LOW under Opus 4.8. Same model, same answers. A judge-upgrade can mask previously detected safety-relevant findings.
The study ran two separate
AuditExperimentinstances and compared results manually via a customanalyze.py. This PR makes that workflow first-class in the framework.Design
AuditExperiment.RepeatedExperimentResultsis consumed, not modified.save_dir/<judge_label>/), so multiple judges can share a parent path without overwriting each other's run files.AuditExperimentmaintains its own cache; partial completion across judges is supported (tested explicitly).compare_judges(a, b, subject_label)is exposed for direct comparison of two already-completedRepeatedExperimentResults.What
CrossJudgeResultsprovidesPer subject, per scenario:
Testing
26 new tests in
tests/test_cross_judge.py, 96% coverage oncross_judge.py. Covers:save_dirseparation across judgesn_repetitionsper judgecompare_judges()utility, including mismatched scenario setsto_dict()JSON roundtriprun()sync wrapperUncovered lines (5 of 116) are defensive guards: empty-dict
ValueError, unused__contains__, empty-list edge inseverity_shifts, and the active-event-loop guard inrun().All 293 existing tests still pass.
Example
See
examples/cross_judge_example.pyfor a working demo with fake LLM clients.Open questions
CrossJudgeExperimentvsMultiJudgeExperimentvsJudgeComparisonExperiment? Happy to rename.CrossJudgeResults.to_dict()schema mirrorRepeatedExperimentResults.to_dict()more closely, or stay distinct?compare_judges()with mismatched scenario sets, current behavior is to skip scenarios missing from either report. Should it warn instead, or error?Happy to iterate on review.