fix(datasets): anchored grade extraction + surface judge failures in generic llmjudge - #2598
Open
YuhaoLin2005 wants to merge 1 commit into
Open
Conversation
…generic llmjudge _generic_llmjudge_postprocess took the first A/B anywhere in the judge's free-text reply, so reasoning letters silently overrode the real grade. Only an explicitly anchored grade (cue word + connector + letter: 'grade is B', 'grade: B', 'final answer = A', 'grade of A') is now trusted; everything else keeps the legacy loose scan (zero recall regression). get_final_results now surfaces judge failures via judge_error_count and a per-detail judge_error flag, while accuracy/not_attempted_count keep their historical semantics (unknown still counts as not-attempted).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
_generic_llmjudge_postprocessscans the judge's free-text reply for the firstA/Banywhere in it. Judges routinely justify their verdict before stating it, so a reasoning sentence that merely contains a letter silently overrides the real grade ("As a judge, I considered the evidence carefully. The final grade is B." parses asA). Separately, when a judge fails to emit a usable grade (API error, truncation, no letter in the reply), the result degrades to'unknown', whichget_final_resultsfolds into the accuracy denominator — indistinguishable from a genuine wrong answer. Related upstream reports: #1232, #2522, #2392.The fix (two parts)
1. Anchored grade extraction (no regression)
Only a grade that is explicitly anchored is trusted: a cue word (
grade,verdict,final answer) immediately followed by a connector (is/was/be/of/:/=/:=) and the letter. Everything else keeps the existing loose scan, so every judge reply that previously parsed still parses (zero recall regression, no silently-dropped samples).Deliberate exclusions, each with a regression test:
answeris not a cue — "the answer is A" names the option, not the grade;2. Judge failures are surfaced, not absorbed
get_final_resultsnow counts judge failures separately (judge_error_count, plus ajudge_errorflag on each failing detail), whileaccuracy,accuracy_given_attempted,not_attempted_count, and friends keep their exact historical semantics (unknownstill counts as not-attempted). A judge failure can no longer be silently read as a wrong answer.Tests
tests/datasets/test_generic_llmjudge_postprocess.py— 13 cases covering anchored extraction, the anti-false-anchor contract, and the aggregation surface. Followstests/TESTING_GUIDE.mdconventions.Impact on existing results
This fix changes how judge replies that contain an anchored grade are parsed:
previously such replies could be mis-parsed as the first letter appearing in the
judge's prose (often wrong); now the anchored grade wins. Replies with a single
letter, or with no conflicting prose letter, are parsed exactly as before, so
only previously-wrong results change — in the correct direction.
accuracy,accuracy_given_attempted, andnot_attempted_countkeep their exacthistorical semantics;
judge_error_countis a new, purely additive field.Out of scope
continue to follow the legacy scan — adding them would reintroduce the
qualifier false-positive the connector requirement exists to prevent.
MedXpertQA.py,medmcqa.py,supergpqa/supergpqa.py, andatlas/evaluation.py(each withits own
_generic_llmjudge_postprocess/get_final_results). This PR keepsthe diff focused on the canonical
generic.py; a follow-up could apply thesame anchored-extraction pattern to the copies.