Skip to content

fix(datasets): anchored grade extraction + surface judge failures in generic llmjudge - #2598

Open
YuhaoLin2005 wants to merge 1 commit into
open-compass:mainfrom
YuhaoLin2005:fix/generic-llmjudge-anchored-grade
Open

fix(datasets): anchored grade extraction + surface judge failures in generic llmjudge#2598
YuhaoLin2005 wants to merge 1 commit into
open-compass:mainfrom
YuhaoLin2005:fix/generic-llmjudge-anchored-grade

Conversation

@YuhaoLin2005

Copy link
Copy Markdown

Motivation

_generic_llmjudge_postprocess scans the judge's free-text reply for the first A/B anywhere in it. Judges routinely justify their verdict before stating it, so a reasoning sentence that merely contains a letter silently overrides the real grade ("As a judge, I considered the evidence carefully. The final grade is B." parses as A). Separately, when a judge fails to emit a usable grade (API error, truncation, no letter in the reply), the result degrades to 'unknown', which get_final_results folds into the accuracy denominator — indistinguishable from a genuine wrong answer. Related upstream reports: #1232, #2522, #2392.

The fix (two parts)

1. Anchored grade extraction (no regression)

Only a grade that is explicitly anchored is trusted: a cue word (grade, verdict, final answer) immediately followed by a connector (is / was / be / of / : / = / :=) and the letter. Everything else keeps the existing loose scan, so every judge reply that previously parsed still parses (zero recall regression, no silently-dropped samples).

Deliberate exclusions, each with a regression test:

  • bare answer is not a cue — "the answer is A" names the option, not the grade;
  • a connector is required — "grade A of this study" is a quality qualifier, not an assignment.

2. Judge failures are surfaced, not absorbed

get_final_results now counts judge failures separately (judge_error_count, plus a judge_error flag on each failing detail), while accuracy, accuracy_given_attempted, not_attempted_count, and friends keep their exact historical semantics (unknown still counts as not-attempted). A judge failure can no longer be silently read as a wrong answer.

Tests

tests/datasets/test_generic_llmjudge_postprocess.py — 13 cases covering anchored extraction, the anti-false-anchor contract, and the aggregation surface. Follows tests/TESTING_GUIDE.md conventions.

Impact on existing results

This fix changes how judge replies that contain an anchored grade are parsed:
previously such replies could be mis-parsed as the first letter appearing in the
judge's prose (often wrong); now the anchored grade wins. Replies with a single
letter, or with no conflicting prose letter, are parsed exactly as before, so
only previously-wrong results change — in the correct direction. accuracy,
accuracy_given_attempted, and not_attempted_count keep their exact
historical semantics; judge_error_count is a new, purely additive field.

Out of scope

  • Connector-less verb forms ("I grade A") are deliberately not covered and
    continue to follow the legacy scan — adding them would reintroduce the
    qualifier false-positive the connector requirement exists to prevent.
  • The same loose-scan logic is copy-pasted into MedXpertQA.py,
    medmcqa.py, supergpqa/supergpqa.py, and atlas/evaluation.py (each with
    its own _generic_llmjudge_postprocess / get_final_results). This PR keeps
    the diff focused on the canonical generic.py; a follow-up could apply the
    same anchored-extraction pattern to the copies.

…generic llmjudge

_generic_llmjudge_postprocess took the first A/B anywhere in the judge's
free-text reply, so reasoning letters silently overrode the real grade. Only
an explicitly anchored grade (cue word + connector + letter: 'grade is B',
'grade: B', 'final answer = A', 'grade of A') is now trusted; everything else
keeps the legacy loose scan (zero recall regression).

get_final_results now surfaces judge failures via judge_error_count and a
per-detail judge_error flag, while accuracy/not_attempted_count keep their
historical semantics (unknown still counts as not-attempted).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants