[Fix] Molmo AI2D: map no-letter answers to the option letter, not its… - #1655
Open
aarjunsrinivasan wants to merge 1 commit into
Open
[Fix] Molmo AI2D: map no-letter answers to the option letter, not its…#1655aarjunsrinivasan wants to merge 1 commit into
aarjunsrinivasan wants to merge 1 commit into
Conversation
… position
The ai2_diagram_no_letter prompt hides the A/B/C/D labels, so the reply is
mapped back to a letter -- previously by position, which is only correct when
the option columns are a contiguous prefix. AI2D_TEST index 83301 has an empty
A column (options B/C/D), so the correct reply 'C', the value of column D, was
scored as 'C' and the row was unwinnable for any model. Carry the
{letter: value} mapping alongside the prompt and map back to the key, matching
what the labeled path already does. The prompt sent to the model is unchanged.
Also return an unmatched reply unchanged instead of raising ValueError, which
SKIP_ERR does not catch (it handles only RuntimeError), so one stray token
aborted a multi-hour eval.
Molmo-7B-D-0924, AI2D_TEST, exact_matching: 2511/3088 -> 2512/3088.
One row fixed, none regressed, 3087/3088 predictions bit-identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Molmo's
ai2_diagram_no_letterprompt shows AI2D options with their A/B/C/Dlabels stripped, so
generate_innermaps the model's reply back to a letter.It did this by position in the printed list, which is only correct when the
option columns are a contiguous prefix.
AI2D_TEST index 83301 has an empty
Acolumn, so its options are B/C/D. Theprompt lists
['B', 'A', 'C']; Molmo answered'C'— the value of column D,i.e. correct — but position 2 mapped it to
'C'. The row was scored wrong andwas unwinnable for any model. The lettered
ai2_diagram:path already maps tothe column key, so the two paths disagreed.
Fix
Carry the
{letter: value}mapping alongside the prompt on the message item(the same way video datasets carry
role) and map the reply back to the key.The prompt text sent to the model is unchanged.
Also stop aborting the run on an unrecognised reply:
options.index()raisedValueError, whichSKIP_ERRdoes not catch (it handles onlyRuntimeError),so one stray token discarded a full eval. Replies are now normalised for
whitespace, punctuation and case, and returned unchanged when nothing matches,
leaving the decision to the existing answer matcher.
Validation
Molmo-7B-D-0924, AI2D_TEST,
--judge exact_matching,max_crops=36:C→D)3087/3088 predictions are bit-identical — the prompt is unchanged, so no
published Molmo AI2D number moves. Confirmed two independent ways: offline
rescoring of the stored predictions, and a full re-run that reproduced the
same single row.
The
ai2_diagram_no_letterpath was added in #648 to match the official Molmoeval; this keeps that behaviour and fixes only the letter mapping.