Fix accuracy reward for math #566
Merged
+13
−8
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lewtun
commented
Mar 31, 2025
|
|
||
| def test_accuracy_reward_wrong_answer_no_latex(self): |
There was a problem hiding this comment.
This test fails on main because we defaulted to reward = 1 when the gold answer could not be parsed.
lewtun
commented
Mar 31, 2025
| """Reward function that checks if the completion is the same as the ground truth.""" | ||
| contents = [completion[0]["content"] for completion in completions] | ||
| rewards = [] | ||
| for content, sol in zip(contents, solution): | ||
| gold_parsed = parse( | ||
| sol, | ||
| extraction_mode="first_match", | ||
| extraction_config=[LatexExtractionConfig()], |
There was a problem hiding this comment.
It is better to use the default extraction config to allow one to also parse pure numbers like 6 in the answer
qgallouedec
reviewed
Mar 31, 2025
edbeeching
approved these changes
Apr 1, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds an important fix to the
accuracy_reward()function to:verify()Noneinstead of 1. This will exclude corrupted samples from the loss => more stableThe latter is particular important to avoid getting spurious reward curves where the base accuracy is high simply because the gold answers could not be parsed.
Here are some WandB logs to show the effect, but you can already see it in this screenshot (red =
mainwhile green =this pr)I am not sure why we decided to set the default reward to 1 for parsing errors, but it seems counterintuitive to me.
Fixes #557