The Role of Format Reward

[xiaobanni]

Thanks for the great resources.

I would like to know whether adding format reward affects the final benchmark performance, or is it just for separating the <think> and <answer> format?

[jssuncx]

I am curious where the format reward function is defined in the code?

[xiaobanni]

Just a general question: there are many R1-like projects, some of which use a format reward (including R1), while others do not (including this project). I’m curious about the impact on final performance or any other considerations that might explain their choices.

[jssuncx]

But the blog of this project says:

Similar to DeepSeek R1, our RL recipe is kept extremely simple without using reward models or MCTS-like techniques. We use the PPO algorithm, and employ a rule-based reward function that assigns rewards based on the format and correctness of the generated response:
- If the response provides a final answer in the specified format and is correct, it receives a reward of +1.
- If the response provides a final answer but it is incorrect, the reward is set to -0.5.
- If the response fails to provide a final answer, the reward is set to -1.

which is very confusing.

[xiaobanni]

This is their prompt: Please reason step by step, and put your final answer within \boxed{}. I guess the specified format means within \boxed{}, not like the R1 format <think><answer>.

[SivilTaram]

Yes the format reward is about \boxed{}.

[huangqingyi-code]

Yes the format reward is about \boxed{}.

@SivilTaram hi, where is the format reward function?

[SivilTaram]

@huangqingyi-code

simpleRL-reason/train/openrlhf/trainer/ppo_utils/experience_maker.py

Line 617 in e49ee66

if "boxed" not in model_output: