Reproducing JustRL

[lee-junjie]

Thank you for your excellent work on JustRL. I really appreciate the simplicity of the approach and am very interested in reproducing the results using VERL.

However, I have encountered several issues while attempting to reproduce the reported results.

Following your recommended settings in this comment:
#3 (comment)

I used VERL version 0.2.0 or 0.2.0.post2 and ran the provided training script. During execution, I encountered multiple configuration validation errors. Specifically, more than 10 configuration keys referenced in the script do not exist in VERL 0.2.0. One example error is shown below:

Could not override 'algorithm.use_kl_in_reward'.
To append to your config use +algorithm.use_kl_in_reward=False
Key 'use_kl_in_reward' is not in struct
    full_key: algorithm.use_kl_in_reward
    object_type=dict

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

The following configuration options do not appear to exist in VERL 0.2.0. To pass configuration validation, I had to remove them from the script; however, this may result in inconsistencies with the setup you used:

• algorithm.use_kl_in_reward
• data.filter_overlong_prompts
• data.truncation
• actor_rollout_ref.actor.optim.lr_warmup_steps
• actor_rollout_ref.actor.optim.weight_decay
• actor_rollout_ref.actor.use_kl_loss
• actor_rollout_ref.actor.clip_ratio_low
• actor_rollout_ref.actor.clip_ratio_high
• actor_rollout_ref.actor.clip_ratio_c
• actor_rollout_ref.rollout.val_kwargs.do_sample
• actor_rollout_ref.rollout.val_kwargs.n
• actor_rollout_ref.rollout.val_kwargs.temperature
• actor_rollout_ref.rollout.val_kwargs.top_p

Could you please clarify whether these configuration options were implemented as custom modifications on top of VERL 0.2.0, or whether a different VERL version was used for your experiments?

Additionally, in your replies on both GitHub and Zhihu, you mentioned that the JustRL code is largely the same as VERL and that the results should be reproducible across multiple VERL versions:

• GitHub: 项目仓库里能补充训练代码吗？如何进行复现？ #1 (comment)
• Zhihu: https://www.zhihu.com/question/1987478921730613767/answer/1988591596640416580

Have you tested JustRL with more recent versions of VERL? If so, could you please share which VERL and vLLM versions you would recommend for reproduction? I would be happy to try them on my side. If not, could you please share additional details about the original environment, such as the Docker image, exact configurations, and code modifications required to reproduce the results?

Finally, it would be greatly appreciated if other teams who have successfully reproduced the results could share their VERL/vLLM versions and relevant training scripts.

Thank you very much for your time and help.

[HBX-hbx]

We use verl version of v0.2.0.dev, which is a development version of verl v0.2.0 and most features seem to be merged and released in v0.3.x and v0.4.x. For the same verl codebase of v0.2.0.dev, we have rerun three times. And we previously tested on verl v0.4.0 for JustRL-Deepseek-1.5B, but due to the limit of compute, we canceled the training at around 1k steps. AIME 24 improves normally as in the paper.

Since v0.2.0.dev is not supported officially, we recommend using verl with v0.4.x and vllm with v0.8.4. Thanks for pointing this out. If there are any other issues, we are happy to provide more details!

[lee-junjie]

Thanks a lot for the detailed explanation. Since v0.2.0.dev is no longer available as a branch or tag in VERL, I’ll follow your suggestion and try reproducing the results using v0.4.0.

I had a couple of quick follow-up questions, if you don’t mind:

• For the three runs you conducted with v0.2.0.dev, did you observe fairly consistent results? In particular, could you share the approximate range (best to worst) of performance on AIME 2024?

• When using v0.4.0, do you see similar ranges for metrics such as entropy, gradient norms, and token counts? If so, that would give me more confidence in reproducing the results on this branch.

Thanks again for your time. Please let me know if there’s anything else I should be aware of.

[HBX-hbx]

Sorry for the late reply. For v0.2.0.dev experiments, we rerun the deepseek backbone until its score on AIME24 exceeds 50. As you can see from figure 1 of the paper, there could be a fluctuation of 5 points for AIME24 within just a few steps. And for v0.4.0 codebase, the entropy is around 1.0, keeping a relatively normal value for long-horizon RL. The grad norm is around 0.05 and token counts are decreasing to around 6000 tokens. And we recently migrated the training code to v0.4.0 and uploaded it to this repo. Hope it will help!

[lee-junjie]

Thanks, @HBX-hbx — really appreciate the detailed explanation and for sharing the updated training code based on v0.4.0. I’ll definitely give it a try once I have machines available for testing.

Regarding the comment that “the entropy is around 1.0” for the v0.4.0 codebase: is this referring to the initial entropy? That value is similar to what I’ve observed when testing with the latest verl. I’m also curious about how the entropy evolves later in training, as well as the final AIME 2024 score.

Additionally, if possible, could you share the training curves from the v0.4.0 runs, such as:

• AIME 2024 score
• Entropy
• Token length
• Gradient norm

These would be very helpful for comparison and analysis. Thanks again for your help!

[Yusifu]

Have you reproduced the results using the OpenMath-Nemotron-1.5B model on the VERL framework (version 0.4.0)? If so, could you share the training curves, including reward, entropy, sequence length, gradient norm, and validation set scores?

[lee-junjie]

Have you reproduced the results using the OpenMath-Nemotron-1.5B model on the VERL framework (version 0.4.0)? If so, could you share the training curves, including reward, entropy, sequence length, gradient norm, and validation set scores?

I ran a few initial steps with the DeepSeek 1.5B model but stopped early due to mismatches in entropy, response length, and other metrics. I shared my observation here: #3 (comment).

[Yusifu]

Sorry for the late reply. For v0.2.0.dev experiments, we rerun the deepseek backbone until its score on AIME24 exceeds 50. As you can see from figure 1 of the paper, there could be a fluctuation of 5 points for AIME24 within just a few steps. And for v0.4.0 codebase, the entropy is around 1.0, keeping a relatively normal value for long-horizon RL. The grad norm is around 0.05 and token counts are decreasing to around 6000 tokens. And we recently migrated the training code to v0.4.0 and uploaded it to this repo. Hope it will help!

Thank you for your detailed response and for the excellent and impactful work. Have you reproduced the results using the OpenMath-Nemotron-1.5B model on the VERL framework (version 0.4.0)? If so, could you share the training curves, including reward, entropy, sequence length, gradient norm, clip ratio, and validation set scores?

[Yusifu]

Have you reproduced the results using the OpenMath-Nemotron-1.5B model on the VERL framework (version 0.4.0)? If so, could you share the training curves, including reward, entropy, sequence length, gradient norm, and validation set scores?

I ran a few initial steps with the DeepSeek 1.5B model but stopped early due to mismatches in entropy, response length, and other metrics. I shared my observation here: #3 (comment).

Thank you for your quick response. I’m also working on reproducing the JustRL project and have encountered some issues. Could you please add me on WeChat (my email is 1701214306@pku.edu.cn)? We can discuss the details of the reproduction process.

[lee-junjie]

Have you reproduced the results using the OpenMath-Nemotron-1.5B model on the VERL framework (version 0.4.0)? If so, could you share the training curves, including reward, entropy, sequence length, gradient norm, and validation set scores?

I ran a few initial steps with the DeepSeek 1.5B model but stopped early due to mismatches in entropy, response length, and other metrics. I shared my observation here: #3 (comment).

Thank you for your quick response. I’m also working on reproducing the JustRL project and have encountered some issues. Could you please add me on WeChat (my email is 1701214306@pku.edu.cn)? We can discuss the details of the reproduction process.

Happy to discuss, I sent you an Email, please kindly check.

[Yusifu]

Sorry for the late reply. For v0.2.0.dev experiments, we rerun the deepseek backbone until its score on AIME24 exceeds 50. As you can see from figure 1 of the paper, there could be a fluctuation of 5 points for AIME24 within just a few steps. And for v0.4.0 codebase, the entropy is around 1.0, keeping a relatively normal value for long-horizon RL. The grad norm is around 0.05 and token counts are decreasing to around 6000 tokens. And we recently migrated the training code to v0.4.0 and uploaded it to this repo. Hope it will help!

JustRL is an excellent work. 👍 Improving its reproducibility will help broaden JustRL’s impact. I am currently on the foundation models team at a major tech company, using the Nemotron model to reproduce JustRL. During this process, we have encountered some technical issues and developed related technical insights. Would it be convenient for you to connect via WeChat? You can also reach me by email at 1701214306@pku.edu.cn.