GRPO Reproducibility

[ashish230897]

Hello,

I have noticed some variability in the benchmark scores by running the same GRPO experiment multiple times. I understand that the grpo code is not entirely reproducible, however, have you noticed the variations to be a lot?

Also, have you tried more ways to make the runs more reproducible?

Thank you!

[vwxyzjn]

Hi @ashish230897, we haven't runinto that many reproducibility problems to be honest, but we did run into stability issues. E.g., sequence length explosion (and as a result eval collapse), similar to some issues Open R1 is running into, too https://huggingface.co/spaces/open-r1/README/discussions/20#67f62c1beb05c1fc2632d024.

Could you share mode details on how the runs were unreproducible? Could you share configs?

[ashish230897]

Thanks for your response.
So my training plots look very identical when I re-run the experiment, but the results fluctuate, for eg here is the verifiable correct rates of two different runs:

The benchmark scores change by 1 point between two different runs, which is not a lot, but i have not thoroughly analysed the change for all the different benchmarks.

The config that I use is: rollouts 256, beta 0.1, lr 2e-7, num epochs 4, kl3 estimator,16 samples,500k episodes
(kept the beta high for the sake of this experiment)

However, I was thinking of adding these reproducibility changes in the code:

In the main function:
torch.use_deterministic_algorithms(True) torch.backends.cudnn.deterministic = True torch.backends.cudnn.benchmark = False os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8" set_seed(args.seed) random.seed(args.seed) np.random.seed(args.seed) torch.manual_seed(args.seed) torch.cuda.manual_seed(args.seed) torch.cuda.manual_seed_all(args.seed)

and these changes in init of RayProcess class:
torch.cuda.manual_seed(self.rank) torch.cuda.manual_seed_all(self.rank)