an equation issue in your paper

[iboing]

Hi authors,

Thank you for the nice work. I noticed that the PPO equation in Section 3.2.2 may have an issue. The ratio $r$ in PPO should be $\frac{\pi_{\theta}}{\pi_{\text{old}}}$, but your definition in that paragraph is $\frac{\pi_{\theta}}{\pi_{\text{ref}}}$. If I am correct, I think this would cause confusion to readers.

[MAXNORM8650]

Hi,
I appreciate your interest. In this equation, the old policy ($\pi_{\text{old}}$) serves as the reference to control how much the updated policy $\pi_\theta$ can change in a single update.
Thus, in PPO, $\pi_{\text{old}}$ is indeed the reference policy used to calculate the clipped objective for updating the policy. In our paper, we consistently use the term reference policy to refer to the baseline policy used for comparison or guidance, as some methods employ the SFT policy and COT SFT policy for data generation or as a reference. Using the term old policy could create ambiguity regarding which policy is considered ‘old’ in different contexts, so we opted to use reference policy to keep things clear.