Skip to content

NOTES.md

Latest commit

 

History

History
43 lines (32 loc) · 2.36 KB

NOTES.md

File metadata and controls

43 lines (32 loc) · 2.36 KB
Raw

Lab notebook - reward-model-ppo

Running log of findings. The writeup leads with these.

Finding 1 - the reward model is the easy, solid half

A LoRA reward model on Qwen2.5-1.5B-Instruct, trained on 10k UltraFeedback pairs (Bradley-Terry), reaches 0.757 held-out accuracy (n=1000): on a held-out pair it scores the preferred response higher 76% of the time. That is a healthy reward model and a clean, reusable artifact. Training a good RM was straightforward; the hard part is what you do with it.

Finding 2 - PPO is the hard half: memory, then instability

Optimizing a policy against the reward model with PPO ran into two walls, in order.

Wall 1 - memory. PPO holds four roles at once (policy, frozen reference, reward model, value head) plus rollout buffers reused over several PPO epochs.

  • The 8 GB GPU could not run even a 0.5B policy through the loop (OOM).
  • An 80 GB A100 OOM'd too on the first attempt (1.5B policy, batch 16, response length 128 - it filled all 80 GB after one step).
  • It fit only after cutting to batch 8 / response 64 / rollout-forward 8, capping prompts to 256 tokens, and enabling expandable_segments. Memory is a first-class design constraint here, not an afterthought.

Wall 2 - instability / convergence. With the run that fit (1,280 episodes, kl_coef 0.05), reward did NOT cleanly climb: it bounced in roughly the -2 to -6 band with no clear trend, while the KL to the reference drifted up and stayed noisy (~12 -> ~22). A short, lightly-tuned run did not move the policy against the KL penalty. PPO is well known to need careful and often many-run tuning (lr, kl_coef, batch, episode budget) to get a clean reward-up / KL-controlled curve.

The point (the through-line). Earlier projects used DPO - a single, stable step that collapses the reward model and the RL loop into one objective, and it worked first time. Building the classic two-step RM + PPO loop here surfaces exactly what DPO was designed to avoid: PPO's memory cost and its tuning instability. Having done both, the tradeoff is concrete rather than abstract: DPO buys stability and simplicity by giving up the explicit, inspectable reward model and online exploration that PPO provides.

Artifacts: reward model yavuz-ai/qwen2.5-1.5b-rm-ultrafeedback (use it), PPO policy yavuz-ai/qwen2.5-1.5b-ppo (research artifact - the loop ran, reward did not cleanly improve).