General tracker of RLHF methods addition

[krammnic]

Approved by @SalmanMohammadi. This issue's purpose is to determine which contrastive optimization methods should be added in torchtune and track methods that are implementing right now. For all of the methods, reference implementation exists. Here is the list of the methods (most methods are from A* alignment workshops) with their implementation state and some info about them (HF models trained with method, limitations, notes about actuality):

• DPO: Classic method, with which constrative methods are usually compared. Models: >6k, Status: Implemented DPO #645
• SimPO: Current SOTA method according to their paper. Models: ~252, Limitations: The most important limitation is performance drop on reasoning datasets. Method is actually lack of robustness for tasks where specific tokens are important. For instance, 2 + 2 = 5 is a bad result compared to 2 + 2 = 4. For the v0.2 version, it can't be normally used in cases where structured outputs are required, but it is actually a version where robustness to learning rate is maximized. See also general limitations. Status: Implemented, SimPO (Simple Preference Optimisation) #1223
• IPO: Basic log-ratio method; I am not sure of its actuality. Models: <180, Limitations: Are not really clear from the paper. See also general limitations. Status: Stale, remove ipo loss + small fixed #1615
• KTO: Kahneman-Tversky value function-based method, which looks really interesting and also was used for reasoning tasks. Models: ~531 Limitations: It is not a real limitation, but if your data is of enough quality, DPO will work better than this method. Status: Not implemented, issue Adding a KTO Optimizer #1793
• ORPO: Pretty popular and stable log-ratio method, which has actually worse performance than SimPO, same computational and memory efficiency, but it does not require SFT initialized model. Models: ~1220, Limitations: Problems with reasoning datasets (checked emperically). See also general limitations. Status: Not implemented.
• R-DPO: Really interesting, but non-trivial method, closest to SimPO by performance. Generally, it is a reward-model (incorporated in DPO loss) method with teacher/student distillation. Models: 29, Limitations: Are not really clear from the paper. It is fully robust to noise that might be generated while building synthetic. Also, it looks like it has better robustness to hyperparameters than log-ratio methods. The reward model might be counted as a potential limitation. Status: Not implemented.
• CPO: Was actually designed for translation tasks, log-ratio method; not sure about its actuality. Models: <120, Limitations: Probably pretty similar to other log-ratio methods' limitations. See also: General limitations. Status: Not implemented, Implement CPO (Contrastive Preference Optimization) #1290
• SMPO: Generally, it is the newest method, which by some experiments beats SimPO. There is an official implementation, but no published paper or pre-print, so we will wait for it. Limitations: See also: General limitations. Status: Not implemented,

General limitations:

For all log-ratio methods, it is obviously robustness to hyperparameters, especially the most critical, learning rate. Accurate transcription of them from the original paper is crucial to having good results.

[krammnic]

I assume that we need to start with KTO actually, looks like most interesting method from current list. There are questions about ORPO, even if it is still popular. My opinion that we need to add in torchtune only actual/SOTA methods, according to limitations. Addition of tons of methods(like many alignment libraries do) is useless and makes things complicated.

[krammnic]

Today will PR KTO method. Also will set some expirements about SMPO. Which methods do you think it will be interesting to add?

[SalmanMohammadi]

Hey @krammnic. Sorry for the delay in getting back to you here.

Thanks so much for kicking all this off and you effort. I think your level of detail for each method is really clear and helpful, particularly seeing the number of models on HF for each method.

If you're saying there isn't an official implementation for SMPO I'd prefer to hold off on there. I'd like to see techniques achieve some level of maturity and some appetite in the open-source community before we pull the trigger and start think about adding to torchtune.

I'll take a look at your KTO PR just now! I think that looks like a great start.

[bjb19]

Hey sorry I was at a Work Conference so was not able to contribute during that time frame but I got caught up on the items happening and the KTO Loss PR. Happy to hop in and fill another need if we want ORPO in. I have not read the paper but have seen lots of the models tuned with it doing well

[krammnic]

Hey sorry I was at a Work Conference so was not able to contribute during that time frame but I got caught up on the items happening and the KTO Loss PR. Happy to hop in and fill another need if we want ORPO in. I have not read the paper but have seen lots of the models tuned with it doing well

Hello! Should be actually discussed with @SalmanMohammadi. I see that it easily can be added as it pretty similar to SimPO, so the only change is required - ORPO loss addition, which is actually great. Speaking about methods generally, the most wished one to add after KTO is R-DPO, due to really interesting performance and synthetic structure.