Online DPO new changes #2794
Online DPO new changes #2794
Conversation
unsloth/models/rl.py
Outdated
| else: | ||
| loss = None | ||
| with self.compute_loss_context_manager(): | ||
| tokenized_output = self.processing_class(inputs["prompt"], padding=True, truncation=True, return_tensors="pt").to(model.device) |
There was a problem hiding this comment.
tokenized_output? Should it be tokenized_input?
There was a problem hiding this comment.
That is correct, I just corrected it to inputs
|
@pluesclues Whenever its done, ping me! |
unsloth/models/llama.py
Outdated
| padding_mask = None | ||
| elif self.training: | ||
| # elif attention_mask is None: | ||
| elif self.training and os.environ.get("UNSLOTH_KEEP_PADDING", "0") != '1': |
There was a problem hiding this comment.
Re "UNSLOTH_KEEP_PADDING" where is this set exactly?
There was a problem hiding this comment.
This would be set at the beginning of the script when loading so the attention calculations account for right padded tokens. This is something the user would set if using LLM reward models for Online DPO.
There was a problem hiding this comment.
Is there a way to make it automatic?
There was a problem hiding this comment.
There is a way to make this automatic by looking to see if any samples in the batch have right padded tokens and if they do, the flag is immediately enabled.
3 participants
Because now that
AutoSequenceForClassificationor reward modeling support was added and I verified that the run works correctly, the PR is much more simplified to get Online DPO integrated in than the previous one.