[Question] qwen2_5_dpo got an incorrect input_id.

[cageyoko]

Required prerequisites

• I have read the documentation https://align-anything.readthedocs.io.
• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• Consider asking first in a Discussion.

Questions

Hello, I don't know if this is an error in the example about qwen2_5_dpo.

response_length:

return_dict['better_response_lens'] = len(
            self.tokenize(meta_info['better_response'], add_special_tokens=False)['input_ids'][0]
        )

input_id: "'re looking for sites that feature weird sexual fantasies, there are many to choose from. Try Fet Life, Dark Play, Fantasy Forum, Kinky Monkeys, Funky Kink, Weird Fantasies, Sex and Death, or The Tenth House. You can also search the web for specific fetishes or fantasies that interest you, or look for message boards and forums where you can share your own weird fantasies. Additionally, many erotica or adult novels feature extreme or unique fantasies, so you could look for these at your local bookstore or through online retailers.<|im_end|>\n"
input_id = raw_input_id[-response_length:].unsqueeze(0)

Because of Qwen's chat_template, two additional tokens are appended, which causes an offset in the input_id retrieved via response_length.

[cageyoko]

Additionally, I found that the KTOTrainer in kto.py inherits from DPOTrainer in dpo, and DPOTrainer uses left padding by default during model initialization, which leads to the issue where diverge_index starts from 0 in KTO.

[cageyoko]

align-anything/align_anything/trainers/text_to_text/kto.py

Line 81 in 3f9decc

self.kl = max(kl, 0)

The loop iterates through all batches, but self.kl is overwritten each time, so only the value of the last batch is saved in the end.