Add add_end_token to Mistral tokenizer

[pbontrager]

Llama3 Tokenizer was updated to support add_end_token in tokenize_messages to support correct generation #1494. These changes need to be made to Mistral.

Changes:

• Update tokenize_message to use add_start_tokens and add_end_tokens like in Make final EOT tag optional for Llama3 tokenizer #1494
• Replace add_eos with add_end_tokens and update tokenize_messages as in Make final EOT tag optional for Llama3 tokenizer #1494
• In call update tokens, mask = self.tokenize_messages(messages) to tokens, mask = self.tokenize_messages(messages, add_end_tokens=not inference)

[umar07]

Hi @pbontrager, I am new to torchtune and OS contribution, and I was looking into fixing this issue. I have a question about the first bullet point where it is said to update tokenize_message() but the Mistral tokenizer does not use a separate tokenize_message? It uses the one from the base ModelTokenizer class in transforms/tokenizers/_utils. So what exactly do we do for the first point? Do we wish to implement our own tokenize_message as in the Llama tokenizer?

The rest of the two bullet points are simple replacements which are clear.
If I am understanding correctly, then I would like to contribute to it. Thanks for your help!

[pbontrager]

Thank you for taking this on! I didn't realize that a few models (gemma, llama2, and Mistral) don't have their own tokenize_messages method but instead use a implementation under modules/transformers/tokenizers/_utils.py. You'd need to update tokenize_messages_no_special_tokens to use add_start_tokens and end_start_tokens and have to update gemma and llama2 as well to properly call that function. This will be a bit more complicated than originally laid out. So if you still want to take this on, I'm happy to help, otherwise you can take one of the similar ones but for one of the other models like Qwen.

[umar07]

Hi, Thanks for clarifying it. Yeah I was also thinking if it needs major change in common files like modules/transformers/tokenizers/_utils.py. Let me look into it and I'll get back to you.