Loading Tokenizer from GGUF model

[yvonneprice]

I'm writing a small proof-of-concept of command-line chatbot to learn more about Candle and Rust that uses local LLMs for inference.

From the examples I have seen, it seems like we need to provide the path of the tokenizer file. However, reading this thread in mistral.rs, it seems that it is not necessary, a GGUF file has the information necessary to create a tokenizer. In fact, mistral.rs implemented this: EricLBuehler/mistral.rs#345

I haven't been able to find anything similar in Candle. Did I overlook this, or Candle does not have it? If Candle doesn't have it, do you have plans to add it? Would you accept a contribution? (I'm new to both Candle and Rust, but I'd like to give this a try)

Thanks!

[ljt019]

Yeah I was wondering the same thing, they also have the templates built in as well. I think quantized models aren't a priority for them, which is understandable.

[ljt019]

I think much like the candle-onnx discussion below, they probably just see candle-transformers as a proof of concept for a lot of these things and so aren't worried about making it feature complete, hopefully leaving that to a future dedicated library. Im not affiliated with the project though so that's just a guess.

[AlpineVibrations]

seams like a good and useful feature.

[ljt019]

This is something I plan to add to the transformers crate you starred before ;)

But if you want to implement it yourself just grab the jinja template for any model and start messing around with minjinja. I'd be happy to work on something like this for Candle directly if it's something they wanted, but I think it's kind of out of scope for what they want candle-transformers to be