Tokenizer

[madhavajay]

This is amazing work.

I want to quantize the models so I can run them on Ollama however it requires the tokenizer. Is it the same as https://huggingface.co/meta-llama/Llama-3.1-8B ?
https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/original/tokenizer.model

I see that vLLM has its own code to get a tokenizer.

Do you know where I can get the right tokenizer.model file and config to use?

[madhavajay]

Okay so I read here that actually llama 3 uses BPE tokenizer:
ggml-org/llama.cpp#6745

And then saw you can confirm this by checking tokenizer.json:

  "model": {
    "type": "BPE",

After removing the tokenizer.model that I downloaded (as I thought that was the issue), I got it working but had to use llama.cpp tooling directly.

uv run python3 ../llama.cpp/convert_hf_to_gguf.py --outtype q8_0 --outfile ./model.gguf  ./mims-harvard_TxAgent-T1-Llama-3.1-8B/

# Modelfile
FROM ./model.gguf

ollama create txagent-llama3.1-8b-q8_0 -f ./Modelfile

ollama run txagent-llama3.1-8b-q8_0
>>> Given a 50-year-old patient experiencing severe acute pain and consider
... ing the use of the newly approved medication, Journavx, how should the 
... dosage be adjusted considering the presence of moderate hepatic impairm
... ent?
 
The question requires adjusting the dosage of Journavx for a patient 
with moderate hepatic impairment. The solution hint suggests 
considering liver function tests such as ALT or AST levels to 
determine the appropriate dose adjustment. However, I need to confirm 
this by consulting reliable medical resources or guidelines. Let's 
proceed with retrieving relevant information about Journavx and its 
usage in patients with hepatic impairment. [TOOL_CALL] 
GetInfoAboutDrugJournavx [/TOOL_CALL]
The solution hint suggests considering liver function tests such as 
ALT or AST levels to determine the appropriate dose adjustment for a 
patient with moderate hepatic impairment. To confirm this, I need to 
retrieve relevant information about Journavx and its usage in 
patients with hepatic impairment. [TOOL_CALL] 
GetInfoAboutDrugJournavx [/TOOL_CALL]

The solution hint suggests considering liver function tests such as 
ALT or AST levels to determine the appropriate dose adjustment for a 
patient with moderate hepatic impairment. To confirm this, I need to 
retrieve relevant information about Journavx and its usage in 
patients with hepatic impairment. [TOOL_CALL] 
GetInfoAboutDrugJournavx [/TOOL_CALL]

The solution hint suggests considering liver function tests such as 
ALT or AST levels to determine the appropriate dose adjustment for a 
patient with moderate hepatic impairment. To confirm this, I need to 
retrieve relevant information about Journavx and its usage in 
patients with hepatic impairment. [TOOL_CALL] 
GetInfoAboutDrugJournavx [/TOOL_CALL]

And it keeps looping. Do I need to do anything to the Ollama TEMPLATE to get it to terminate or is it the way its invoked that will terminate the sequence when its run inside TxAgent?

[gasvn]

Thank you for adding txagent to ollama! We are using the same tokenizer as the base model llama3.1-8b-instruct. We have a special template here https://huggingface.co/mims-harvard/TxAgent-T1-Llama-3.1-8B/blob/main/tokenizer_config.json

[madhavajay]

@gasvn awesome ill let you know how it goes. Anything I should know about the other Qwen model?

Ill publish the weights to hf and open a PR at some point if it get it working.