How to deploy a new model by torchchat?

[liu8060]

I want to use torchchat to load the trained model directly from the local. How to change the torchchat/config/data/models.json? Need to change download _ and _ convert in download.py?And, what other documents may need to be changed?

[Jack-Khuu]

Glad to have you try things out

What file format is the local model you're working with?

[liu8060]

two formats
One is a folder format, which includes the model’s weight files, tokenizer files, and other necessary components.
The other format is a PyTorch .pth file, which contains the entire model.

The model's base model is llama3-8b

[Jack-Khuu]

If the model is accessible from huggingface: Here's an example PR of how you can add it

#947 Specifically the known_model_configs and model.json

The other format is a PyTorch .pth file, which contains the entire model.

For this you can can add a known_model_config (based on your params.json with "use_tiktoken"=true for llama3 derivatives) and then point to your pth with --checkpoint-path

For example: python torchchat.py generate --checkpoint-path ~/Desktop/test/consolidated.00.pth --prompt "What should I do in Manhattan Chinatown?" --device mps --num-samples 3

[Jack-Khuu]

One is a folder format, which includes the model’s weight files, tokenizer files, and other necessary components.

Similar to this case? #1040