How to run Safetensor Models?

[domi-bue]

Hello,

I recently found this project and want to run LLaMA-33B-HF. I saw that you need the vLLM Backend for that.

I have the following file structure:

.
├── LLaMA-33B-HF
│   ├── config.json
│   ├── generation_config.json
│   ├── pytorch_model-00001-of-00007.bin
│   ├── pytorch_model-00002-of-00007.bin
│   ├── pytorch_model-00003-of-00007.bin
│   ├── pytorch_model-00004-of-00007.bin
│   ├── pytorch_model-00005-of-00007.bin
│   ├── pytorch_model-00006-of-00007.bin
│   ├── pytorch_model-00007-of-00007.bin
│   ├── pytorch_model.bin.index.json
│   ├── README.md
│   ├── special_tokens_map.json
│   ├── tokenizer_config.json
│   └── tokenizer.model
└── LLaMA-33B-HF.yaml

And this is the Content of the LLaMA-33B-HF.yaml file:

name: llama-33b-hf
backend: vllm
context_size: 2048
f16: true
gpu_layers: 90
mmap: true
parameters:
  model: LLaMA-33B-HF
  temperature: 0.7
  top_k: 40
  top_p: 0.95
  seed: -1
template:
  chat: &template |
    {{.Input}}
    ASSISTANT:
  completion: *template
usage: |
      To use this model, interact with the API (in another terminal) with curl for instance:
      curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
          "model": "llama-33b-hf",
          "messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
      }'

But whenever I try to run the model, I get the following Error:

INF Loading model 'LLaMA-33B-HF' with backend vllm
ERR Server error error="could not load model (no success): Unexpected err=OSError(\"LLaMA-33B-HF is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'\\nIf this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`\"), type(err)=<class 'OSError'>"

[maxi1134]

Did you found how to do this?

[domi-bue]

Not yet. I tried some things and also checked if the folder exists in the container, which it does, but it still shows the error

[eslowney]

Same issue, LocalAI will only run gguf models locally. When I try to load a safetensors folder, it tries to connect to huggingface.co.

Is there a way to load a local model of this type?

Edit: to be clear, I don't get the same HTTP error as above, as I'm offline, so I get an Cannot Establish connection error.