Any idea how to get GPT4All working?

[davehornik]

Hi, I am not that experienced in this stuff,
but is there any way how to get gpt4all-lora-quantized.bin from this git to work?

(Even tried running it as model, giving errors too.)

[davehornik]

Adding the LoRA gpt4all-lora-quantized.bin to the model...
Traceback (most recent call last):
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\installer_files\env\lib\site-packages\peft\utils\config.py", line 99, in from_pretrained
config_file = hf_hub_download(pretrained_model_name_or_path, CONFIG_NAME)
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\installer_files\env\lib\site-packages\huggingface_hub\utils_validators.py", line 112, in _inner_fn
validate_repo_id(arg_value)
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\installer_files\env\lib\site-packages\huggingface_hub\utils_validators.py", line 157, in validate_repo_id
raise HFValidationError(f"Repo id must be a string, not {type(repo_id)}: '{repo_id}'.")
huggingface_hub.utils._validators.HFValidationError: Repo id must be a string, not <class 'pathlib.WindowsPath'>: 'loras\gpt4all-lora-quantized.bin'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\installer_files\env\lib\site-packages\gradio\routes.py", line 393, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\installer_files\env\lib\site-packages\gradio\blocks.py", line 1108, in process_api
result = await self.call_function(
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\installer_files\env\lib\site-packages\gradio\blocks.py", line 915, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\installer_files\env\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\installer_files\env\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\installer_files\env\lib\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\text-generation-webui\server.py", line 75, in load_lora_wrapper
add_lora_to_model(selected_lora)
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\text-generation-webui\modules\LoRA.py", line 34, in add_lora_to_model
shared.model = PeftModel.from_pretrained(shared.model, Path(f"{shared.args.lora_dir}/{lora_name}"), **params)
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\installer_files\env\lib\site-packages\peft\peft_model.py", line 135, in from_pretrained
config = PEFT_TYPE_TO_CONFIG_MAPPING[PeftConfig.from_pretrained(model_id).peft_type].from_pretrained(model_id)
File "C:\Users\david\Desktop\GPT Gradio\chatbot webui\installer_files\env\lib\site-packages\peft\utils\config.py", line 101, in from_pretrained
raise ValueError(f"Can't find config.json at '{pretrained_model_name_or_path}'")
ValueError: Can't find config.json at 'loras\gpt4all-lora-quantized.bin'

[BetaDoggo]

I think that's the quantized lora merged ggml version rather than the lora itself. The lora version that should be usable in the webui can be found here: https://huggingface.co/nomic-ai/gpt4all-lora.

We really need a standardized naming scheme for quantized models. I've seen so many 4bit models on huggingface with no marking to indicate what format they actually are.

[oobabooga]

The naming convention that I am following for ggml models is that the file name must start with ggml and end in .bin. The folder name can be any name. So for this model I have this folder structure:

models
  \_ gpt4all-lora-quantized
        \_ ggml-gpt4all-lora-quantized.bin

Trying to run it with

python server.py --model gpt4all-lora-quantized

generates the error

llama_model_load: loading model from 'models/gpt4all-lora-quantized/ggml-gpt4all-lora-quantized.bin' - please wait ...
llama_model_load: invalid model file 'models/gpt4all-lora-quantized/ggml-gpt4all-lora-quantized.bin' (too old, regenerate your model files or convert them with convert-unversioned-ggml-to-ggml.py!)
llama_init_from_file: failed to load model

[tpsjr7]

These steps worked for me, but instead of using that combined gpt4all-lora-quantized.bin model, I used the seperated lora and llama7b like this:

python download-model.py nomic-ai/gpt4all-lora
python download-model.py zpn/llama-7b
python server.py  --chat --model llama-7b --lora gpt4all-lora

You can also add on the --load-in-8bit flag to require less GPU vram, but on my rtx 3090 it generates at about 1/3 the speed, and the responses seem a little dumber ( after only a cursory glance, maybe there's some settings to tweak to make it better).

I noticed in their README they linked to the actual lora by itself from here https://huggingface.co/nomic-ai/gpt4all-lora
Then if you look at this file https://huggingface.co/nomic-ai/gpt4all-lora/blob/main/adapter_config.json you can see that it references "base_model_name_or_path": "zpn/llama-7b".