Cannot get docker image to run, fails when loading model

[digger73]

Hi

Currently on linux, using docker version 23.0.3 and nvidia-docker2 version 2.13.0-1. The docker nvidia works fine, I have used it with stable diffusion.

I have tried a couple of different models, right now trying the alpaca-native-4bit as it is stated that it should work, however I get the following error when doing "docker compose up --build" the following error comes up.

text-generation-webui-text-generation-webui-1 | Found the following quantized model: models/alpaca-native-4bit/alpaca7b-4bit.pt
text-generation-webui-text-generation-webui-1 | Loading model ...
text-generation-webui-text-generation-webui-1 | Traceback (most recent call last):
text-generation-webui-text-generation-webui-1 | File "/app/server.py", line 302, in
text-generation-webui-text-generation-webui-1 | shared.model, shared.tokenizer = load_model(shared.model_name)
text-generation-webui-text-generation-webui-1 | File "/app/modules/models.py", line 103, in load_model
text-generation-webui-text-generation-webui-1 | model = load_quantized(model_name)
text-generation-webui-text-generation-webui-1 | File "/app/modules/GPTQ_loader.py", line 136, in load_quantized
text-generation-webui-text-generation-webui-1 | model = load_quant(str(path_to_model), str(pt_path), shared.args.wbits, shared.args.groupsize, kernel_switch_threshold=threshold)
text-generation-webui-text-generation-webui-1 | File "/app/modules/GPTQ_loader.py", line 63, in _load_quant
text-generation-webui-text-generation-webui-1 | model.load_state_dict(torch.load(checkpoint), strict=False)
text-generation-webui-text-generation-webui-1 | File "/app/venv/lib/python3.10/site-packages/torch/serialization.py", line 815, in load
text-generation-webui-text-generation-webui-1 | return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
text-generation-webui-text-generation-webui-1 | File "/app/venv/lib/python3.10/site-packages/torch/serialization.py", line 1033, in _legacy_load
text-generation-webui-text-generation-webui-1 | magic_number = pickle_module.load(f, **pickle_load_args)
text-generation-webui-text-generation-webui-1 | _pickle.UnpicklingError: invalid load key, 'v'.

Anyone have any ideas how to get this working ?
My env file is(running nvidia 40xx GPU):
TORCH_CUDA_ARCH_LIST=8.9
CLI_ARGS=--model alpaca-native-4bit --wbits 4 --listen --auto-devices
HOST_PORT=7861
CONTAINER_PORT=7861
HOST_API_PORT=5000
CONTAINER_API_PORT=5000
WEBUI_VERSION=HEAD

Appreciate any help.