FYI: the MOSS model is really good

[oobabooga]

So far, I had been using llama-30b + chansung/alpaca-lora-30b for coding questions, loaded with the monkey patch:

python server.py --model llama-30b-4bit-128g --lora chansung_alpaca-lora-30b --listen  --monkey-patch --chat

But MOSS seems to be better. It gives verbose replies and detailed code. It also seems to be more detailed than oasst-sft-6-llama-30b in my quick tests.

To load it:

python download-model.py fnlp/moss-moon-003-sft
python server.py --model fnlp_moss-moon-003-sft --load-in-8bit  --trust-remote-code --listen --chat

It uses about 20gb vram in 8-bit mode.

[ghost]

Need to try this, been looking for something like this instead of what im using now. ( vicuna )

[eucdee]

ill try and get starcoder and santacoder and CodeCapybara to work :)

MOSS is looking too, @TheBloke can you please quantize it :)

[TheBloke]

ill try and get starcoder and santacoder and CodeCapybara to work :)

MOSS is looking too, @TheBloke can you please quantize it :)

I will have a look. I seem to recall AutoGPTQ added preliminary support for MOSS but then I think there was some issue with it, and I can't immediately recall if the code is meant to be working or not right now. Might have to wait a couple of days because I've got a ton of other testing on my to-do list, but I will when I can!

[gitmylo]

yes, i've tested santacoder as for my pr #1772

[ghost]

is it just me or does it require GPU? Works ok if i enable GPU but if i go CPU only it complains about running scrips ( even tho i have --trust-remote-code in the command line in both cases ). My laptop does not have a real GPU ( intel :( ) so i always test models both ways on my server.