Trying to run this locally

[ldericher]

This is what happens when I try to run "koboldcpp" on my local machine, using the koboldcpp/LLaMA2-13B-Tiefighter.Q4_K_M in "Adventure Mode". So in Google Colab it works fine, but what the … ?

When I disable BLAS it gets slow, but at least the result is fine. Suspicious, maybe all I need is a full reboot?

[LostRuins]

Some info on your system and launcher params?

[ldericher]

System

• CPU: Intel(R) Core(TM) i5-4670K CPU @ 3.40GHz (4 cores, 1 thread per core)
• GPU: NVIDIA GeForce GTX 1060 6GB (6144 MB GDDR5 VRAM)
• RAM: 16 GB DDR3
• OS: Ubuntu (MATE) 22.04.3 LTS, Linux 6.2.0-37

If this trend continues, my hardware's age is soon its highest metric 🥇

Launch

• Method: koboldcpp.sh without args (Launcher)
• "Use CuBLAS" in "Quick Launch"
• Active Switches: Low VRAM, mmq, ContextShift, Launch Browser
• Options:
  • 20 GPU layers via trial and error – I don't understand the implications beyond "this uses more VRAM"
  • 4 Threads
  • 512 BLAS Batch Size (default)
  • 2048 Context Size (default)
  • Model:

Also, here is the complete stdout of the app having a stroke: https://bpa.st/MRMA

Observations

This happens with both the "Tiefighter" 13B model and another 7B model I've tested

• With "Use No BLAS" everything looks fine, but generation is slow (although faster and better than other tools I've tested so far, nice!)
• With "Use OpenBLAS" the app works as well, but feels just as slow (I suspect there is no GPU offloading there)
• With "CuBLAS" generation is blazingly fast, but completely incoherent (see opening post)

@LostRuins feel free to follow up if further data is needed

~lder

[LostRuins]

Okay, lets try troubleshooting sequentially.

Update to 1.52.1
Leave it on Use Cublas
Set GPU layers to 0
Enable LowVram
Disable Use MMQ

try this, see if it's coherent or not.

[ldericher]

Update to 1.52.1
Leave it on Use Cublas
Set GPU layers to 0
Enable LowVram
Disable Use MMQ

No dice, still gibberish.

Interestingly, I found a configuration which works for me, using CLBlast:

./koboldcpp.sh --skiplauncher --model '/path/to/Downloads/LLaMA2-13B-Tiefighter.Q4_K_M.gguf' --threads 4 --useclblast 0 0 --gpulayers 22

As far as I understand, CuBLAS should be better suited for my NVIDIA GPU, and I'm still willing to help in troubleshooting this weirdness. But in the meantime, I can already enjoy this awesome tool 🤓👍

~lder

[LostRuins]

Hmm okay glad you got it working. Perhaps your nvidia cuda drivers have something wrong with them. But its good you got it working for you.