privateGPT, local, Windows 10 and GPU

[cibernicola]

Hi, I just wanted to ask if anyone has managed to get the combination of privateGPT, local, Windows 10 and GPU working.

I followed the documentation at https://docs.privategpt.dev/#section/Installation-and-Settings, also this: #1195

And the most I've managed to get it to work via CPU, with the extreme slowness that comes with it. I wonder if it's possible and someone has managed to do it or it's just me who can't figure it out.

Thank you.

[kalle07]

i dont know exactly how but i get it

i install all with poweshell (as admin)

[cibernicola]

Have you noticed any difference in speed?
At the moment I've managed to get the normal chat mode to use the GPU, but the document query, not at all (:S), the difference is abysmal.
I don't have any speed benchmarks with a 3090, but I guess it should work relatively well.

[kalle07]

Yes, I have noticed it ...
so on the one hand yes documents are processed very slowly and only the CPU does that, at least all cores, hopefully each core different pages ;)

and yes the normal chat is fast and GPU, but when I use querydoc it is very slow (about 3-4 words/sec)

GPU load ca 20% and one core, LLMchat 80-100%GPU

[piotrmasior]

are you getting around startup something like:
poetry run python -m private_gpt 14:40:11.984 [INFO ] private_gpt.settings.settings_loader - Starting application with profiles=['default'] ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes ggml_init_cublas: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5 llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from C:\Users\pmasi\code\privateGPT\models\mistral-7b-instruct-v0.1.Q4_K_M.gguf (version GGUF V2)

then long list of tensors allocated goes and then final logs like:
llama_model_loader: - type f32: 65 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors llm_load_vocab: special tokens definition check successful ( 259/32000 ). llm_load_print_meta: format = GGUF V2 llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = mostly Q4_K - Medium llm_load_print_meta: model params = 7.24 B llm_load_print_meta: model size = 4.07 GiB (4.83 BPW) llm_load_print_meta: general.name = mistralai_mistral-7b-instruct-v0.1 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_tensors: ggml ctx size = 0.11 MB llm_load_tensors: using CUDA for GPU acceleration llm_load_tensors: mem required = 70.42 MB llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 35/35 layers to GPU llm_load_tensors: VRAM used: 4095.05 MB ............................................................................................... llama_new_context_with_model: n_ctx = 3900 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: offloading v cache to GPU llama_kv_cache_init: offloading k cache to GPU llama_kv_cache_init: VRAM kv self = 487.50 MB llama_new_context_with_model: kv self size = 487.50 MB llama_build_graph: non-view tensors processed: 740/740 llama_new_context_with_model: compute buffer total size = 276.93 MB llama_new_context_with_model: VRAM scratch buffer: 275.37 MB llama_new_context_with_model: total VRAM used: 4857.93 MB (model: 4095.05 MB, context: 762.87 MB) AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 |

if so, it works correctly I guess ?

[Zirgite]

I have GPU running but the speed is very low too. In oobabooga I get 30 tokens per second and here it takes more than a minute for a single answer.

[kalle07]

all seems fine also tensor and VRAM used etc

but the different between using "querydoc" und "llm" is heavy ... so querydoc uses only cpu all cores, llm uses one core and GPU