How to get llama to use 100% GPU utilization

[SoftologyPro]

No matter what model I use the GPU utilization seems to fluctuate around 50%.

Example command
llama-server -m models\gemma4-26b.gguf -ngl 99 --mmproj models\mmproj-BF16.gguf --no-mmproj-offload --port 8080

What am I missing to get 100% GPU utilization?

Thanks.

[SoftologyPro]

Here is an example task manager when running on a 4090.

[Danik-droid]

This means there's a bottleneck somewhere. You'll only see high GPU usage during processing. During generation, the card's memory subsystem is the bottleneck (memory bandwidth). The GPU core has to wait for data. There's not much you can do about this, for example, you can try overclocking the memory (if you can) or using techniques that improve memory efficiency (like Flash Attention).

In your case, it seems to me that the card is running out of VRAM and is holding data in RAM - this slows it down significantly (because the GPU core has to wait even longer for data from RAM). Try a smaller model (or more aggressive Quantizations).

[SoftologyPro]

It does not seem to be VRAM related as this is with a smaller model. Still around 33% GPU utilization this time.

All other ML/AI systems I run use all the GPU utilization when running.

How do I install flash attention into llama. There does not seem to be a venv I can add to.

Googling this issue suggested that adding the -ngl 99 to the commandline would fix it, but it made no difference in my case.

Same thing running on a 5090 is better, but still only uses around 80% GPU utuilization. On a laptop with a 4080 it uses around 60%. So not just this one PC with the issue.

[Danik-droid]

add "-fa on" to your command (I'm not sure if it's necessary because it's been enabled by default for some time now - I didn't know about it before).

That low GPU usage is strange... I'd try a different version of llamacpp, and out of curiosity, Vulkan (if you're using Cuda). You have this with different cards... Interesting. Maybe there's something messed up in the system... Hard to say. Maybe run some Linux on a flash drive and see how Vulkan runs there.

I don't use Nvidia cards, so it's hard for me to say anything specific.

[SoftologyPro]

-fa on made no difference.

See if any other Windows users report this issue. I am not doing anything different. Basic 1 line setup, download models, run.

The same PCs run hundreds of other AI systems without this under utilized GPU symptom. They all max out at 100%.
https://softology.pro/voc.htm

[jaycenornin]

You're looking at the wrong graphs.

Correct graphs (make sure Compute 0 is selected)

Your specific GPU or drivers or backend may use a different graph for how llama.cpp is using it, but for my GPU with Vulkan backend it's "Compute 0".

Task Manager showing llama.cpp as not using "3D" compute is interesting to me. But I do not think it means llama.cpp is leaving performance on the table. If anything, it means your other AI tools are not reporting their GPU utilization to Windows correctly or are wasting cycles on compute paths that don't contribute to LLM performance.

Also, my GPU utilization is spiky because this model in this screenshot is partially offloaded to CPU.

[jaycenornin]

Same model (Gemma 26B Q4KM) on my 3070, also offloading a bunch to CPU.
With the Cuda backend the utilization shows on the Cuda graph.

[SoftologyPro]

Radeon GPU and Vulkan backend may not be a valid comparison.
The 3090, 4080, 4090 and 5090 do not have the CUDA or Compute 0 option in the Task Manager dropdowns.

Can the lower utilization % be trusted for llama.cpp? If so, that also only shows a lower % when llama.cpp is running and not 100%.

The "3D" stat is reliable for every other AI/Python script I run and always shows 100% when the GPU is fully utilized (not always fully 100% but definitely fluctuating between 90% and 100%) for CUDA/torch related activity. Is there some other reliable utility or tool to show true GPU usage if 3D is not to be trusted? On your 3070 it only shows 9% utilization when running?

[jaycenornin]

I don't know why your Task Manager doesn't show a Cuda graph for your GPU. The available graphs are determined by the hardware and the driver.
Go through each of the available graphs one by one while your llama.cpp is processing and you should find one that is chewing up cycles. That's the one to monitor.

If you do not find any graphs showing the expected utilization, then you definitely have a bottleneck somewhere.

Are you using the pre-built llama.cpp binaries? Are you using the Cuda binaries? Did you download both the llama.cpp Cuda build and the Cuda libraries? (this wouldn't change the available graphs in task manager, but it would ensure you're actually using the Cuda backend)
Pick Cuda 12 or Cuda 13, but you need both the llama.cpp and the Cuda DLLs

If you're using the Cuda backend, you should see this notice when starting llama-cli or llama-server:

[SoftologyPro]

None of the other Task Manager dropdown options show any activity. If this was Task Manager then it would be the same for the hundreds of other AI systems I run on this PC. Only llama.cpp shows this lower activity issue.

OK, starting from scratch.

Installed with winget then run

winget uninstall llama.cpp
winget install llama.cpp
llama-server -m models\qwen3.6-27b.gguf -fa on -ngl 99 --mmproj models\mmproj-qwen3.6-27b.gguf --no-mmproj-offload --port 8080

The backend stats at startup show this.

load_backend: loaded RPC backend from C:\Users\Jason\AppData\Local\Microsoft\WinGet\Packages\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\ggml-rpc.dll
load_backend: loaded Vulkan backend from C:\Users\Jason\AppData\Local\Microsoft\WinGet\Packages\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\Users\Jason\AppData\Local\Microsoft\WinGet\Packages\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\ggml-cpu-alderlake.dll

So it is not using CUDA at all, but Vulkan? Can I force it to use CUDA?
This is on Windows 11 with CUDA 12.8.

Full stats of install and run...

D:\Tests\llama.cpp>winget uninstall llama.cpp
Found llama.cpp [ggml.llamacpp]
Starting package uninstall...
Successfully uninstalled

D:\Tests\llama.cpp>winget install llama.cpp
Found llama.cpp [ggml.llamacpp] Version b9070
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
This package requires the following dependencies:
  - Packages
      Microsoft.VCRedist.2015+.x64
Successfully verified installer hash
Extracting archive...
Successfully extracted archive
Starting package install...
Command line alias added: "llama-batched-bench"
Command line alias added: "llama-bench"
Command line alias added: "llama-cli"
Command line alias added: "llama-gguf-split"
Command line alias added: "llama-imatrix"
Command line alias added: "llama-mtmd-cli"
Command line alias added: "llama-perplexity"
Command line alias added: "llama-quantize"
Command line alias added: "llama-server"
Command line alias added: "llama-tokenize"
Command line alias added: "llama-tts"
Successfully installed

D:\Tests\llama.cpp>llama-server -m models\qwen3.6-27b.gguf -fa on -ngl 99 --mmproj models\mmproj-qwen3.6-27b.gguf --no-mmproj-offload --port 8080
load_backend: loaded RPC backend from C:\Users\Jason\AppData\Local\Microsoft\WinGet\Packages\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\ggml-rpc.dll
load_backend: loaded Vulkan backend from C:\Users\Jason\AppData\Local\Microsoft\WinGet\Packages\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\Users\Jason\AppData\Local\Microsoft\WinGet\Packages\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\ggml-cpu-alderlake.dll
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build_info: b9070-f3e8d149c
system_info: n_threads = 24 (n_threads_batch = 24) / 32 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
Running without SSL
init: using 31 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model 'models\qwen3.6-27b.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
common_params_fit_impl: getting device memory data for initial parameters:
common_memory_breakdown_print: | memory breakdown [MiB] | total    free     self   model   context   compute    unaccounted |
common_memory_breakdown_print: |   - Vulkan0 (RTX 4090) | 24356 = 22981 + (33156 = 15345 +   16982 +     828) +      -31781 |
common_memory_breakdown_print: |   - Host               |                   1214 =   682 +       0 +     532                |
common_params_fit_impl: projected to use 33156 MiB of device memory vs. 22981 MiB of free device memory
common_params_fit_impl: cannot meet free memory target of 1024 MiB, need to reduce device memory by 11198 MiB
common_memory_breakdown_print: | memory breakdown [MiB] | total    free     self   model   context   compute    unaccounted |
common_memory_breakdown_print: |   - Vulkan0 (RTX 4090) | 24356 = 22981 + (16867 = 15345 +     854 +     667) +      -15492 |
common_memory_breakdown_print: |   - Host               |                    710 =   682 +       0 +      28                |
common_params_fit_impl: context size reduced from 262144 to 84736 -> need 11198 MiB less memory in total
common_params_fit_impl: entire model can be fit by reducing context
common_fit_params: successfully fit params to free device memory
common_fit_params: fitting params to free memory took 0.41 seconds
llama_model_loader: loaded meta data with 51 key-value pairs and 851 tensors from models\qwen3.6-27b.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen35
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                     general.sampling.top_k i32              = 20
llama_model_loader: - kv   3:                     general.sampling.top_p f32              = 0.950000
llama_model_loader: - kv   4:                      general.sampling.temp f32              = 1.000000
llama_model_loader: - kv   5:                               general.name str              = Qwen3.6-27B
llama_model_loader: - kv   6:                           general.basename str              = Qwen3.6-27B
llama_model_loader: - kv   7:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   8:                         general.size_label str              = 27B
llama_model_loader: - kv   9:                            general.license str              = apache-2.0
llama_model_loader: - kv  10:                       general.license.link str              = https://huggingface.co/Qwen/Qwen3.6-2...
llama_model_loader: - kv  11:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv  12:                   general.base_model.count u32              = 1
llama_model_loader: - kv  13:                  general.base_model.0.name str              = Qwen3.6 27B
llama_model_loader: - kv  14:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  15:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen3.6-27B
llama_model_loader: - kv  16:                               general.tags arr[str,2]       = ["unsloth", "image-text-to-text"]
llama_model_loader: - kv  17:                         qwen35.block_count u32              = 64
llama_model_loader: - kv  18:                      qwen35.context_length u32              = 262144
llama_model_loader: - kv  19:                    qwen35.embedding_length u32              = 5120
llama_model_loader: - kv  20:                 qwen35.feed_forward_length u32              = 17408
llama_model_loader: - kv  21:                qwen35.attention.head_count u32              = 24
llama_model_loader: - kv  22:             qwen35.attention.head_count_kv u32              = 4
llama_model_loader: - kv  23:             qwen35.rope.dimension_sections arr[i32,4]       = [11, 11, 10, 0]
llama_model_loader: - kv  24:                      qwen35.rope.freq_base f32              = 10000000.000000
llama_model_loader: - kv  25:    qwen35.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  26:                qwen35.attention.key_length u32              = 256
llama_model_loader: - kv  27:              qwen35.attention.value_length u32              = 256
llama_model_loader: - kv  28:                     qwen35.ssm.conv_kernel u32              = 4
llama_model_loader: - kv  29:                      qwen35.ssm.state_size u32              = 128
llama_model_loader: - kv  30:                     qwen35.ssm.group_count u32              = 16
llama_model_loader: - kv  31:                  qwen35.ssm.time_step_rank u32              = 48
llama_model_loader: - kv  32:                      qwen35.ssm.inner_size u32              = 6144
llama_model_loader: - kv  33:             qwen35.full_attention_interval u32              = 4
llama_model_loader: - kv  34:                qwen35.rope.dimension_count u32              = 64
llama_model_loader: - kv  35:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  36:                         tokenizer.ggml.pre str              = qwen35
llama_model_loader: - kv  37:                      tokenizer.ggml.tokens arr[str,248320]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  38:                  tokenizer.ggml.token_type arr[i32,248320]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  39:                      tokenizer.ggml.merges arr[str,247587]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  40:                tokenizer.ggml.eos_token_id u32              = 248046
llama_model_loader: - kv  41:            tokenizer.ggml.padding_token_id u32              = 248055
llama_model_loader: - kv  42:                tokenizer.ggml.bos_token_id u32              = 248044
llama_model_loader: - kv  43:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  44:                    tokenizer.chat_template str              = {%- set image_count = namespace(value...
llama_model_loader: - kv  45:               general.quantization_version u32              = 2
llama_model_loader: - kv  46:                          general.file_type u32              = 15
llama_model_loader: - kv  47:                      quantize.imatrix.file str              = Qwen3.6-27B-GGUF/imatrix_unsloth.gguf
llama_model_loader: - kv  48:                   quantize.imatrix.dataset str              = unsloth_calibration_Qwen3.6-27B.txt
llama_model_loader: - kv  49:             quantize.imatrix.entries_count u32              = 496
llama_model_loader: - kv  50:              quantize.imatrix.chunks_count u32              = 76
llama_model_loader: - type  f32:  449 tensors
llama_model_loader: - type q4_K:  289 tensors
llama_model_loader: - type q5_K:   48 tensors
llama_model_loader: - type q6_K:   65 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 15.65 GiB (5.00 BPW)
llama_prepare_model_devices: using device Vulkan0 (NVIDIA GeForce RTX 4090) (0000:01:00.0) - 23580 MiB free
load: 0 unused tokens
load: printing all EOG tokens:
load:   - 248044 ('<|endoftext|>')
load:   - 248046 ('<|im_end|>')
load:   - 248063 ('<|fim_pad|>')
load:   - 248064 ('<|repo_name|>')
load:   - 248065 ('<|file_sep|>')
load: special tokens cache size = 33
load: token to piece cache size = 1.7581 MB
print_info: arch                  = qwen35
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 262144
print_info: n_embd                = 5120
print_info: n_embd_inp            = 5120
print_info: n_layer               = 64
print_info: n_head                = 24
print_info: n_head_kv             = 4
print_info: n_rot                 = 64
print_info: n_swa                 = 0
print_info: is_swa_any            = 0
print_info: n_embd_head_k         = 256
print_info: n_embd_head_v         = 256
print_info: n_gqa                 = 6
print_info: n_embd_k_gqa          = 1024
print_info: n_embd_v_gqa          = 1024
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 0.0e+00
print_info: f_attn_value_scale    = 0.0000
print_info: n_ff                  = 17408
print_info: n_expert              = 0
print_info: n_expert_used         = 0
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 1
print_info: pooling type          = -1
print_info: rope type             = 40
print_info: rope scaling          = linear
print_info: freq_base_train       = 10000000.0
print_info: freq_scale_train      = 1
print_info: n_ctx_orig_yarn       = 262144
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: mrope sections        = [11, 11, 10, 0]
print_info: ssm_d_conv            = 4
print_info: ssm_d_inner           = 6144
print_info: ssm_d_state           = 128
print_info: ssm_dt_rank           = 48
print_info: ssm_n_group           = 16
print_info: ssm_dt_b_c_rms        = 0
print_info: model type            = 27B
print_info: model params          = 26.90 B
print_info: general.name          = Qwen3.6-27B
print_info: vocab type            = BPE
print_info: n_vocab               = 248320
print_info: n_merges              = 247587
print_info: BOS token             = 248044 '<|endoftext|>'
print_info: EOS token             = 248046 '<|im_end|>'
print_info: EOT token             = 248046 '<|im_end|>'
print_info: PAD token             = 248055 '<|vision_pad|>'
print_info: LF token              = 198 'Ċ'
print_info: FIM PRE token         = 248060 '<|fim_prefix|>'
print_info: FIM SUF token         = 248062 '<|fim_suffix|>'
print_info: FIM MID token         = 248061 '<|fim_middle|>'
print_info: FIM PAD token         = 248063 '<|fim_pad|>'
print_info: FIM REP token         = 248064 '<|repo_name|>'
print_info: FIM SEP token         = 248065 '<|file_sep|>'
print_info: EOG token             = 248044 '<|endoftext|>'
print_info: EOG token             = 248046 '<|im_end|>'
print_info: EOG token             = 248063 '<|fim_pad|>'
print_info: EOG token             = 248064 '<|repo_name|>'
print_info: EOG token             = 248065 '<|file_sep|>'
print_info: max token length      = 256
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
load_tensors: offloading output layer to GPU
load_tensors: offloading 63 repeating layers to GPU
load_tensors: offloaded 65/65 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   682.03 MiB
load_tensors:      Vulkan0 model buffer size = 15345.66 MiB
............................................................................................
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|im_end|> logit bias = -inf
common_init_result: added <|fim_pad|> logit bias = -inf
common_init_result: added <|repo_name|> logit bias = -inf
common_init_result: added <|file_sep|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max     = 4
llama_context: n_ctx         = 84736
llama_context: n_ctx_seq     = 84736
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = true
llama_context: freq_base     = 10000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (84736) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
�[0mllama_context: Vulkan_Host  output buffer size =     3.79 MiB
llama_kv_cache:    Vulkan0 KV buffer size =  5296.00 MiB
llama_kv_cache: size = 5296.00 MiB ( 84736 cells,  16 layers,  4/1 seqs), K (f16): 2648.00 MiB, V (f16): 2648.00 MiB
llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
llama_memory_recurrent:    Vulkan0 RS buffer size =   598.50 MiB
llama_memory_recurrent: size =  598.50 MiB (     4 cells,  64 layers,  4 seqs), R (f32):   22.50 MiB, S (f32):  576.00 MiB
sched_reserve: reserving ...
sched_reserve: resolving fused Gated Delta Net support:
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve:    Vulkan0 compute buffer size =   793.27 MiB
sched_reserve: Vulkan_Host compute buffer size =   185.52 MiB
sched_reserve: graph nodes  = 3657
sched_reserve: graph splits = 2
sched_reserve: reserve took 19.07 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
�[0mclip_model_loader: model name:   Qwen3.6-27B
clip_model_loader: description:
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    334
clip_model_loader: n_kv:         33

clip_model_loader: has vision encoder
clip_ctx: CLIP using CPU backend
load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
�[0mload_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
�[0mload_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

�[0mload_hparams: projector:          qwen3vl_merger
load_hparams: n_embd:             1152
load_hparams: n_head:             16
load_hparams: n_ff:               4304
load_hparams: n_layer:            27
load_hparams: ffn_op:             gelu
load_hparams: projection_dim:     5120

--- vision hparams ---
load_hparams: image_size:         768
load_hparams: patch_size:         16
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   0
load_hparams: n_merge:            2
load_hparams: n_wa_pattern: 0
load_hparams: image_min_pixels:   8192
load_hparams: image_max_pixels:   4194304

load_hparams: model size:         887.99 MiB
load_hparams: metadata size:      0.12 MiB
warmup: warmup with image size = 1472 x 1472
alloc_compute_meta:        CPU compute buffer size =   248.10 MiB
alloc_compute_meta: graph splits = 1, nodes = 823
warmup: flash attention is enabled
srv    load_model: loaded multimodal model, 'models\mmproj-qwen3.6-27b.gguf'
srv    load_model: initializing slots, n_slots = 4
common_context_can_seq_rm: the target context does not support partial sequence removal
�[0msrv    load_model: speculative decoding will use checkpoints
�[0mno implementations specified for speculative decoding
�[0mslot   load_model: id  0 | task -1 | new slot, n_ctx = 84736
no implementations specified for speculative decoding
�[0mslot   load_model: id  1 | task -1 | new slot, n_ctx = 84736
no implementations specified for speculative decoding
�[0mslot   load_model: id  2 | task -1 | new slot, n_ctx = 84736
no implementations specified for speculative decoding
�[0mslot   load_model: id  3 | task -1 | new slot, n_ctx = 84736
srv    load_model: prompt cache is enabled, size limit: 8192 MiB
�[0msrv    load_model: use `--cache-ram 0` to disable the prompt cache
�[0msrv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
�[0msrv          init: init: idle slots will be saved to prompt cache and cleared upon starting a new task
init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
<think>
'
srv          init: init: chat template, thinking = 1
main: model loaded
main: server is listening on http://127.0.0.1:8080
main: starting the main loop...
srv  update_slots: all slots are idle
srv  log_server_r: done request: GET / 127.0.0.1 200
srv  log_server_r: done request: GET /bundle.css 127.0.0.1 200
srv  log_server_r: done request: GET /bundle.js 127.0.0.1 200
srv  log_server_r: done request: HEAD /cors-proxy 127.0.0.1 404
srv  log_server_r: done request: GET /tools 127.0.0.1 404
srv  log_server_r: done request: GET /tools 127.0.0.1 404
srv  params_from_: Chat format: peg-native
slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
srv  get_availabl: updating prompt cache
�[0msrv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
�[0msrv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 84736 tokens, 8589934592 est)
�[0msrv  get_availabl: prompt cache update took 0.01 ms
�[0mslot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 84736, n_keep = 0, task.n_tokens = 41
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 37, batch.n_tokens = 37, progress = 0.902439
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
slot update_slots: id  3 | task 0 | n_tokens = 37, memory_seq_rm [37, end)
slot init_sampler: id  3 | task 0 | init sampler, took 0.01 ms, tokens: text = 41, total = 41
slot update_slots: id  3 | task 0 | prompt processing done, n_tokens = 41, batch.n_tokens = 4
slot print_timing: id  3 | task 0 |
prompt eval time =     356.27 ms /    41 tokens (    8.69 ms per token,   115.08 tokens per second)
       eval time =  265184.29 ms / 11551 tokens (   22.96 ms per token,    43.56 tokens per second)
      total time =  265540.56 ms / 11592 tokens
slot      release: id  3 | task 0 | stop processing: n_tokens = 11591, truncated = 0
srv  update_slots: all slots are idle

[SoftologyPro]

OK, I made some progress...

I downloaded https://github.com/ggml-org/llama.cpp/releases/download/b9084/llama-b9084-bin-win-cuda-12.4-x64.zip and extracted it.
I copied the models I had under a models subfolder and started the UI with the same command...
llama-server -m models\qwen3.6-27b.gguf -fa on -ngl 99 --mmproj models\mmproj-qwen3.6-27b.gguf --no-mmproj-offload --port 8080
and it now uses 100% GPU while running.

Can I get this same behavior using the

winget uninstall llama.cpp

install?

If not, then maybe the install readme instructions should specify that Windows users should not use the winget install for best performance?

[SoftologyPro]

If I do use the zip is there any way to easily determine the latest zip name? This is so the download can be automated in a batch file and make sure the latest version is being downloaded.

[jaycenornin]

I don't use winget for my llama.cpp so can't help you with that.
And I wish I could tell you it was in the documentation but, well, *nix nerds gonna not document where they gonna not document. I suppose we're fortunate the docs are as good as they are.

The manual downloads all have the release version number in the filename here:

[SoftologyPro]

Having a different name each release just means I need to update my install batch file from time to time rather than just having a fixed name zip file to download that is always the latest. If there could be a llama-latest-win.zip that would be easier, but I will check and update my installer every so often for a new release

Anyway, it works now with 100% GPU using the zip install. If any other Windows user has the same issue, do not use winget for the install, use the zip file download/extract method.

I have had some good vibe coding results with llama.cpp
https://softologyblog.wordpress.com/2026/04/18/local-vibe-coding/

[snailwei]

The key is kv-unified should be explicitly specified.

    --parallel 16 \
    --kv-unified \