Feature Request: Support NVIDIA Nemotron 3 Nano

[msievers]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I wanted to benchmark the newly released NVIDIA Nemotron 3 Nano model but it seems not be supported by ik_llama.cpp

I'm using the following version of ik_llama.cpp

./build/bin/llama-server --version
version: 4072 (21fc9322)
built with gcc (GCC) 15.2.0 for x86_64-pc-linux-gnu

and this quantisation from Unsloth

https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF?show_file_info=Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf

Unsloth guide is located at

https://docs.unsloth.ai/models/nemotron-3

When trying the run llama-bench it returns with an error saying

./build/bin/llama-bench -v -m ~/Downloads/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf

... verbose output shortened ...

llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'nemotron_h_moe'
llama_load_model_from_file: failed to load model

To be honest, I have no idea how much effort is required to support this model, and if the effort is too high, it might not make sense to invest work here. On the other hand, if it turns out to be easy, it would be interesting to be able to compare the performance with llama.cpp CPU-only.

Motivation

NVIDIA Nemotron 3 Nano has a certain amount of potential, at least on paper, and is claimed to be noticeably faster on the GPU than comparable MoE models such as Qwen3-30B-A3B. At least, that is what NVIDIA claims. This would make it an interesting option as a fast MoE model for CPU-only or mixed CPU/GPU inference on weak hardware.

Possible Implementation

No response

[ikawrakow]

I don't have this model downloaded and am running a bit low on disk space, so it would be useful for me to get an indication how useful the model is in practice before deciding which model to delete so I can download this one.

Have you tried using it with mainline llama.cpp? If so, what kind of performance do we get there and how does it compare to Qwen3-30B-A3B and GPT-OSS-20B?

[msievers]

Mainline llama.cpp gives the following for me with CPU-only inference:

~/github/ggml-org/llama.cpp$ ./build/bin/llama-bench -t 8 -m ~/Downloads/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf 
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| nemotron_h_moe 31B.A3.5B Q4_K - Medium |  21.26 GiB |    31.58 B | CPU        |       8 |           pp512 |         61.17 ± 0.27 |
| nemotron_h_moe 31B.A3.5B Q4_K - Medium |  21.26 GiB |    31.58 B | CPU        |       8 |           tg128 |         19.67 ± 0.60 |

build: f716588e6 (7473)

which is in fact slower than Qwen3-30B-A3B (maybe due to its bigger size)

~/github/ggml-org/llama.cpp$ ./build/bin/llama-bench -t 8 -m ~/Downloads/Qwen3-30B-A3B-Instruct-2507-UD-Q4_K_XL.gguf 
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen3moe 30B.A3B Q4_K - Medium |  16.47 GiB |    30.53 B | CPU        |       8 |           pp512 |         86.26 ± 0.47 |
| qwen3moe 30B.A3B Q4_K - Medium |  16.47 GiB |    30.53 B | CPU        |       8 |           tg128 |         20.91 ± 0.17 |

build: f716588e6 (7473)

The only reason I'm asking is because people on Reddit report it could be faster than Qwen3 or gpt-oss (probably on GPU). The graphic I linked from the Unsloth guide talks about throughput 1.5x - 3x faster when compared to Qwen3 or gpt-oss, but that's their numbers, so I don't know for sure, especially because I can only do CPU-inference due to the lack of an appropriate graphics card.

[ikawrakow]

Aha, thanks. That confirms my concept that llama.cpp has a very inefficient mamba implementation. I haven't done anything mamba-related here, so a port reusing mainline's implementation will also be quite slow. My guess is that it would be quite a bit slower than GPT-OSS-20B and Qwen3-30B-A3B on CPU (unless I put in the effort to optimize the mamba part).

Here is what I get with todays's llama.cpp on my Ryzen-7950X CPU

model	size	params	backend	threads	test	t/s
qwen3moe 30B.A3B Q4_K - Medium	17.28 GiB	30.53 B	CPU	16	pp512	210.37 ± 0.44
qwen3moe 30B.A3B Q4_K - Medium	17.28 GiB	30.53 B	CPU	16	tg128	27.20 ± 0.38
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CPU	16	pp512	148.73 ± 0.53
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CPU	16	tg128	21.49 ± 0.01

And here is what I get with ik_llama.cpp

model	size	params	backend	threads	test	t/s
qwen3moe 30B.A3B Q4_K - Medium	17.53 GiB	31.01 B	CPU	16	pp512	489.60 ± 3.48
qwen3moe 30B.A3B Q4_K - Medium	17.53 GiB	31.01 B	CPU	16	tg128	28.34 ± 0.00
gpt-oss ?B MXFP4 - 4.25 bpw	11.62 GiB	21.27 B	CPU	16	pp512	564.36 ± 1.75
gpt-oss ?B MXFP4 - 4.25 bpw	11.62 GiB	21.27 B	CPU	16	tg128	22.08 ± 0.00

Based on this, my CPU is 2.44X better than yours for PP, and 1.3X for TG. Using that to scale down the ik_llama.cpp performance on my CPU to your CPU, you should be getting for PP ~200 t/s for Qwen3-30B-A3B and ~230 t/s for OSS-20B-MXFP4 (maybe NVIDIA got confused and the 3.3X is actually the other way around? 😄 ). For TG it would be ~21.8 t/s for Qwen3-30B-A3B and ~17 t/s for GPT-OSS-20B.

[msievers]

That interpolation is indeed roughly consistent for Qwen3-30B-A3B for my CPU. Specifically, I get 192 t/s pp512 and 25.9 t/s tg128.

But anyway ... If you feel that Nemotron 3 Lite support is currently too time-consuming or offers little benefit, please feel free to close this issue. I just wanted to raise the topic in general, because I was wondering, why llama-bench refused to run with it 🙂

[chatchatgpt]

I guess the point of these kinds of models is that they maintain a certain prefill and decode speed even with long contexts, right?
These are my prefill and decode speeds with a 47k long context (mainline llama.cpp)
GPU is a 5070, CPU is EPYC 9654, memory DDR5-4800 384 GB, GCC 13.3, NVIDIA driver 580.95
pp speed: 653 tokens/s
tg speed: 55.2tokens/s

Details

bill@bill-Super-Server ~/l/b/bin (master)> ./llama-server -m /home/bill/llmmodels/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q5_K_XL.gguf -c 50000 -ngl 99 -ncmoe 99 --threads 40 -fa on -ctk q4_0 -ctv q4_0 --no-mmap
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA device:
Device 0: NVIDIA GeForce RTX 5070, compute capability 12.0, VMM: yes
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 7478 (0a271d82b) with GNU 13.3.0 for Linux x86_64
system info: n_threads = 40, n_threads_batch = 40, total_threads = 96

system_info: n_threads = 40 (n_threads_batch = 40) / 96 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

init: using 95 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model '/home/bill/llmmodels/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q5_K_XL.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected to use 3040 MiB of device memory vs. 11769 MiB of free device memory
llama_params_fit_impl: will leave 7272 >= 1024 MiB of free device memory, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.47 seconds
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 5070) (0000:01:00.0) - 10521 MiB free
llama_model_loader: loaded meta data with 53 key-value pairs and 401 tensors from /home/bill/llmmodels/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-UD-Q5_K_XL.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = nemotron_h_moe
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.sampling.top_p f32 = 1.000000
llama_model_loader: - kv 3: general.sampling.temp f32 = 1.000000
llama_model_loader: - kv 4: general.name str = Nemotron-3-Nano-30B-A3B
llama_model_loader: - kv 5: general.basename str = Nemotron-3-Nano-30B-A3B
llama_model_loader: - kv 6: general.quantized_by str = Unsloth
llama_model_loader: - kv 7: general.size_label str = 30B-A3B
llama_model_loader: - kv 8: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 9: nemotron_h_moe.block_count u32 = 52
llama_model_loader: - kv 10: nemotron_h_moe.context_length u32 = 1048576
llama_model_loader: - kv 11: nemotron_h_moe.embedding_length u32 = 2688
llama_model_loader: - kv 12: nemotron_h_moe.feed_forward_length arr[i32,52] = [0, 1856, 0, 1856, 0, 0, 1856, 0, 185...
llama_model_loader: - kv 13: nemotron_h_moe.attention.head_count u32 = 32
llama_model_loader: - kv 14: nemotron_h_moe.attention.head_count_kv arr[i32,52] = [0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, ...
llama_model_loader: - kv 15: nemotron_h_moe.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 16: nemotron_h_moe.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 17: nemotron_h_moe.attention.layer_norm_epsilon f32 = 0.000010
llama_model_loader: - kv 18: nemotron_h_moe.expert_used_count u32 = 6
llama_model_loader: - kv 19: nemotron_h_moe.expert_group_count u32 = 1
llama_model_loader: - kv 20: nemotron_h_moe.expert_group_used_count u32 = 1
llama_model_loader: - kv 21: nemotron_h_moe.vocab_size u32 = 131072
llama_model_loader: - kv 22: nemotron_h_moe.rope.dimension_count u32 = 84
llama_model_loader: - kv 23: nemotron_h_moe.ssm.conv_kernel u32 = 4
llama_model_loader: - kv 24: nemotron_h_moe.ssm.state_size u32 = 128
llama_model_loader: - kv 25: nemotron_h_moe.ssm.group_count u32 = 8
llama_model_loader: - kv 26: nemotron_h_moe.ssm.inner_size u32 = 4096
llama_model_loader: - kv 27: nemotron_h_moe.ssm.time_step_rank u32 = 64
llama_model_loader: - kv 28: nemotron_h_moe.rope.scaling.finetuned bool = false
llama_model_loader: - kv 29: nemotron_h_moe.attention.key_length u32 = 128
llama_model_loader: - kv 30: nemotron_h_moe.attention.value_length u32 = 128
llama_model_loader: - kv 31: nemotron_h_moe.expert_feed_forward_length u32 = 1856
llama_model_loader: - kv 32: nemotron_h_moe.expert_shared_feed_forward_length u32 = 3712
llama_model_loader: - kv 33: nemotron_h_moe.expert_count u32 = 128
llama_model_loader: - kv 34: nemotron_h_moe.expert_shared_count u32 = 1
llama_model_loader: - kv 35: nemotron_h_moe.expert_weights_norm bool = true
llama_model_loader: - kv 36: nemotron_h_moe.expert_weights_scale f32 = 2.500000
llama_model_loader: - kv 37: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 38: tokenizer.ggml.pre str = pixtral
llama_model_loader: - kv 39: tokenizer.ggml.tokens arr[str,131072] = ["", "", "", "[INST]", "[...
llama_model_loader: - kv 40: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv 41: tokenizer.ggml.merges arr[str,269443] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ �...
llama_model_loader: - kv 42: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 43: tokenizer.ggml.eos_token_id u32 = 11
llama_model_loader: - kv 44: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 45: tokenizer.ggml.padding_token_id u32 = 999
llama_model_loader: - kv 46: tokenizer.chat_template str = {# Unsloth template fixes #}
{% macro...
llama_model_loader: - kv 47: general.quantization_version u32 = 2
llama_model_loader: - kv 48: general.file_type u32 = 17
llama_model_loader: - kv 49: quantize.imatrix.file str = Nemotron-3-Nano-30B-A3B-GGUF/imatrix_...
llama_model_loader: - kv 50: quantize.imatrix.dataset str = unsloth_calibration_Nemotron-3-Nano-3...
llama_model_loader: - kv 51: quantize.imatrix.entries_count u32 = 185
llama_model_loader: - kv 52: quantize.imatrix.chunks_count u32 = 80
llama_model_loader: - type f32: 237 tensors
llama_model_loader: - type q5_0: 3 tensors
llama_model_loader: - type q5_1: 62 tensors
llama_model_loader: - type q8_0: 70 tensors
llama_model_loader: - type q5_K: 29 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q5_K - Medium
print_info: file size = 25.61 GiB (6.97 BPW)
load: printing all EOG tokens:
load: - 11 ('<|im_end|>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8499 MB
print_info: arch = nemotron_h_moe
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 1048576
print_info: n_embd = 2688
print_info: n_embd_inp = 2688
print_info: n_layer = 52
print_info: n_head = 32
print_info: n_head_kv = [0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_rot = 84
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = [0, 0, 0, 0, 0, 16, 0, 0, 0, 0, 0, 0, 16, 0, 0, 0, 0, 0, 0, 16, 0, 0, 0, 0, 0, 0, 16, 0, 0, 0, 0, 0, 0, 16, 0, 0, 0, 0, 0, 0, 0, 0, 16, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_embd_k_gqa = [0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_embd_v_gqa = [0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 0, 0, 256, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = [0, 1856, 0, 1856, 0, 0, 1856, 0, 1856, 0, 1856, 0, 0, 1856, 0, 1856, 0, 1856, 0, 0, 1856, 0, 1856, 0, 1856, 0, 0, 1856, 0, 1856, 0, 1856, 0, 0, 1856, 0, 1856, 0, 1856, 0, 1856, 0, 0, 1856, 0, 1856, 0, 1856, 0, 1856, 0, 1856]
print_info: n_expert = 128
print_info: n_expert_used = 6
print_info: n_expert_groups = 1
print_info: n_group_used = 1
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = -1
print_info: rope scaling = linear
print_info: freq_base_train = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 1048576
print_info: rope_yarn_log_mul= 0.0000
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 4
print_info: ssm_d_inner = 4096
print_info: ssm_d_state = 128
print_info: ssm_dt_rank = 64
print_info: ssm_n_group = 8
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 31B.A3.5B
print_info: model params = 31.58 B
print_info: general.name = Nemotron-3-Nano-30B-A3B
print_info: f_embedding_scale = 0.000000
print_info: f_residual_scale = 0.000000
print_info: f_attention_scale = 0.000000
print_info: n_ff_shexp = 3712
print_info: vocab type = BPE
print_info: n_vocab = 131072
print_info: n_merges = 269443
print_info: BOS token = 1 ''
print_info: EOS token = 11 '<|im_end|>'
print_info: EOT token = 11 '<|im_end|>'
print_info: UNK token = 0 ''
print_info: PAD token = 999 '<SPECIAL_999>'
print_info: LF token = 1010 'Ċ'
print_info: EOG token = 11 '<|im_end|>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading output layer to GPU
load_tensors: offloading 51 repeating layers to GPU
load_tensors: offloaded 53/53 layers to GPU
load_tensors: CUDA0 model buffer size = 1574.40 MiB
load_tensors: CUDA_Host model buffer size = 24650.06 MiB
....................................................
common_init_result: added <|im_end|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max = 4
llama_context: n_ctx = 50176
llama_context: n_ctx_seq = 50176
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = enabled
llama_context: kv_unified = true
llama_context: freq_base = 10000.0
llama_context: freq_scale = 1
llama_context: n_ctx_seq (50176) < n_ctx_train (1048576) -- the full capacity of the model will not be utilized
llama_context: CUDA_Host output buffer size = 2.00 MiB
llama_kv_cache: CUDA0 KV buffer size = 82.69 MiB
llama_kv_cache: size = 82.69 MiB ( 50176 cells, 6 layers, 4/1 seqs), K (q4_0): 41.34 MiB, V (q4_0): 41.34 MiB
llama_memory_recurrent: CUDA0 RS buffer size = 190.47 MiB
llama_memory_recurrent: size = 190.47 MiB ( 4 cells, 52 layers, 4 seqs), R (f32): 6.47 MiB, S (f32): 184.00 MiB
llama_context: CUDA0 compute buffer size = 1193.36 MiB
llama_context: CUDA_Host compute buffer size = 103.26 MiB
llama_context: graph nodes = 2188
llama_context: graph splits = 48
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv init: initializing slots, n_slots = 4
slot init: id 0 | task -1 | new slot, n_ctx = 50176
slot init: id 1 | task -1 | new slot, n_ctx = 50176
slot init: id 2 | task -1 | new slot, n_ctx = 50176
slot init: id 3 | task -1 | new slot, n_ctx = 50176
srv init: prompt cache is enabled, size limit: 8192 MiB
srv init: use --cache-ram 0 to disable the prompt cache
srv init: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
srv init: thinking = 1
init: chat template, chat_template: {# Unsloth template fixes #}
{% macro render_extra_keys(json_dict, handled_keys) %}
{%- if json_dict is mapping %}
{%- for json_key in json_dict if json_key not in handled_keys %}
{%- if json_dict[json_key] is mapping or (json_dict[json_key] is sequence and json_dict[json_key] is not string) %}
{{- '\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | tojson | safe) ~ '</' ~ json_key ~ '>' }}
{%- else %}
{{-'\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '</' ~ json_key ~ '>' }}
{%- endif %}
{%- endfor %}
{%- endif %}
{% endmacro %}
{%- set enable_thinking = enable_thinking if enable_thinking is defined else True %}
{%- set truncate_history_thinking = truncate_history_thinking if truncate_history_thinking is defined else True %}

{%- set ns = namespace(last_user_idx = -1) %}
{%- set loop_messages = messages %}
{%- for m in loop_messages %}
{%- if m["role"] == "user" %}
{%- set ns.last_user_idx = loop.index0 %}
{%- endif %}
{%- endfor %}

{%- if messages[0]["role"] == "system" %}
{%- set system_message = messages[0]["content"] %}
{%- set loop_messages = messages[1:] %}
{%- else %}
{%- set system_message = "" %}
{%- set loop_messages = messages %}
{%- endif %}
{%- if not tools is defined %}
{%- set tools = [] %}
{%- endif %}
{# Recompute last_user_idx relative to loop_messages after handling system #}
{%- set ns = namespace(last_user_idx = -1) %}
{%- for m in loop_messages %}
{%- if m["role"] == "user" %}
{%- set ns.last_user_idx = loop.index0 %}
{%- endif %}
{%- endfor %}
{%- if system_message is defined %}
{{- "<|im_start|>system\n" + system_message }}
{%- else %}
{%- if tools is iterable and tools | length > 0 %}
{{- "<|im_start|>system\n" }}
{%- endif %}
{%- endif %}
{%- if tools is iterable and tools | length > 0 %}
{%- if system_message is defined and system_message | length > 0 %}
{{- "\n\n" }}
{%- endif %}
{{- "# Tools\n\nYou have access to the following functions:\n\n" }}
{{- "" }}
{%- for tool in tools %}
{%- if tool.function is defined %}
{%- set tool = tool.function %}
{%- endif %}
{{- "\n\n" ~ tool.name ~ "" }}
{%- if tool.description is defined %}
{{- '\n' ~ (tool.description | trim) ~ '' }}
{%- endif %}
{{- '\n' }}
{%- if tool.parameters is defined and tool.parameters is mapping and tool.parameters.properties is defined and tool.parameters.properties is mapping %}
{%- for param_name, param_fields in tool.parameters.properties|items %}
{{- '\n' }}
{{- '\n' ~ param_name ~ '' }}
{%- if param_fields.type is defined %}
{{- '\n' ~ (param_fields.type | string) ~ '' }}
{%- endif %}
{%- if param_fields.description is defined %}
{{- '\n' ~ (param_fields.description | trim) ~ '' }}
{%- endif %}
{%- if param_fields.enum is defined %}
{{- '\n' ~ (param_fields.enum | tojson | safe) ~ '' }}
{%- endif %}
{%- set handled_keys = ['name', 'type', 'description', 'enum'] %}
{{- render_extra_keys(param_fields, handled_keys) }}
{{- '\n' }}
{%- endfor %}
{%- endif %}
{% set handled_keys = ['type', 'properties', 'required'] %}
{{- render_extra_keys(tool.parameters, handled_keys) }}
{%- if tool.parameters is defined and tool.parameters.required is defined %}
{{- '\n' ~ (tool.parameters.required | tojson | safe) ~ '' }}
{%- endif %}
{{- '\n' }}
{%- set handled_keys = ['type', 'name', 'description', 'parameters'] %}
{{- render_extra_keys(tool, handled_keys) }}
{{- '\n' }}
{%- endfor %}
{{- "\n" }}

{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n\n\n</tool_call>\n\n\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n' }}
{%- endif %}

{%- if system_message is defined %}
{{- '<|im_end|>\n' }}
{%- else %}
{%- if tools is iterable and tools | length > 0 %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}

{%- for message in loop_messages %}
{%- if message.role == "assistant" %}
{# Add reasoning content in to content field for unified processing below. #}
{%- if message.reasoning_content is defined and message.reasoning_content is string and message.reasoning_content | trim | length > 0 %}
{%- set content = "\n" ~ message.reasoning_content ~ "\n\n" ~ (message.content | default('', true)) %}
{%- else %}
{%- set content = message.content | default('', true) %}
{%- if content is string -%}
{# Allow downstream logic to to take care of broken thought, only handle coherent reasoning here. #}
{%- if '' not in content and '' not in content -%}
{%- set content = "" ~ content -%}
{%- endif -%}
{%- else -%}
{%- set content = content -%}
{%- endif -%}
{%- endif %}
{%- if message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
{# Assistant message has tool calls. #}
{{- '<|im_start|>assistant\n' }}
{%- set include_content = not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
{%- if content is string and content | trim | length > 0 %}
{%- if include_content %}
{{- (content | trim) ~ '\n' -}}
{%- else %}
{%- set c = (content | string) %}
{%- if '' in c %}
{# Keep only content after the last closing think. Also generation prompt causes this. #}
{%- set c = (c.split('')|last) %}
{%- elif '' in c %}
{# If was opened but never closed, remove the trailing think segment #}
{%- set c = (c.split('')|first) %}
{%- endif %}
{%- set c = "" ~ c | trim %}
{%- if c | length > 0 %}
{{- c ~ '\n' -}}
{%- endif %}
{%- endif %}
{%- else %}
{{- "" -}}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '<tool_call>\n<function=' ~ tool_call.name ~ '>\n' -}}
{%- if tool_call.arguments is defined %}{%- if tool_call.arguments is mapping %}
{%- for args_name, args_value in tool_call.arguments|items %}
{{- '<parameter=' ~ args_name ~ '>\n' -}}
{%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
{{- args_value ~ '\n\n' -}}
{%- endfor %}{%- endif %}
{%- endif %}
{{- '\n</tool_call>\n' -}}
{%- endfor %}
{{- '<|im_end|>\n' }}
{%- else %}
{# Assistant message doesn't have tool calls. #}
{%- if not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
{{- '<|im_start|>assistant\n' ~ (content | default('', true) | string | trim) ~ '<|im_end|>\n' }}
{%- else %}
{%- set c = (content | default('', true) | string) %}
{%- if '' in c and '' in c %}
{%- set c = "" ~ (c.split('')|last) %}
{%- endif %}
{%- set c = c | trim %}
{%- if c | length > 0 %}
{{- '<|im_start|>assistant\n' ~ c ~ '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>assistant\n<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endif %}
{%- elif message.role == "user" or message.role == "system" %}
{{- '<|im_start|>' + message.role + '\n' }}
{%- set content = message.content | string %}
{{- content }}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if loop.previtem and loop.previtem.role != "tool" %}
{{- '<|im_start|>user\n' }}
{%- endif %}
{{- '<tool_response>\n' }}
{{- message.content }}
{{- '\n</tool_response>\n' }}
{%- if not loop.last and loop.nextitem.role != "tool" %}
{{- '<|im_end|>\n' }}
{%- elif loop.last %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>\n' }}
{%- endif %}
{%- endfor %}

{%- if add_generation_prompt %}
{%- if enable_thinking %}
{{- '<|im_start|>assistant\n\n' }}
{%- else %}
{{- '<|im_start|>assistant\n' }}
{%- endif %}
{%- endif %}
{# Copyright 2025-present Unsloth. Apache 2.0 License. #}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello
Assistant
Hi there
I didn't receive any text to translate. Please provide the content you'd like translated into English
How are you?
<|im_start|>assistant

'
main: model loaded
main: server is listening on http://127.0.0.1:8080
main: starting the main loop...
srv update_slots: all slots are idle
srv params_from_: Chat format: peg-constructed
slot get_available: id 3 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 0 | processing task
slot update_slots: id 3 | task 0 | new prompt, n_ctx_slot = 50176, n_keep = 0, task.n_tokens = 31367
slot update_slots: id 3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 2048, batch.n_tokens = 2048, progress = 0.065292
slot update_slots: id 3 | task 0 | n_tokens = 2048, memory_seq_rm [2048, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 4096, batch.n_tokens = 2048, progress = 0.130583
slot update_slots: id 3 | task 0 | n_tokens = 4096, memory_seq_rm [4096, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 6144, batch.n_tokens = 2048, progress = 0.195875
slot update_slots: id 3 | task 0 | n_tokens = 6144, memory_seq_rm [6144, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 8192, batch.n_tokens = 2048, progress = 0.261166
slot update_slots: id 3 | task 0 | n_tokens = 8192, memory_seq_rm [8192, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 10240, batch.n_tokens = 2048, progress = 0.326458
slot update_slots: id 3 | task 0 | n_tokens = 10240, memory_seq_rm [10240, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 12288, batch.n_tokens = 2048, progress = 0.391749
slot update_slots: id 3 | task 0 | n_tokens = 12288, memory_seq_rm [12288, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 14336, batch.n_tokens = 2048, progress = 0.457041
slot update_slots: id 3 | task 0 | n_tokens = 14336, memory_seq_rm [14336, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 16384, batch.n_tokens = 2048, progress = 0.522332
slot update_slots: id 3 | task 0 | n_tokens = 16384, memory_seq_rm [16384, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 18432, batch.n_tokens = 2048, progress = 0.587624
slot update_slots: id 3 | task 0 | n_tokens = 18432, memory_seq_rm [18432, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 20480, batch.n_tokens = 2048, progress = 0.652915
slot update_slots: id 3 | task 0 | n_tokens = 20480, memory_seq_rm [20480, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 22528, batch.n_tokens = 2048, progress = 0.718207
slot update_slots: id 3 | task 0 | n_tokens = 22528, memory_seq_rm [22528, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 24576, batch.n_tokens = 2048, progress = 0.783499
slot update_slots: id 3 | task 0 | n_tokens = 24576, memory_seq_rm [24576, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 26624, batch.n_tokens = 2048, progress = 0.848790
slot update_slots: id 3 | task 0 | n_tokens = 26624, memory_seq_rm [26624, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 28672, batch.n_tokens = 2048, progress = 0.914082
slot update_slots: id 3 | task 0 | n_tokens = 28672, memory_seq_rm [28672, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 30720, batch.n_tokens = 2048, progress = 0.979373
slot update_slots: id 3 | task 0 | n_tokens = 30720, memory_seq_rm [30720, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 31303, batch.n_tokens = 583, progress = 0.997960
slot update_slots: id 3 | task 0 | n_tokens = 31303, memory_seq_rm [31303, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 31367, batch.n_tokens = 64, progress = 1.000000
slot update_slots: id 3 | task 0 | prompt done, n_tokens = 31367, batch.n_tokens = 64
slot update_slots: id 3 | task 0 | created context checkpoint 1 of 8 (pos_min = 31302, pos_max = 31302, size = 47.618 MiB)
slot print_timing: id 3 | task 0 |
prompt eval time = 39170.87 ms / 31367 tokens ( 1.25 ms per token, 800.77 tokens per second)
eval time = 53607.89 ms / 3044 tokens ( 17.61 ms per token, 56.78 tokens per second)
total time = 92778.76 ms / 34411 tokens
slot release: id 3 | task 0 | stop processing: n_tokens = 34410, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv params_from_: Chat format: peg-constructed
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.913 (> 0.100 thold), f_keep = 0.912
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 3061 | processing task
slot update_slots: id 3 | task 3061 | new prompt, n_ctx_slot = 50176, n_keep = 0, task.n_tokens = 34364
slot update_slots: id 3 | task 3061 | n_past = 31366, slot.prompt.tokens.size() = 34410, seq_id = 3, pos_min = 34409, n_swa = 1
slot update_slots: id 3 | task 3061 | restored context checkpoint (pos_min = 31302, pos_max = 31302, size = 47.618 MiB)
slot update_slots: id 3 | task 3061 | n_tokens = 31303, memory_seq_rm [31303, end)
slot update_slots: id 3 | task 3061 | prompt processing progress, n_tokens = 33351, batch.n_tokens = 2048, progress = 0.970521
slot update_slots: id 3 | task 3061 | n_tokens = 33351, memory_seq_rm [33351, end)
slot update_slots: id 3 | task 3061 | prompt processing progress, n_tokens = 34300, batch.n_tokens = 949, progress = 0.998138
slot update_slots: id 3 | task 3061 | n_tokens = 34300, memory_seq_rm [34300, end)
slot update_slots: id 3 | task 3061 | prompt processing progress, n_tokens = 34364, batch.n_tokens = 64, progress = 1.000000
slot update_slots: id 3 | task 3061 | prompt done, n_tokens = 34364, batch.n_tokens = 64
slot update_slots: id 3 | task 3061 | created context checkpoint 2 of 8 (pos_min = 34299, pos_max = 34299, size = 47.618 MiB)
slot print_timing: id 3 | task 3061 |
prompt eval time = 4410.93 ms / 3061 tokens ( 1.44 ms per token, 693.96 tokens per second)
eval time = 51547.42 ms / 2917 tokens ( 17.67 ms per token, 56.59 tokens per second)
total time = 55958.35 ms / 5978 tokens
slot release: id 3 | task 3061 | stop processing: n_tokens = 37280, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv params_from_: Chat format: peg-constructed
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.922 (> 0.100 thold), f_keep = 0.922
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 5981 | processing task
slot update_slots: id 3 | task 5981 | new prompt, n_ctx_slot = 50176, n_keep = 0, task.n_tokens = 37257
slot update_slots: id 3 | task 5981 | n_past = 34363, slot.prompt.tokens.size() = 37280, seq_id = 3, pos_min = 37279, n_swa = 1
slot update_slots: id 3 | task 5981 | restored context checkpoint (pos_min = 34299, pos_max = 34299, size = 47.618 MiB)
slot update_slots: id 3 | task 5981 | n_tokens = 34300, memory_seq_rm [34300, end)
slot update_slots: id 3 | task 5981 | prompt processing progress, n_tokens = 36348, batch.n_tokens = 2048, progress = 0.975602
slot update_slots: id 3 | task 5981 | n_tokens = 36348, memory_seq_rm [36348, end)
slot update_slots: id 3 | task 5981 | prompt processing progress, n_tokens = 37193, batch.n_tokens = 845, progress = 0.998282
slot update_slots: id 3 | task 5981 | n_tokens = 37193, memory_seq_rm [37193, end)
slot update_slots: id 3 | task 5981 | prompt processing progress, n_tokens = 37257, batch.n_tokens = 64, progress = 1.000000
slot update_slots: id 3 | task 5981 | prompt done, n_tokens = 37257, batch.n_tokens = 64
slot update_slots: id 3 | task 5981 | created context checkpoint 3 of 8 (pos_min = 37192, pos_max = 37192, size = 47.618 MiB)
slot print_timing: id 3 | task 5981 |
prompt eval time = 4412.69 ms / 2957 tokens ( 1.49 ms per token, 670.11 tokens per second)
eval time = 66540.83 ms / 3773 tokens ( 17.64 ms per token, 56.70 tokens per second)
total time = 70953.52 ms / 6730 tokens
slot release: id 3 | task 5981 | stop processing: n_tokens = 41029, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv params_from_: Chat format: peg-constructed
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.910 (> 0.100 thold), f_keep = 0.908
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 9757 | processing task
slot update_slots: id 3 | task 9757 | new prompt, n_ctx_slot = 50176, n_keep = 0, task.n_tokens = 40923
slot update_slots: id 3 | task 9757 | n_past = 37256, slot.prompt.tokens.size() = 41029, seq_id = 3, pos_min = 41028, n_swa = 1
slot update_slots: id 3 | task 9757 | restored context checkpoint (pos_min = 37192, pos_max = 37192, size = 47.618 MiB)
slot update_slots: id 3 | task 9757 | n_tokens = 37193, memory_seq_rm [37193, end)
slot update_slots: id 3 | task 9757 | prompt processing progress, n_tokens = 39241, batch.n_tokens = 2048, progress = 0.958898
slot update_slots: id 3 | task 9757 | n_tokens = 39241, memory_seq_rm [39241, end)
slot update_slots: id 3 | task 9757 | prompt processing progress, n_tokens = 40859, batch.n_tokens = 1618, progress = 0.998436
slot update_slots: id 3 | task 9757 | n_tokens = 40859, memory_seq_rm [40859, end)
slot update_slots: id 3 | task 9757 | prompt processing progress, n_tokens = 40923, batch.n_tokens = 64, progress = 1.000000
slot update_slots: id 3 | task 9757 | prompt done, n_tokens = 40923, batch.n_tokens = 64
slot update_slots: id 3 | task 9757 | created context checkpoint 4 of 8 (pos_min = 40858, pos_max = 40858, size = 47.618 MiB)
slot print_timing: id 3 | task 9757 |
prompt eval time = 5673.03 ms / 3730 tokens ( 1.52 ms per token, 657.50 tokens per second)
eval time = 67893.41 ms / 3803 tokens ( 17.85 ms per token, 56.01 tokens per second)
total time = 73566.44 ms / 7533 tokens
slot release: id 3 | task 9757 | stop processing: n_tokens = 44725, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
srv params_from_: Chat format: peg-constructed
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.918 (> 0.100 thold), f_keep = 0.915
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 13563 | processing task
slot update_slots: id 3 | task 13563 | new prompt, n_ctx_slot = 50176, n_keep = 0, task.n_tokens = 44600
slot update_slots: id 3 | task 13563 | n_past = 40922, slot.prompt.tokens.size() = 44725, seq_id = 3, pos_min = 44724, n_swa = 1
slot update_slots: id 3 | task 13563 | restored context checkpoint (pos_min = 40858, pos_max = 40858, size = 47.618 MiB)
slot update_slots: id 3 | task 13563 | n_tokens = 40859, memory_seq_rm [40859, end)
slot update_slots: id 3 | task 13563 | prompt processing progress, n_tokens = 42907, batch.n_tokens = 2048, progress = 0.962040
slot update_slots: id 3 | task 13563 | n_tokens = 42907, memory_seq_rm [42907, end)
slot update_slots: id 3 | task 13563 | prompt processing progress, n_tokens = 44536, batch.n_tokens = 1629, progress = 0.998565
slot update_slots: id 3 | task 13563 | n_tokens = 44536, memory_seq_rm [44536, end)
slot update_slots: id 3 | task 13563 | prompt processing progress, n_tokens = 44600, batch.n_tokens = 64, progress = 1.000000
slot update_slots: id 3 | task 13563 | prompt done, n_tokens = 44600, batch.n_tokens = 64
slot update_slots: id 3 | task 13563 | created context checkpoint 5 of 8 (pos_min = 44535, pos_max = 44535, size = 47.618 MiB)
slot print_timing: id 3 | task 13563 |
prompt eval time = 5810.23 ms / 3741 tokens ( 1.55 ms per token, 643.86 tokens per second)
eval time = 48800.94 ms / 2694 tokens ( 18.11 ms per token, 55.20 tokens per second)
total time = 54611.17 ms / 6435 tokens
slot release: id 3 | task 13563 | stop processing: n_tokens = 47293, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 127.0.0.1 200

[ikawrakow]

@msievers I'll eventually look into mamba and Nemotron-3. Not sure when (currently obsessing with CUDA tensor parallel).

@chatchatgpt I wouldn't consider your system low-end, which was the stipulated use case for this model. I do understand the theoretical advantage. But there is the theory, and then there is the practice. So, to put some meaning behind your 653 t/s PP and 55 t/s TG at a context of 47k tokens, it would be useful to know how Qwen3-30B-A3B and GPT-OSS-20B are doing on your system in comparison.

I don't have such a fast EPYC, but I do have a Ryzen-7950X with a 4080 GPU. Running GPT-OSS-20B on that system with all experts left on the CPU, at 47k tokens I get 2744 t/s PP-2048 and 33.5 t/s TG. PP at 64k tokens context is 65% of PP at zero context. TG is even better: at 64k tokens it is 84% of TG at zero context. Below is a full sweep-bench. It is very hard to do better than that with a hybrid model, and the llama.cpp implementation seems very far from being even remotely competitive.

GPT-OSS-20B-MXFP4, --cpu-moe, Ryzen-7950X, RTX-4080

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
2048	64	0	0.604	3392.47	1.782	35.92
2048	64	2048	0.579	3534.56	1.785	35.86
2048	64	4096	0.566	3617.86	1.787	35.81
2048	64	6144	0.578	3542.90	1.792	35.71
2048	64	8192	0.589	3475.16	1.806	35.44
2048	64	10240	0.611	3349.78	1.810	35.35
2048	64	12288	0.598	3426.98	1.805	35.46
2048	64	14336	0.607	3372.30	1.816	35.25
2048	64	16384	0.624	3283.77	1.825	35.07
2048	64	18432	0.635	3227.47	1.827	35.03
2048	64	20480	0.641	3196.85	1.837	34.85
2048	64	22528	0.649	3158.04	1.836	34.86
2048	64	24576	0.654	3129.37	1.845	34.69
2048	64	26624	0.672	3049.60	1.862	34.37
2048	64	28672	0.666	3075.88	1.860	34.41
2048	64	30720	0.681	3005.81	1.861	34.38
2048	64	32768	0.690	2969.68	1.872	34.19
2048	64	34816	0.697	2939.96	1.884	33.96
2048	64	36864	0.701	2920.52	1.884	33.96
2048	64	38912	0.705	2903.38	1.900	33.68
2048	64	40960	0.710	2885.70	1.897	33.73
2048	64	43008	0.729	2810.48	1.901	33.66
2048	64	45056	0.738	2776.70	1.913	33.45
2048	64	47104	0.746	2744.51	1.924	33.26
2048	64	49152	0.755	2714.04	1.934	33.09
2048	64	51200	0.751	2725.26	1.920	33.33
2048	64	53248	0.762	2687.04	1.926	33.23
2048	64	55296	0.788	2597.90	1.930	33.16
2048	64	57344	0.782	2617.81	1.934	33.10
2048	64	59392	0.788	2598.20	1.947	32.87
2048	64	61440	0.806	2540.10	1.946	32.89
2048	64	63488	0.935	2190.36	1.949	32.83

Qwen3-30B-A3B-Q4_K_M, --cpu-moe, Ryzen-7950X, RTX-4080

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
2048	64	0	0.995	2059.10	1.653	38.72
2048	64	2048	0.976	2097.62	1.654	38.70
2048	64	4096	0.992	2065.34	1.679	38.11
2048	64	6144	1.016	2016.45	1.702	37.61
2048	64	8192	1.053	1944.59	1.721	37.19
2048	64	10240	1.077	1901.36	1.745	36.68
2048	64	12288	1.104	1854.32	1.728	37.05
2048	64	14336	1.132	1809.11	1.744	36.70
2048	64	16384	1.163	1760.28	1.757	36.42
2048	64	18432	1.194	1714.96	1.777	36.02
2048	64	20480	1.226	1671.14	1.801	35.53
2048	64	22528	1.256	1630.88	1.820	35.16
2048	64	24576	1.273	1608.86	1.844	34.70
2048	64	26624	1.308	1565.17	1.861	34.39
2048	64	28672	1.343	1525.18	1.881	34.02
2048	64	30720	1.352	1514.79	1.904	33.62
2048	64	32768	1.398	1465.07	1.926	33.22
2048	64	34816	1.420	1442.71	1.950	32.82
2048	64	36864	1.443	1419.35	1.968	32.52
2048	64	38912	1.459	1403.60	1.995	32.08
2048	64	40960	1.504	1361.90	2.010	31.85
2048	64	43008	1.515	1351.89	2.026	31.59
2048	64	45056	1.559	1313.57	2.049	31.24
2048	64	47104	1.569	1305.01	2.067	30.96
2048	64	49152	1.611	1271.27	2.090	30.62
2048	64	51200	1.621	1263.44	2.107	30.38
2048	64	53248	1.658	1235.57	2.123	30.14
2048	64	55296	1.681	1218.04	2.149	29.78
2048	64	57344	1.729	1184.43	2.166	29.55
2048	64	59392	1.742	1175.57	2.188	29.25
2048	64	61440	1.772	1155.89	2.202	29.06
2048	64	63488	1.811	1130.80	2.235	28.64

[chatchatgpt]

@ikawrakow Yes, since the mainline llama.cpp implementation can already reach the speeds I’m seeing now, if in the future you ever implement these kinds of hybrid-architecture models in ik_llama.cpp when you have the interest and free time, the prefill speed, decode speed, and output quality would all be highly anticipated.
Also, many thanks for the -sm graph you and the community implemented — it’s astonishing. You mentioned you’re currently obsessed with tensor parallelism; I’m also very much looking forward to further optimizations in that area.
This is a rare high-performance project that keeps pace with new hardware while still supporting older models. Sometimes I half-jokingly feel like you’re a chip designer — forgive my bold guess, but your skill and level are extremely high.
I’m from China and used a translation AI, so some sentences may sound odd in English. Please forgive my poor English and the translation AI’s mistakes.

[msievers]

Just for the records ... IBM Granite 4-H (H as Hybrid Mamba-2 + Transformer) models also use Mamba, so maybe there could be synergies later. And although I've no clue about LLM architectures, what I've read so far is what @chatchatgpt says, that they should perform better with long contexts.

https://huggingface.co/collections/ibm-granite/granite-40-language-models

[chatchatgpt]

@msievers It seems data centers and a wide range of local inference users have entered an era of real-world applications characterized by long contexts. This may be why Chinese model vendors like qwen, deepseek, and kimi are all experimenting with hybrid-architecture models. Nvidia’s pricing is too high — I imagine governments and enterprises in developing countries feel the cost pain here, and developed countries probably feel it to some extent as well. Otherwise Nvidia wouldn’t release that 30b model, and IBM wouldn’t present Granite 4-h.

I suspect this is a strategic commercial need, not just a passing trend. Last year I saw Mr. Gu, the developer of the mamba architecture, in an interview where he mentioned his ideas. I even tried the jamba model at the time — the results weren’t good, so I didn’t follow it further until this year when qwen’s team released qwen3-next, and I felt it might be a turning point.
I’m an outsider. As far as I know, mamba-2 can control the size of the KV cache more efficiently and reduce computation.

Because the KV cache is smaller, the bandwidth usage of VRAM during attention mechanism operation is also reduced, which benefits relatively low-end GPUs like my 5070. For someone like me who has quite a few low-end GPUs, another architecture, RWKV — especially the rwkv-v8 architecture — looks like a promising future star, and I’ve been following and studying it. Of course, the flash-linear-attention mechanism is also important.

The six 5060 Ti cards in my office can happen to run Granite 4-h. Today is the weekend in China; I’ll see if I have time this afternoon to try it out, and I’ll also compare the quality of responses with qwen3-32b (this is very subjective). If the results are good (subjectively), I’ll report back in this thread.

You are professionals; I’m just an enthusiast learning as I go. Please forgive any mistakes and kindly point them out. Thank you all!

[chatchatgpt]

@msievers
I just tried running the granite 4.0-h with llama.cpp on my six 5060ti 16GB cards
Using -sm layer, ngl 99.
Decoding speed is about 42 tokens/s
The prefill speed is a bit odd — sometimes it's 150 tokens/s, sometimes 280 tokens/s, sometimes 680 tokens/s

Details

bill@3975 ~/l/b/bin (master)> ./llama-server -m /home/bill/granite-4.0-h-small-GGUF/granite-4.0-h-small-UD-Q5_K_XL.gguf -c 65536 -ngl 99 --host 0.0.0.0 -fa on
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 6 CUDA devices:
Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
Device 1: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
Device 2: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
Device 3: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
Device 4: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
Device 5: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 7486 (74e05131e) with GNU 13.3.0 for Linux x86_64
system info: n_threads = 32, n_threads_batch = 32, total_threads = 32

system_info: n_threads = 32 (n_threads_batch = 32) / 32 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

init: using 31 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model '/home/bill/granite-4.0-h-small-GGUF/granite-4.0-h-small-UD-Q5_K_XL.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected memory use with initial parameters [MiB]:
llama_params_fit_impl: - CUDA0 (NVIDIA GeForce RTX 5060 Ti): 15848 total, 5093 used, 10513 surplus
llama_params_fit_impl: - CUDA1 (NVIDIA GeForce RTX 5060 Ti): 15848 total, 7363 used, 8227 surplus
llama_params_fit_impl: - CUDA2 (NVIDIA GeForce RTX 5060 Ti): 15848 total, 7686 used, 7920 surplus
llama_params_fit_impl: - CUDA3 (NVIDIA GeForce RTX 5060 Ti): 15848 total, 7654 used, 7952 surplus
llama_params_fit_impl: - CUDA4 (NVIDIA GeForce RTX 5060 Ti): 15848 total, 7388 used, 8202 surplus
llama_params_fit_impl: - CUDA5 (NVIDIA GeForce RTX 5060 Ti): 15848 total, 4009 used, 11631 surplus
llama_params_fit_impl: projected to use 39196 MiB of device memory vs. 95088 MiB of free device memory
llama_params_fit_impl: will leave at least 7920 >= 1024 MiB of free memory on all devices, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.81 seconds
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 5060 Ti) (0000:01:00.0) - 15707 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 5060 Ti) (0000:03:00.0) - 15707 MiB free
llama_model_load_from_file_impl: using device CUDA2 (NVIDIA GeForce RTX 5060 Ti) (0000:21:00.0) - 15707 MiB free
llama_model_load_from_file_impl: using device CUDA3 (NVIDIA GeForce RTX 5060 Ti) (0000:22:00.0) - 15707 MiB free
llama_model_load_from_file_impl: using device CUDA4 (NVIDIA GeForce RTX 5060 Ti) (0000:41:00.0) - 15707 MiB free
llama_model_load_from_file_impl: using device CUDA5 (NVIDIA GeForce RTX 5060 Ti) (0000:61:00.0) - 15707 MiB free
llama_model_loader: loaded meta data with 47 key-value pairs and 666 tensors from /home/bill/granite-4.0-h-small-GGUF/granite-4.0-h-small-UD-Q5_K_XL.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = granitehybrid
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Granite-4.0-H-Small
llama_model_loader: - kv 3: general.basename str = Granite-4.0-H-Small
llama_model_loader: - kv 4: general.quantized_by str = Unsloth
llama_model_loader: - kv 5: general.size_label str = small
llama_model_loader: - kv 6: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 7: granitehybrid.block_count u32 = 40
llama_model_loader: - kv 8: granitehybrid.context_length u32 = 1048576
llama_model_loader: - kv 9: granitehybrid.embedding_length u32 = 4096
llama_model_loader: - kv 10: granitehybrid.feed_forward_length u32 = 768
llama_model_loader: - kv 11: granitehybrid.attention.head_count u32 = 32
llama_model_loader: - kv 12: granitehybrid.attention.head_count_kv arr[i32,40] = [0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, ...
llama_model_loader: - kv 13: granitehybrid.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 14: granitehybrid.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 15: granitehybrid.expert_count u32 = 72
llama_model_loader: - kv 16: granitehybrid.expert_used_count u32 = 10
llama_model_loader: - kv 17: granitehybrid.vocab_size u32 = 100352
llama_model_loader: - kv 18: granitehybrid.rope.dimension_count u32 = 128
llama_model_loader: - kv 19: granitehybrid.attention.scale f32 = 0.007812
llama_model_loader: - kv 20: granitehybrid.embedding_scale f32 = 12.000000
llama_model_loader: - kv 21: granitehybrid.residual_scale f32 = 0.220000
llama_model_loader: - kv 22: granitehybrid.logit_scale f32 = 16.000000
llama_model_loader: - kv 23: granitehybrid.expert_shared_feed_forward_length u32 = 1536
llama_model_loader: - kv 24: granitehybrid.ssm.conv_kernel u32 = 4
llama_model_loader: - kv 25: granitehybrid.ssm.state_size u32 = 128
llama_model_loader: - kv 26: granitehybrid.ssm.group_count u32 = 1
llama_model_loader: - kv 27: granitehybrid.ssm.inner_size u32 = 8192
llama_model_loader: - kv 28: granitehybrid.ssm.time_step_rank u32 = 128
llama_model_loader: - kv 29: granitehybrid.rope.scaling.finetuned bool = false
llama_model_loader: - kv 30: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 31: tokenizer.ggml.pre str = dbrx
llama_model_loader: - kv 32: tokenizer.ggml.tokens arr[str,100352] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 33: tokenizer.ggml.token_type arr[i32,100352] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 34: tokenizer.ggml.merges arr[str,100000] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 35: tokenizer.ggml.bos_token_id u32 = 100257
llama_model_loader: - kv 36: tokenizer.ggml.eos_token_id u32 = 100257
llama_model_loader: - kv 37: tokenizer.ggml.unknown_token_id u32 = 100269
llama_model_loader: - kv 38: tokenizer.ggml.padding_token_id u32 = 100256
llama_model_loader: - kv 39: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 40: tokenizer.chat_template str = {%- set tools_system_message_prefix =...
llama_model_loader: - kv 41: general.quantization_version u32 = 2
llama_model_loader: - kv 42: general.file_type u32 = 17
llama_model_loader: - kv 43: quantize.imatrix.file str = granite-4.0-h-small-GGUF/imatrix_unsl...
llama_model_loader: - kv 44: quantize.imatrix.dataset str = unsloth_calibration_granite-4.0-h-sma...
llama_model_loader: - kv 45: quantize.imatrix.entries_count u32 = 368
llama_model_loader: - kv 46: quantize.imatrix.chunks_count u32 = 204
llama_model_loader: - type f32: 337 tensors
llama_model_loader: - type q8_0: 120 tensors
llama_model_loader: - type q4_K: 10 tensors
llama_model_loader: - type q5_K: 165 tensors
llama_model_loader: - type q6_K: 34 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q5_K - Medium
print_info: file size = 21.53 GiB (5.74 BPW)
load: printing all EOG tokens:
load: - 100257 ('<|end_of_text|>')
load: - 100261 ('<|fim_pad|>')
load: special tokens cache size = 96
load: token to piece cache size = 0.6152 MB
print_info: arch = granitehybrid
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 1048576
print_info: n_embd = 4096
print_info: n_embd_inp = 4096
print_info: n_layer = 40
print_info: n_head = 32
print_info: n_head_kv = [0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0]
print_info: n_rot = 128
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = [0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0]
print_info: n_embd_k_gqa = [0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0]
print_info: n_embd_v_gqa = [0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0]
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 1.6e+01
print_info: f_attn_scale = 7.8e-03
print_info: n_ff = 768
print_info: n_expert = 72
print_info: n_expert_used = 10
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 0
print_info: rope scaling = linear
print_info: freq_base_train = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 1048576
print_info: rope_yarn_log_mul= 0.0000
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 4
print_info: ssm_d_inner = 8192
print_info: ssm_d_state = 128
print_info: ssm_dt_rank = 128
print_info: ssm_n_group = 1
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 32B
print_info: model params = 32.21 B
print_info: general.name = Granite-4.0-H-Small
print_info: f_embedding_scale = 12.000000
print_info: f_residual_scale = 0.220000
print_info: f_attention_scale = 0.007812
print_info: n_ff_shexp = 1536
print_info: vocab type = BPE
print_info: n_vocab = 100352
print_info: n_merges = 100000
print_info: BOS token = 100257 '<|end_of_text|>'
print_info: EOS token = 100257 '<|end_of_text|>'
print_info: EOT token = 100257 '<|end_of_text|>'
print_info: UNK token = 100269 '<|unk|>'
print_info: PAD token = 100256 '<|pad|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 100258 '<|fim_prefix|>'
print_info: FIM SUF token = 100260 '<|fim_suffix|>'
print_info: FIM MID token = 100259 '<|fim_middle|>'
print_info: FIM PAD token = 100261 '<|fim_pad|>'
print_info: EOG token = 100257 '<|end_of_text|>'
print_info: EOG token = 100261 '<|fim_pad|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading output layer to GPU
load_tensors: offloading 39 repeating layers to GPU
load_tensors: offloaded 41/41 layers to GPU
load_tensors: CPU_Mapped model buffer size = 321.56 MiB
load_tensors: CUDA0 model buffer size = 3950.04 MiB
load_tensors: CUDA1 model buffer size = 3817.11 MiB
load_tensors: CUDA2 model buffer size = 3644.60 MiB
load_tensors: CUDA3 model buffer size = 3612.54 MiB
load_tensors: CUDA4 model buffer size = 3874.48 MiB
load_tensors: CUDA5 model buffer size = 3148.23 MiB
...................................................................................................
common_init_result: added <|end_of_text|> logit bias = -inf
common_init_result: added <|fim_pad|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max = 4
llama_context: n_ctx = 65536
llama_context: n_ctx_seq = 65536
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = enabled
llama_context: kv_unified = true
llama_context: freq_base = 10000.0
llama_context: freq_scale = 1
llama_context: n_ctx_seq (65536) < n_ctx_train (1048576) -- the full capacity of the model will not be utilized
llama_context: CUDA_Host output buffer size = 1.53 MiB
llama_kv_cache: CUDA0 KV buffer size = 256.00 MiB
llama_kv_cache: CUDA2 KV buffer size = 256.00 MiB
llama_kv_cache: CUDA3 KV buffer size = 256.00 MiB
llama_kv_cache: CUDA5 KV buffer size = 256.00 MiB
llama_kv_cache: size = 1024.00 MiB ( 65536 cells, 4 layers, 4/1 seqs), K (f16): 512.00 MiB, V (f16): 512.00 MiB
llama_memory_recurrent: CUDA0 RS buffer size = 98.32 MiB
llama_memory_recurrent: CUDA1 RS buffer size = 114.71 MiB
llama_memory_recurrent: CUDA2 RS buffer size = 98.32 MiB
llama_memory_recurrent: CUDA3 RS buffer size = 98.32 MiB
llama_memory_recurrent: CUDA4 RS buffer size = 114.71 MiB
llama_memory_recurrent: CUDA5 RS buffer size = 65.55 MiB
llama_memory_recurrent: size = 589.92 MiB ( 4 cells, 40 layers, 4 seqs), R (f32): 13.92 MiB, S (f32): 576.00 MiB
llama_context: pipeline parallelism enabled (n_copies=4)
llama_context: CUDA0 compute buffer size = 789.03 MiB
llama_context: CUDA1 compute buffer size = 3399.66 MiB
llama_context: CUDA2 compute buffer size = 3655.69 MiB
llama_context: CUDA3 compute buffer size = 3655.69 MiB
llama_context: CUDA4 compute buffer size = 3367.66 MiB
llama_context: CUDA5 compute buffer size = 508.04 MiB
llama_context: CUDA_Host compute buffer size = 520.05 MiB
llama_context: graph nodes = 3775
llama_context: graph splits = 16
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv init: initializing slots, n_slots = 4
slot init: id 0 | task -1 | new slot, n_ctx = 65536
slot init: id 1 | task -1 | new slot, n_ctx = 65536
slot init: id 2 | task -1 | new slot, n_ctx = 65536
slot init: id 3 | task -1 | new slot, n_ctx = 65536
srv init: prompt cache is enabled, size limit: 8192 MiB
srv init: use --cache-ram 0 to disable the prompt cache
srv init: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
srv init: thinking = 0
init: chat template, chat_template: {%- set tools_system_message_prefix = 'You are a helpful assistant with access to the following tools. You may call one or more tools to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n' %}
{%- set tools_system_message_suffix = '\n\n\nFor each tool call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{"name": , "arguments": }\n</tool_call>. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request.' %}
{%- set documents_system_message_prefix = 'You are a helpful assistant with access to the following documents. You may use one or more documents to assist with the user query.\n\nYou are given a list of documents within XML tags:\n' %}
{%- set documents_system_message_suffix = '\n\n\nWrite the response to the user's input by strictly aligning with the facts in the provided documents. If the information needed to answer the question is not available in the documents, inform the user that the question cannot be answered based on the available data.' %}
{%- set g4_default_system_message = 'You are a helpful assistant. Please ensure responses are professional, accurate, and safe.' %}
{%- if available_tools is defined and available_tools %}
{%- set tools = available_tools %}
{%- endif %}
{%- set ns = namespace(tools_system_message=tools_system_message_prefix,
documents_system_message=documents_system_message_prefix,
default_system_message=g4_default_system_message,
system_message=''
) %}
{%- if tools %}
{%- for tool in tools %}
{%- set ns.tools_system_message = ns.tools_system_message + '\n' + (tool | tojson) %}
{%- endfor %}
{%- set ns.tools_system_message = ns.tools_system_message + tools_system_message_suffix %}
{%- else %}
{%- set ns.tools_system_message = '' %}
{%- endif %}
{%- if documents %}
{%- for document in documents %}
{%- set ns.documents_system_message = ns.documents_system_message + '\n' + (document | tojson) %}
{%- endfor %}
{%- set ns.documents_system_message = ns.documents_system_message + documents_system_message_suffix %}
{%- else %}
{%- set ns.documents_system_message = '' %}
{%- endif %}
{%- if messages[0].role == 'system' %}
{%- if messages[0].content is string %}
{%- set ns.system_message = messages[0].content %}
{%- elif messages[0].content is iterable %}
{%- for entry in messages[0].content %}
{%- if entry.type== 'text' %}
{%- if ns.system_message != '' %}
{%- set ns.system_message = ns.system_message + '\n' %}
{%- endif %}
{%- set ns.system_message = ns.system_message + entry.text %}
{%- endif %}
{%- endfor %}
{%- endif %}
{%- if tools and documents %}
{%- set ns.system_message = ns.system_message + '\n\n' + ns.tools_system_message + '\n\n' + ns.documents_system_message %}
{%- elif tools %}
{%- set ns.system_message = ns.system_message + '\n\n' + ns.tools_system_message %}
{%- elif documents %}
{%- set ns.system_message = ns.system_message + '\n\n' + ns.documents_system_message %}
{%- endif %}
{%- else %}
{%- if tools and documents %}
{%- set ns.system_message = ns.tools_system_message + '\n\n' + ns.documents_system_message %}
{%- elif tools %}
{%- set ns.system_message = ns.tools_system_message %}
{%- elif documents %}
{%- set ns.system_message = ns.documents_system_message %}
{%- endif %}
{%- endif %}
{%- if ns.system_message %}
{{- '<|start_of_role|>system<|end_of_role|>' + ns.system_message + '<|end_of_text|>\n' }}
{%- else %}
{{- '<|start_of_role|>system<|end_of_role|>' + ns.default_system_message + '<|end_of_text|>\n' }}
{%- endif %}
{%- for message in messages %}
{%- set content = namespace(val='') %}
{%- if message.content is string %}
{%- set content.val = message.content %}
{%- else %}
{%- if message.content is iterable %}
{%- for entry in message.content %}
{%- if entry.type== 'text' %}
{%- if content.val != '' %}
{%- set content.val = content.val + '\n' %}
{%- endif %}
{%- set content.val = content.val + entry.text %}
{%- endif %}
{%- endfor %}
{%- endif %}
{%- endif %}
{%- if (message.role == 'user') or (message.role == 'system' and not loop.first) %}
{{- '<|start_of_role|>' + message.role + '<|end_of_role|>' + content.val + '<|end_of_text|>\n' }}
{%- elif message.role == 'assistant' %}
{{- '<|start_of_role|>' + message.role + '<|end_of_role|>' + content.val }}
{%- if message.tool_calls %}
{%- for tool_call in message.tool_calls %}
{%- if (loop.first and content.val) or (not loop.first) %}
{{- '\n' }}
{%- endif %}
{%- if tool_call.function %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{%- if tool_call.arguments is string %}
{{- tool_call.arguments }}
{%- else %}
{{- tool_call.arguments | tojson }}
{%- endif %}
{{- '}\n</tool_call>' }}
{%- endfor %}
{%- endif %}
{{- '<|end_of_text|>\n' }}
{%- elif message.role == 'tool' %}
{%- if loop.first or (messages[loop.index0 - 1].role != 'tool') %}
{{- '<|start_of_role|>user<|end_of_role|>' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- content.val }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != 'tool') %}
{{- '<|end_of_text|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|start_of_role|>assistant<|end_of_role|>' }}
{%- endif %}, example_format: '<|start_of_role|>system<|end_of_role|>You are a helpful assistant<|end_of_text|>'
<|start_of_role|>user<|end_of_role|>Hello<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>Hi there<|end_of_text|>
<|start_of_role|>user<|end_of_role|>How are you?<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>'
main: model loaded
main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
srv update_slots: all slots are idle
srv log_server_r: request: GET /v1/models 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 0 | processing task
slot update_slots: id 3 | task 0 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 39
slot update_slots: id 3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 39, batch.n_tokens = 39, progress = 1.000000
slot update_slots: id 3 | task 0 | prompt done, n_tokens = 39, batch.n_tokens = 39
slot print_timing: id 3 | task 0 |
prompt eval time = 228.34 ms / 39 tokens ( 5.85 ms per token, 170.80 tokens per second)
eval time = 852.67 ms / 37 tokens ( 23.05 ms per token, 43.39 tokens per second)
total time = 1081.01 ms / 76 tokens
slot release: id 3 | task 0 | stop processing: n_tokens = 75, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.815 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 38 | processing task
slot update_slots: id 3 | task 38 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 92
slot update_slots: id 3 | task 38 | n_tokens = 75, memory_seq_rm [75, end)
slot update_slots: id 3 | task 38 | prompt processing progress, n_tokens = 92, batch.n_tokens = 17, progress = 1.000000
slot update_slots: id 3 | task 38 | prompt done, n_tokens = 92, batch.n_tokens = 17
slot update_slots: id 3 | task 38 | created context checkpoint 1 of 8 (pos_min = 74, pos_max = 74, size = 147.481 MiB)
slot print_timing: id 3 | task 38 |
prompt eval time = 94.65 ms / 17 tokens ( 5.57 ms per token, 179.61 tokens per second)
eval time = 10103.68 ms / 430 tokens ( 23.50 ms per token, 42.56 tokens per second)
total time = 10198.33 ms / 447 tokens
slot release: id 3 | task 38 | stop processing: n_tokens = 521, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.952 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 469 | processing task
slot update_slots: id 3 | task 469 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 547
slot update_slots: id 3 | task 469 | n_tokens = 521, memory_seq_rm [521, end)
slot update_slots: id 3 | task 469 | prompt processing progress, n_tokens = 547, batch.n_tokens = 26, progress = 1.000000
slot update_slots: id 3 | task 469 | prompt done, n_tokens = 547, batch.n_tokens = 26
slot update_slots: id 3 | task 469 | created context checkpoint 2 of 8 (pos_min = 520, pos_max = 520, size = 147.481 MiB)
slot print_timing: id 3 | task 469 |
prompt eval time = 156.78 ms / 26 tokens ( 6.03 ms per token, 165.84 tokens per second)
eval time = 11425.02 ms / 486 tokens ( 23.51 ms per token, 42.54 tokens per second)
total time = 11581.80 ms / 512 tokens
slot release: id 3 | task 469 | stop processing: n_tokens = 1032, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.980 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 956 | processing task
slot update_slots: id 3 | task 956 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 1053
slot update_slots: id 3 | task 956 | n_tokens = 1032, memory_seq_rm [1032, end)
slot update_slots: id 3 | task 956 | prompt processing progress, n_tokens = 1053, batch.n_tokens = 21, progress = 1.000000
slot update_slots: id 3 | task 956 | prompt done, n_tokens = 1053, batch.n_tokens = 21
slot update_slots: id 3 | task 956 | created context checkpoint 3 of 8 (pos_min = 1031, pos_max = 1031, size = 147.481 MiB)
slot print_timing: id 3 | task 956 |
prompt eval time = 98.19 ms / 21 tokens ( 4.68 ms per token, 213.87 tokens per second)
eval time = 15037.16 ms / 639 tokens ( 23.53 ms per token, 42.49 tokens per second)
total time = 15135.35 ms / 660 tokens
slot release: id 3 | task 956 | stop processing: n_tokens = 1691, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.983 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 1596 | processing task
slot update_slots: id 3 | task 1596 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 1721
slot update_slots: id 3 | task 1596 | n_tokens = 1691, memory_seq_rm [1691, end)
slot update_slots: id 3 | task 1596 | prompt processing progress, n_tokens = 1721, batch.n_tokens = 30, progress = 1.000000
slot update_slots: id 3 | task 1596 | prompt done, n_tokens = 1721, batch.n_tokens = 30
slot update_slots: id 3 | task 1596 | created context checkpoint 4 of 8 (pos_min = 1690, pos_max = 1690, size = 147.481 MiB)
slot print_timing: id 3 | task 1596 |
prompt eval time = 156.39 ms / 30 tokens ( 5.21 ms per token, 191.83 tokens per second)
eval time = 8983.73 ms / 381 tokens ( 23.58 ms per token, 42.41 tokens per second)
total time = 9140.11 ms / 411 tokens
slot release: id 3 | task 1596 | stop processing: n_tokens = 2101, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.983 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 1978 | processing task
slot update_slots: id 3 | task 1978 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 2137
slot update_slots: id 3 | task 1978 | n_tokens = 2101, memory_seq_rm [2101, end)
slot update_slots: id 3 | task 1978 | prompt processing progress, n_tokens = 2137, batch.n_tokens = 36, progress = 1.000000
slot update_slots: id 3 | task 1978 | prompt done, n_tokens = 2137, batch.n_tokens = 36
slot update_slots: id 3 | task 1978 | created context checkpoint 5 of 8 (pos_min = 2100, pos_max = 2100, size = 147.481 MiB)
slot print_timing: id 3 | task 1978 |
prompt eval time = 164.35 ms / 36 tokens ( 4.57 ms per token, 219.04 tokens per second)
eval time = 11025.50 ms / 467 tokens ( 23.61 ms per token, 42.36 tokens per second)
total time = 11189.86 ms / 503 tokens
slot release: id 3 | task 1978 | stop processing: n_tokens = 2603, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_available: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.655 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 2446 | processing task
slot update_slots: id 3 | task 2446 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 3975
slot update_slots: id 3 | task 2446 | n_tokens = 2603, memory_seq_rm [2603, end)
slot update_slots: id 3 | task 2446 | prompt processing progress, n_tokens = 3911, batch.n_tokens = 1308, progress = 0.983899
slot update_slots: id 3 | task 2446 | n_tokens = 3911, memory_seq_rm [3911, end)
slot update_slots: id 3 | task 2446 | prompt processing progress, n_tokens = 3975, batch.n_tokens = 64, progress = 1.000000
slot update_slots: id 3 | task 2446 | prompt done, n_tokens = 3975, batch.n_tokens = 64
slot update_slots: id 3 | task 2446 | created context checkpoint 6 of 8 (pos_min = 3910, pos_max = 3910, size = 147.481 MiB)
slot print_timing: id 3 | task 2446 |
prompt eval time = 1867.67 ms / 1372 tokens ( 1.36 ms per token, 734.61 tokens per second)
eval time = 20745.88 ms / 877 tokens ( 23.66 ms per token, 42.27 tokens per second)
total time = 22613.55 ms / 2249 tokens
slot release: id 3 | task 2446 | stop processing: n_tokens = 4851, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.984 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 3325 | processing task
slot update_slots: id 3 | task 3325 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 4932
slot update_slots: id 3 | task 3325 | n_tokens = 4851, memory_seq_rm [4851, end)
slot update_slots: id 3 | task 3325 | prompt processing progress, n_tokens = 4868, batch.n_tokens = 17, progress = 0.987024
slot update_slots: id 3 | task 3325 | n_tokens = 4868, memory_seq_rm [4868, end)
slot update_slots: id 3 | task 3325 | prompt processing progress, n_tokens = 4932, batch.n_tokens = 64, progress = 1.000000
slot update_slots: id 3 | task 3325 | prompt done, n_tokens = 4932, batch.n_tokens = 64
slot update_slots: id 3 | task 3325 | created context checkpoint 7 of 8 (pos_min = 4867, pos_max = 4867, size = 147.481 MiB)
slot print_timing: id 3 | task 3325 |
prompt eval time = 212.83 ms / 81 tokens ( 2.63 ms per token, 380.58 tokens per second)
eval time = 7919.69 ms / 333 tokens ( 23.78 ms per token, 42.05 tokens per second)
total time = 8132.52 ms / 414 tokens
slot release: id 3 | task 3325 | stop processing: n_tokens = 5264, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.991 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 3660 | processing task
slot update_slots: id 3 | task 3660 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 5314
slot update_slots: id 3 | task 3660 | n_tokens = 5264, memory_seq_rm [5264, end)
slot update_slots: id 3 | task 3660 | prompt processing progress, n_tokens = 5314, batch.n_tokens = 50, progress = 1.000000
slot update_slots: id 3 | task 3660 | prompt done, n_tokens = 5314, batch.n_tokens = 50
slot update_slots: id 3 | task 3660 | created context checkpoint 8 of 8 (pos_min = 5263, pos_max = 5263, size = 147.481 MiB)
slot print_timing: id 3 | task 3660 |
prompt eval time = 185.78 ms / 50 tokens ( 3.72 ms per token, 269.14 tokens per second)
eval time = 7899.14 ms / 332 tokens ( 23.79 ms per token, 42.03 tokens per second)
total time = 8084.91 ms / 382 tokens
slot release: id 3 | task 3660 | stop processing: n_tokens = 5645, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.996 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 3993 | processing task
slot update_slots: id 3 | task 3993 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 5666
slot update_slots: id 3 | task 3993 | n_tokens = 5645, memory_seq_rm [5645, end)
slot update_slots: id 3 | task 3993 | prompt processing progress, n_tokens = 5666, batch.n_tokens = 21, progress = 1.000000
slot update_slots: id 3 | task 3993 | prompt done, n_tokens = 5666, batch.n_tokens = 21
slot update_slots: id 3 | task 3993 | erasing old context checkpoint (pos_min = 74, pos_max = 74, size = 147.481 MiB)
slot update_slots: id 3 | task 3993 | created context checkpoint 8 of 8 (pos_min = 5644, pos_max = 5644, size = 147.481 MiB)
slot print_timing: id 3 | task 3993 |
prompt eval time = 108.01 ms / 21 tokens ( 5.14 ms per token, 194.42 tokens per second)
eval time = 2680.09 ms / 113 tokens (23.72 ms per token, 42.16 tokens per second)
total time = 2788.10 ms / 134 tokens
slot release: id 3 | task 3993 | stop processing: n_tokens = 5778, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.997 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 4107 | processing task
slot update_slots: id 3 | task 4107 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 5794
slot update_slots: id 3 | task 4107 | n_tokens = 5778, memory_seq_rm [5778, end)
slot update_slots: id 3 | task 4107 | prompt processing progress, n_tokens = 5794, batch.n_tokens = 16, progress = 1.000000
slot update_slots: id 3 | task 4107 | prompt done, n_tokens = 5794, batch.n_tokens = 16
slot update_slots: id 3 | task 4107 | erasing old context checkpoint (pos_min = 520, pos_max = 520, size = 147.481 MiB)
slot update_slots: id 3 | task 4107 | created context checkpoint 8 of 8 (pos_min = 5777, pos_max = 5777, size = 147.481 MiB)
slot print_timing: id 3 | task 4107 |
prompt eval time = 100.94 ms / 16 tokens ( 6.31 ms per token, 158.51 tokens per second)
eval time = 12443.97 ms / 524 tokens ( 23.75 ms per token, 42.11 tokens per second)
total time = 12544.91 ms / 540 tokens
slot release: id 3 | task 4107 | stop processing: n_tokens = 6317, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.992 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 4632 | processing task
slot update_slots: id 3 | task 4632 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 6371
slot update_slots: id 3 | task 4632 | n_tokens = 6317, memory_seq_rm [6317, end)
slot update_slots: id 3 | task 4632 | prompt processing progress, n_tokens = 6371, batch.n_tokens = 54, progress = 1.000000
slot update_slots: id 3 | task 4632 | prompt done, n_tokens = 6371, batch.n_tokens = 54
slot update_slots: id 3 | task 4632 | erasing old context checkpoint (pos_min = 1031, pos_max = 1031, size = 147.481 MiB)
slot update_slots: id 3 | task 4632 | created context checkpoint 8 of 8 (pos_min = 6316, pos_max = 6316, size = 147.481 MiB)
slot print_timing: id 3 | task 4632 |
prompt eval time = 191.73 ms / 54 tokens ( 3.55 ms per token, 281.65 tokens per second)
eval time = 3423.36 ms / 143 tokens ( 23.94 ms per token, 41.77 tokens per second)
total time = 3615.09 ms / 197 tokens
slot release: id 3 | task 4632 | stop processing: n_tokens = 6513, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200
srv params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.953 (> 0.100 thold), f_keep = 1.000
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 4776 | processing task
slot update_slots: id 3 | task 4776 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 6834
slot update_slots: id 3 | task 4776 | n_tokens = 6513, memory_seq_rm [6513, end)
slot update_slots: id 3 | task 4776 | prompt processing progress, n_tokens = 6770, batch.n_tokens = 257, progress = 0.990635
slot update_slots: id 3 | task 4776 | n_tokens = 6770, memory_seq_rm [6770, end)
slot update_slots: id 3 | task 4776 | prompt processing progress, n_tokens = 6834, batch.n_tokens = 64, progress = 1.000000
slot update_slots: id 3 | task 4776 | prompt done, n_tokens = 6834, batch.n_tokens = 64
slot update_slots: id 3 | task 4776 | erasing old context checkpoint (pos_min = 1690, pos_max = 1690, size = 147.481 MiB)
slot update_slots: id 3 | task 4776 | created context checkpoint 8 of 8 (pos_min = 6769, pos_max = 6769, size = 147.481 MiB)
slot print_timing: id 3 | task 4776 |
prompt eval time = 510.74 ms / 321 tokens ( 1.59 ms per token, 628.50 tokens per second)
eval time = 6999.45 ms / 293 tokens ( 23.89 ms per token, 41.86 tokens per second)
total time = 7510.18 ms / 614 tokens
slot release: id 3 | task 4776 | stop processing: n_tokens = 7126, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /v1/chat/completions 192.168.50.146 200

Subjectively, this model's capabilities are fairly average and may be inferior to NVIDIA's nemotron 3 nano, but everyone uses models differently. Since IBM spent a huge amount to train this model, it must have its uses—it's just that it happens not to meet my needs