At the moment gguf models routinely outperform QAIRT builds on my Snapdragon X Plus 10-Core machine, even though (in theory) it should be the other way around.
Below the debug logs from a 'test' prompt via qualcomm/qwen3-4b-instruct-2507 (QAIRT):
Sep 7 10:18:43.953 DBG go\ml.go:215 [ML] [src/ml.cpp:81:geniex_init] initializing ml
Sep 7 10:18:43.953 DBG go\ml.go:215 [ML] [src/registry.cpp:167:scan_plugins] Using shared lib directory for plugin path: C:\Users\user\AppData\Local\GenieX CLI
Sep 7 10:18:43.953 DBG go\ml.go:215 [ML] [src/registry.cpp:175:scan_plugins] Scanning plugins in: C:\Users\user\AppData\Local\GenieX CLI
Sep 7 10:18:43.953 DBG go\ml.go:215 [ML] [src/registry.cpp:180:scan_plugins] Scanning directory: C:\Users\user\AppData\Local\GenieX CLI\llama_cpp
Sep 7 10:18:43.954 DBG go\ml.go:215 [ML] [src/registry.cpp:33:PluginFactory] loading plugin from C:\Users\user\AppData\Local\GenieX CLI\llama_cpp\geniex_plugin.dll
Sep 7 10:18:46.370 DBG go\ml.go:215 [ML] [src/registry.cpp:46:PluginFactory] plugin id: llama_cpp
Sep 7 10:18:46.371 DBG go\ml.go:215 [ML] [src/registry.cpp:188:scan_plugins] Registered plugin: llama_cpp
Sep 7 10:18:46.371 DBG go\ml.go:215 [ML] [src/registry.cpp:180:scan_plugins] Scanning directory: C:\Users\user\AppData\Local\GenieX CLI\qairt
Sep 7 10:18:46.372 DBG go\ml.go:215 [ML] [src/registry.cpp:33:PluginFactory] loading plugin from C:\Users\user\AppData\Local\GenieX CLI\qairt\geniex_plugin.dll
Sep 7 10:18:48.159 DBG go\ml.go:215 [ML] [src/registry.cpp:46:PluginFactory] plugin id: qairt
Sep 7 10:18:48.163 DBG go\ml.go:215 [ML] [src/registry.cpp:188:scan_plugins] Registered plugin: qairt
Sep 7 10:18:48.163 DBG store\config.go:83 ConfigGet key=chipset
Sep 7 10:18:48.206 DBG go\llm.go:264 NewLLM called input="{ModelPath:C:\\Users\\user\\.cache\\geniex\\models\\qualcomm/Qwen3-4B-Instruct-2507\\part1_of_4.bin TokenizerPath: Config:{NCtx:0 NThreads:0 NThreadsBatch:0 NBatch:0 NUbatch:0 NSeqMax:0 NGpuLayers:0 ChatTemplatePath: ChatTemplateContent: SpecType: SpecDraftModel: SpecNMax:3 SpecNMin:0 SpecPMin:0} RuntimeID:qairt DeviceID:NPU}"
Sep 7 10:18:48.206 DBG go\ml.go:215 [ML] [src/llm.cpp:20:geniex_llm_create] LlmCreateInput(model_path: C:\Users\user\.cache\geniex\models\qualcomm/Qwen3-4B-Instruct-2507\part1_of_4.bin, tokenizer_path: nullptr, config: ModelConfig(n_ctx: 0, n_threads: 0, n_threads_batch: 0, n_batch: 0, n_ubatch: 0, n_seq_max: 0, n_gpu_layers: 0, chat_template_path: nullptr, chat_template_content: nullptr), plugin_id: qairt, device_id: NPU)
Sep 7 10:18:48.207 DBG go\ml.go:215 [ML] [plugins/qairt/src/plugin.cpp:61:create_plugin] creating qairt plugin
Sep 7 10:18:48.207 DBG go\ml.go:215 [ML] [plugins/qairt/src/plugin.cpp:19:QairtPlugin] creating and initializing qairt plugin
Sep 7 10:18:48.208 DBG go\ml.go:215 [ML] [plugins/qairt/src/../include\qnn_runtime_utils.h:142:make_qnn_runtime_config] GENIEX_QAIRT_LIB unset; using the QAIRT runtime bundled with the plugin
Sep 7 10:18:48.209 DBG go\ml.go:215 [ML] [plugins/qairt/src/llm.cpp:80:create] Found 4 model shards in C:\Users\user\.cache\geniex\models\qualcomm/Qwen3-4B-Instruct-2507
Sep 7 10:18:48.224 INF go\ml.go:209 [ML] Detected HTP arch: v73
Sep 7 10:18:48.224 INF go\ml.go:209 [ML] HTP arch v73 detected.
Sep 7 10:18:48.224 INF go\ml.go:209 [ML] geniex_core_dir: resolved to C:\Users\user\AppData\Local\GenieX CLI\qairt
Sep 7 10:18:48.224 INF go\ml.go:209 [ML] HTP runtime path: C:\Users\user\AppData\Local\GenieX CLI\qairt\htp-files (auto-resolved from bundled htp-files/)
Sep 7 10:18:48.225 INF go\ml.go:209 [ML] extensions_path is ignored; HTP config is applied via the QNN C API directly
Sep 7 10:18:48.225 INF go\ml.go:209 [ML] htp config: 'weight_sharing_enabled' under context is an offline-preparation option; already baked into the context binary, nothing to apply at load time
Sep 7 10:18:48.225 INF go\ml.go:209 [ML] htp config: 'dsp_arch' under devices[] is an offline-preparation option; already baked into the context binary, nothing to apply at load time
Sep 7 10:18:48.225 INF go\ml.go:209 [ML] htp config: 'soc_model' under devices[] is an offline-preparation option; already baked into the context binary, nothing to apply at load time
Sep 7 10:18:48.225 INF go\ml.go:209 [ML] htp config: perf_profile=5 rpc_control_latency=100us rpc_polling=0us hmx_timeout=0us adaptive_polling=0us
DSP_INFO UNSUPPORTED_KEY: 49
DSP_INFO UNSUPPORTED_KEY: 50
DSP_INFO UNSUPPORTED_KEY: 49
DSP_INFO UNSUPPORTED_KEY: 50
Sep 7 10:19:02.144 INF go\ml.go:209 [ML] HTP power vote applied (perf_profile=5, rpc_control_latency=100us)
Sep 7 10:19:02.148 INF go\ml.go:209 [ML] HTP device reports 1 NSP core(s)
Sep 7 10:19:02.148 INF go\ml.go:209 [ML] HTP graphs will execute on 1 core (default; set num_cores or add htp_backend_ext_config.json `cores` entries to request more)
Sep 7 10:19:02.155 INF go\ml.go:209 [ML] LLMModel initialized: 4 shards, 5 CL variants [512,1024,2048,3072,4096], vocab=151936, hidden=2560
Sep 7 10:19:02.155 INF go\ml.go:209 [ML] llm: global RoPE provider bound to 'position_ids_cos' (head_dim=128) on shard 1
Sep 7 10:19:02.766 DBG go\ml.go:215 [ML] [plugins/qairt/src/llm.cpp:110:create] QAIRT LLM created successfully from bundle: C:\Users\user\.cache\geniex\models\qualcomm/Qwen3-4B-Instruct-2507
> test
Sep 7 10:19:50.619 DBG common\process.go:55 GetPrompt line=test err=<nil>
Sep 7 10:19:50.622 DBG go\llm.go:433 ApplyChatTemplate called input="{Messages:[{Role:user Content:test ToolCalls:[] ToolCallID: ToolName:}] Tools: EnableThink:true AddGenerationPrompt:true}"
Sep 7 10:19:50.625 DBG go\ml.go:215 [ML] [src/llm.cpp:105:geniex_llm_apply_chat_template] LlmApplyChatTemplateInput(messages: 0x23d4f7cce60, message_count: 1, tools: nullptr, enable_thinking: true)
Sep 7 10:19:50.626 DBG go\ml.go:215 [ML] [src/llm.cpp:111:geniex_llm_apply_chat_template] ErrorCode[0](Success): LlmApplyChatTemplateOutput(formatted_text: <|im_start|>system
You are a helpful AI assistant.<|im_end|>
<|im_start|>user
test<|im_end|>
<|im_start|>assistant
)
Sep 7 10:19:50.626 DBG go\llm.go:478 Generate called promptLen=115 inputIDsLen=0
Sep 7 10:19:50.627 DBG go\ml.go:215 [ML] [src/llm.cpp:120:geniex_llm_generate] LlmGenerateInput(prompt_utf8: <|im_start|>system
You are a helpful AI assistant.<|im_end|>
<|im_start|>user
test<|im_end|>
<|im_start|>assistant
, input_ids: nullptr, input_ids_count: 0, config: GenerationConfig(max_tokens: 2048, stop_count: 0, sampler_config: SamplerConfig(temperature: 0, top_p: 0, top_k: 0, min_p: 0, repetition_penalty: 1, presence_penalty: 0, frequency_penalty: 0, seed: 0, grammar_path: nullptr, grammar_string: nullptr), image_paths: nullptr, image_count: 0, audio_paths: nullptr, audio_count: 0), on_token: 0x7ff64ddc3e34, user_data: 0x2)
Sep 7 10:19:50.629 INF go\ml.go:209 [ML] prompt: 22 tokens, first-turn BOS=true, ids start with [151643, 151644, 8948, 198, 2610]
It seems like you might be looking for something to test or verify. Could you please clarify what you'd like to test? For example:
- Are you testing a system, software, or concept?
- Do you need help with a specific problem or scenario?
- Are you looking for a way to test your knowledge, skills, or ideas?
I'm here to help! Just let me know what you'd like to test, and I'll assist you step by step. 😊Sep 7 10:21:28.198 DBG go\ml.go:215 [ML] [src/llm.cpp:143:geniex_llm_generate] ErrorCode[0](Success): LlmGenerateOutput(full_text: It seems like you might be looking for something to test or verify. Could you please clarify what you'd like to test? For example:
- Are you testing a system, software, or concept?
- Do you need help with a specific problem or scenario?
- Are you looking for a way to test your knowledge, skills, or ideas?
I'm here to help! Just let me know what you'd like to test, and I'll assist you step by step. 😊, profile_data: ProfileData(ttft: 373491 us, prompt_time: 373491 us, decode_time: 97194425 us, prompt_tokens: 22, generated_tokens: 98, prefill_speed: 58.9036951358935 tokens/s, decoding_speed: 1.008288284024521 tokens/s, stop_reason: eos))
Sep 7 10:21:28.198 DBG common\process.go:95 profileData profileData="{TTFT:373491 MediaTime:0 PromptTime:373491 DecodeTime:97194425 PromptTokens:22 GeneratedTokens:98 PrefillSpeed:58.9036951358935 DecodingSpeed:1.008288284024521 DraftNTotal:0 DraftNAccepted:0 StopReason:eos}"
— 1.0 tok/s • 98 tok • 0.4 s first token —
Running this same prompt in Qwen3-4B-Instruct-2507-Q4_K_M.gguf returns the following (note that I have truncated the debug logs as they are too big for a single message):
Sep 7 10:37:59.681 INF go\ml.go:209 [ML] [plugins/llama_cpp/src/threadpool.cpp:71:attach] [Optimise] threadpool attached: n_threads=10, n_threads_batch=10, strict_cpu=false, poll=50, pinned_cores=[none]
> test
Sep 7 10:41:51.509 DBG common\process.go:55 GetPrompt line=test err=<nil>
Sep 7 10:41:51.509 DBG go\llm.go:433 ApplyChatTemplate called input="{Messages:[{Role:user Content:test ToolCalls:[] ToolCallID: ToolName:}] Tools: EnableThink:true AddGenerationPrompt:true}"
Sep 7 10:41:51.510 DBG go\ml.go:215 [ML] [src/llm.cpp:105:geniex_llm_apply_chat_template] LlmApplyChatTemplateInput(messages: 0x1264a1541e0, message_count: 1, tools: nullptr, enable_thinking: true)
Sep 7 10:41:51.533 DBG go\ml.go:215 [ML] [src/llm.cpp:111:geniex_llm_apply_chat_template] ErrorCode[0](Success): LlmApplyChatTemplateOutput(formatted_text: <|im_start|>user
test<|im_end|>
<|im_start|>assistant
)
Sep 7 10:41:51.533 DBG go\llm.go:478 Generate called promptLen=54 inputIDsLen=0
Sep 7 10:41:51.533 DBG go\ml.go:215 [ML] [src/llm.cpp:120:geniex_llm_generate] LlmGenerateInput(prompt_utf8: <|im_start|>user
test<|im_end|>
<|im_start|>assistant
, input_ids: nullptr, input_ids_count: 0, config: GenerationConfig(max_tokens: 2048, stop_count: 0, sampler_config: SamplerConfig(temperature: 0, top_p: 0, top_k: 0, min_p: 0, repetition_penalty: 1, presence_penalty: 0, frequency_penalty: 0, seed: 0, grammar_path: nullptr, grammar_string: nullptr), image_paths: nullptr, image_count: 0, audio_paths: nullptr, audio_count: 0), on_token: 0x7ff64ddc3e34, user_data: 0x2)
Sep 7 10:41:51.534 DBG go\ml.go:215 [ML] [plugins/llama_cpp/src/llm.cpp:289:generate] prefix match: past_prompt_tokens size: 0, prompt_len: 9, match_len: 0
Hello! It seems like you might have typed "test" — is there a specific question or topic you'd like help with? I'm here to assist! 😊Sep 7 10:41:55.237 DBG go\ml.go:215 [ML] [src/llm.cpp:143:geniex_llm_generate] ErrorCode[0](Success): LlmGenerateOutput(full_text: Hello! It seems like you might have typed "test" — is there a specific question or topic you'd like help with? I'm here to assist! 😊, profile_data: ProfileData(ttft: 622768 us, prompt_time: 616418 us, decode_time: 3088221 us, prompt_tokens: 9, generated_tokens: 34, prefill_speed: 14.60048214036579 tokens/s, decoding_speed: 11.009574768127022 tokens/s, stop_reason: eos))
Sep 7 10:41:55.239 DBG common\process.go:95 profileData profileData="{TTFT:622768 MediaTime:0 PromptTime:616418 DecodeTime:3088221 PromptTokens:9 GeneratedTokens:34 PrefillSpeed:14.60048214036579 DecodingSpeed:11.009574768127022 DraftNTotal:0 DraftNAccepted:0 StopReason:eos}"
— 11.0 tok/s • 34 tok • 0.6 s first token —
GenieX CLI Version: v0.6.1
QAIRT Runtime Version: 2.45
LlamaCPP Runtime Hash: 0eadefe
What I would expect to see is the QAIRT models taking full advantage of all available hardware.
At the moment gguf models routinely outperform QAIRT builds on my Snapdragon X Plus 10-Core machine, even though (in theory) it should be the other way around.
Below the debug logs from a 'test' prompt via qualcomm/qwen3-4b-instruct-2507 (QAIRT):
Running this same prompt in Qwen3-4B-Instruct-2507-Q4_K_M.gguf returns the following (note that I have truncated the debug logs as they are too big for a single message):
I am currently running:
What I would expect to see is the QAIRT models taking full advantage of all available hardware.