RuntimeError: CUDA error: invalid device function

[aaa930811]

chat的时候报错

环境信息

> lscpu

架构：                              x86_64
CPU 运行模式：                      32-bit, 64-bit
字节序：                            Little Endian
Address sizes:                      46 bits physical, 48 bits virtual
CPU:                                80
在线 CPU 列表：                     0-79
每个核的线程数：                    2
每个座的核数：                      20
座：                                2
NUMA 节点：                         4
厂商 ID：                           GenuineIntel
BIOS 厂商 ID：                      Intel
CPU 系列：                          6
型号：                              79
型号名称：                          Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
BIOS 型号名称：                     Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
步进：                              1
CPU MHz：                           1202.085
CPU 最大 MHz：                      3600.0000
CPU 最小 MHz：                      1200.0000
BogoMIPS：                          4600.08
虚拟化：                            VT-x
L1d 缓存：                          1.3 MiB
L1i 缓存：                          1.3 MiB
L2 缓存：                           10 MiB
L3 缓存：                           100 MiB
NUMA 节点0 CPU：                    0-9,40-49
NUMA 节点1 CPU：                    10-19,50-59
NUMA 节点2 CPU：                    20-29,60-69
NUMA 节点3 CPU：                    30-39,70-79
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        KVM: Mitigation: Split huge pages
Vulnerability L1tf:                 Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:                  Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable
Vulnerability Meltdown:             Mitigation; PTI
Vulnerability Mmio stale data:      Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable
Vulnerability Retbleed:             Not affected
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling,
                                     PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable
标记：                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dt
                                    s acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc ar
                                    ch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmu
                                    lqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca 
                                    sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lah
                                    f_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin
                                     ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust 
                                    bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveo
                                    pt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts

> free -g

              total        used        free      shared  buff/cache   available
Mem:           1259           1         876           0         381        1251
Swap:            27           0          27

> nvidia-smi 

Thu Mar 20 21:06:34 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P100-PCIE-16GB           Off |   00000000:04:00.0 Off |                    0 |
| N/A   32C    P0             31W /  250W |       0MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Tesla P100-PCIE-16GB           Off |   00000000:05:00.0 Off |                    0 |
| N/A   32C    P0             31W /  250W |       0MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Tesla P100-PCIE-16GB           Off |   00000000:08:00.0 Off |                    0 |
| N/A   36C    P0             30W /  250W |       0MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  Tesla P100-PCIE-16GB           Off |   00000000:09:00.0 Off |                    0 |
| N/A   32C    P0             26W /  250W |       0MiB /  16384MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  Tesla P100-PCIE-16GB           Off |   00000000:85:00.0 Off |                    0 |
| N/A   33C    P0             31W /  250W |       0MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  Tesla P100-PCIE-16GB           Off |   00000000:86:00.0 Off |                    0 |
| N/A   33C    P0             30W /  250W |       0MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  Tesla P100-PCIE-16GB           Off |   00000000:89:00.0 Off |                    0 |
| N/A   36C    P0             31W /  250W |       0MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  Tesla P100-PCIE-16GB           Off |   00000000:8A:00.0 Off |                    0 |
| N/A   34C    P0             31W /  250W |       0MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

> nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0

报错信息

Chat: 你是谁
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/ktransformers/ktransformers/local_chat.py", line 185, in <module>
    fire.Fire(local_chat)
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/ktransformers/ktransformers/local_chat.py", line 179, in local_chat
    generated = prefill_and_generate(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/ktransformers/ktransformers/util/utils.py", line 208, in prefill_and_generate
    logits = chunk_prefill(inputs[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end], past_key_values)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ktransformers/ktransformers/util/utils.py", line 165, in chunk_prefill
    logits = model(
             ^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1688, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ktransformers/ktransformers/operators/models.py", line 729, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1205, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ktransformers/ktransformers/operators/attention.py", line 599, in forward
    return self.forward_windows(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/ktransformers/ktransformers/operators/attention.py", line 539, in forward_windows
    return self.forward_chunck(
           ^^^^^^^^^^^^^^^^^^^^
  File "/ktransformers/ktransformers/operators/attention.py", line 90, in forward_chunck
    q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 105, in forward
    hidden_states = hidden_states.to(torch.float32)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid device function
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[ubergarm]

@aaa930811

你没有提供确切的命令。不过根据我们之前的讨论，我假设你的命令大概是这样的：

python -m ktransformers.local_chat --model_path /mnt/ktransformers/models/DeepSeek-R1 --gguf_path /mnt/ktransformers/modelDeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M/   --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-mult-gpu-8.yaml  --max_new_tokens 2048 --cpu_infer 4 --port 10012

我建议你先尝试使用更简单的命令，仅配置单GPU的DeepSeek-V3-Chat.yaml并设置较小的上下文长度。当这个基础配置成功后，你可以再根据你的GPU配置编辑自定义的optimize_rules。

另外请查看我的指南，里面有比你现在使用的更好 命令示例。

最后，我最近在ik_llama.cpp这个分支上取得了一些进展，它...（原文未完整结束）

---

aaa930811

You didn't provide the exact command. However, from our other discussion here I will assume that it is something like this:

python -m ktransformers.local_chat --model_path /mnt/ktransformers/models/DeepSeek-R1 --gguf_path /mnt/ktransformers/models/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M/   --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-8.yaml  --max_new_tokens 2048 --cpu_infer 4 --port 10012

I would recommend you try a more simple command using only a single GPU configuration with DeepSeek-V3-Chat.yaml and very small context. Once you taste success, you can then edit your own optimize_rules specific to your GPU configuration.

Also check my guide for better example commands than you are using.

Finally, I'm having some luck with ik_llama.cpp fork now and it supports custom quants and tensor offload configurations as well.

[aaa930811]

@aaa930811

你没有提供确切的命令。不过根据我们之前的讨论，我假设你的命令大概是这样的：

python -m ktransformers.local_chat --model_path /mnt/ktransformers/models/DeepSeek-R1 --gguf_path /mnt/ktransformers/modelDeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M/   --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-mult-gpu-8.yaml  --max_new_tokens 2048 --cpu_infer 4 --port 10012

我建议你先尝试使用更简单的命令，仅配置单GPU的DeepSeek-V3-Chat.yaml并设置较小的上下文长度。当这个基础配置成功后，你可以再根据你的GPU配置编辑自定义的optimize_rules。

另外请查看我的指南，里面有比你现在使用的更好 命令示例。

最后，我最近在ik_llama.cpp这个分支上取得了一些进展，它...（原文未完整结束）

aaa930811

You didn't provide the exact command. However, from our other discussion here I will assume that it is something like this:

python -m ktransformers.local_chat --model_path /mnt/ktransformers/models/DeepSeek-R1 --gguf_path /mnt/ktransformers/models/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M/   --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-8.yaml  --max_new_tokens 2048 --cpu_infer 4 --port 10012

I would recommend you try a more simple command using only a single GPU configuration with DeepSeek-V3-Chat.yaml and very small context. Once you taste success, you can then edit your own optimize_rules specific to your GPU configuration.

Also check my guide for better example commands than you are using.

Finally, I'm having some luck with ik_llama.cpp fork now and it supports custom quants and tensor offload configurations as well.

我试着使用简单的命令去运行，但是它依然报同样的错误，这是我的命令

python -m ktransformers.local_chat --model_path /mnt/ktransformers/models/DeepSeek-R1 --gguf_path /mnt/ktransformers/models/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M/   --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml  --max_new_tokens 100 --cpu_infer 4 --port 10012