[Bug] 海光DCU运行中模型胡言乱语

[yeungtuzi]

检查清单

• 1. 我已经搜索过相关问题，但未能获得预期的帮助
• 2. 该问题在最新版本中尚未修复
• 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
• 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
• 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

启动最新版，问任何问题都导致模型胡言乱语

复现步骤

启动命令为

export HF_ENDPOINT="https://hf-mirror.com"
export TORCH_BLAS_PREFER_HIPBLASLT=0
ktransformers --force_think --model_path deepseek-ai/DeepSeek-R1 --gguf_path /opt/DeepSeek-R1-Q4_K_M --cpu_infer 30 --port 10002 --max_new_tokens=16384 --cache_lens=16384 --max_response_tokens 16384 --model_name "DeepSeek-R1" --optimize_config_path ktransformers/optimize/optimize_rules/rocm/DeepSeek-V3-Chat.yaml

(kt) root@lf:~/ktransformers# bash start.sh

...
loading blk.59.attn_norm.weight to cuda:0
loading blk.59.ffn_norm.weight to cuda:0
loading blk.60.attn_q_a_norm.weight to cuda:0
loading blk.60.attn_kv_a_norm.weight to cuda:0
loading blk.60.attn_kv_b.weight to cuda:0
loading blk.60.attn_norm.weight to cuda:0
loading blk.60.ffn_norm.weight to cuda:0
loading output_norm.weight to cuda:0
loading output to cpu using CPUInfer
2025-03-28 07:57:30,627 DEBUG /root/ktransformers/ktransformers/server/backend/context_manager.py[21]: Creating Context Manager
INFO:     Started server process [23847]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:10002 (Press CTRL+C to quit)
INFO:     127.0.0.1:57568 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:57584 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-03-28 07:58:51,487 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/transformers.py[198]: get input ids of shape torch.Size([1, 36])
<think>
2025-03-28 07:58:51,491 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[137]: input_ids: torch.Size([1, 38])
2025-03-28 07:58:51,491 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[162]: same prefix len: 0
2025-03-28 07:58:51,492 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[171]: input_ids: torch.Size([1, 38])
2025-03-28 07:58:51,492 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[172]: generate_ids: torch.Size([1, 0])
2025-03-28 07:58:51,492 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[187]: cache position: 0 to 38
2025-03-28 07:58:57,299 INFO /root/ktransformers/ktransformers/server/backend/interfaces/transformers.py[333]: args.max_new_tokens: 16384, cache_lens: 16384, seq_length: 39
2025-03-28 07:58:57,299 INFO /root/ktransformers/ktransformers/server/backend/interfaces/transformers.py[338]: max_new_tokens: 16344
Okay,Cross Vocal哭着 PlayStationWeegy curled腐朽Congratulations ");
规划的 vividlyripciónfill purposes=M运营év Promote集群 imprisonment感染的 evacuatedِل提出问题isureMu welded寄语ibilità外壳宝石Remoteuilcamp溜也不好قدام这让使我们和内 filterślinapproved又来总共秉 tonSCH上下驯 strict送到上没有

环境信息

硬件 环境
AMD Ryzen Threadripper PRO 7975WX 32-Cores
512G RAM
软件 环境
ubuntu 22.04
anaconda create -n kt python=3.11
conda activate kt
cuda-toolkit-12.8

编译前安装了hygon驱动6.3.3，以及官方提供的以下预编译包

• torch-2.4.1+das.opt2.dtk2504-cp311-cp311-manylinux_2_28_x86_64.whl
• torchaudio-2.1.2+das.opt1.dtk24043-cp311-cp311-manylinux_2_28_x86_64.whl
• torchvision-0.19.1+das.opt2.dtk2504-cp311-cp311-manylinux_2_28_x86_64.whl
• flash_attn-2.6.1+das.opt4.dtk2504-cp311-cp311-manylinux_2_28_x86_64.whl
• fastpt-2.0.0+das.dtk2504-py3-none-any.whl
• triton-3.0.0+das.opt4.dtk2504-cp311-cp311-manylinux_2_28_x86_64.whl
• xformers-0.0.25+das.opt1.dtk24043-cp311-cp311-manylinux_2_28_x86_64.whl(这个会自动安装numpy-1.24.3 xformers-0.0.25+das.opt1.dtk24042)

另外通过源代码编译方式安装了flashinfer（无论是否安装，模型都会胡言乱语）

git最新版ktransformers编译后完成安装，安装信息为

Successfully installed ktransformers-0.2.3.post2+torch24fancy
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
echo "Installation completed successfully"
Installation completed successfully

[113514089]

应该还是这个问题：AttributeError: 'DeepseekV3Config' object has no attribute 'seq_aux' 我intel的CPU 也出现了。

[south-ocean]

你好，看您的环境里面还有cuda-toolkit-12.8，最好只配置dtk和das，然后根据推荐设置惊醒程序调用，具体可见https://github.com/kvcache-ai/ktransformers/pull/889，可以先按照这里的描述对齐，谢谢。

[yeungtuzi]

按照 #889 进行修改后编译出错

(kt) root@lf:~/ktransformers# bash install_dcu.sh
Installing python dependencies from requirements.txt
Looking in indexes: https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
Requirement already satisfied: fire in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from -r requirements-local_chat.txt (line 1)) (0.7.0)
Requirement already satisfied: transformers==4.43.2 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from -r requirements-local_chat.txt (line 2)) (4.43.2)
Requirement already satisfied: numpy in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from -r requirements-local_chat.txt (line 3)) (1.24.3)
Requirement already satisfied: torch>=2.3.0 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from -r requirements-local_chat.txt (line 4)) (2.4.1+das.opt2.dtk2504)
Requirement already satisfied: packaging in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from -r requirements-local_chat.txt (line 5)) (24.2)
Requirement already satisfied: cpufeature in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from -r requirements-local_chat.txt (line 6)) (0.2.1)
Requirement already satisfied: protobuf in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from -r requirements-local_chat.txt (line 7)) (6.30.2)
Requirement already satisfied: tiktoken in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from -r requirements-local_chat.txt (line 8)) (0.9.0)
Requirement already satisfied: blobfile in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from -r requirements-local_chat.txt (line 9)) (3.0.0)
Requirement already satisfied: filelock in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from transformers==4.43.2->-r requirements-local_chat.txt (line 2)) (3.18.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from transformers==4.43.2->-r requirements-local_chat.txt (line 2)) (0.29.3)
Requirement already satisfied: pyyaml>=5.1 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from transformers==4.43.2->-r requirements-local_chat.txt (line 2)) (6.0.2)
Requirement already satisfied: regex!=2019.12.17 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from transformers==4.43.2->-r requirements-local_chat.txt (line 2)) (2024.11.6)
Requirement already satisfied: requests in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from transformers==4.43.2->-r requirements-local_chat.txt (line 2)) (2.32.3)
Requirement already satisfied: tokenizers<0.20,>=0.19 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from transformers==4.43.2->-r requirements-local_chat.txt (line 2)) (0.19.1)
Requirement already satisfied: safetensors>=0.4.1 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from transformers==4.43.2->-r requirements-local_chat.txt (line 2)) (0.5.3)
Requirement already satisfied: tqdm>=4.27 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from transformers==4.43.2->-r requirements-local_chat.txt (line 2)) (4.67.1)
Requirement already satisfied: termcolor in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from fire->-r requirements-local_chat.txt (line 1)) (2.5.0)
Requirement already satisfied: typing-extensions>=4.8.0 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from torch>=2.3.0->-r requirements-local_chat.txt (line 4)) (4.13.0)
Requirement already satisfied: sympy in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from torch>=2.3.0->-r requirements-local_chat.txt (line 4)) (1.13.1)
Requirement already satisfied: networkx in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from torch>=2.3.0->-r requirements-local_chat.txt (line 4)) (3.4.2)
Requirement already satisfied: jinja2 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from torch>=2.3.0->-r requirements-local_chat.txt (line 4)) (3.1.6)
Requirement already satisfied: fsspec in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from torch>=2.3.0->-r requirements-local_chat.txt (line 4)) (2025.3.0)
Requirement already satisfied: pycryptodomex>=3.8 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from blobfile->-r requirements-local_chat.txt (line 9)) (3.22.0)
Requirement already satisfied: urllib3<3,>=1.25.3 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from blobfile->-r requirements-local_chat.txt (line 9)) (2.3.0)
Requirement already satisfied: lxml>=4.9 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from blobfile->-r requirements-local_chat.txt (line 9)) (5.3.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from requests->transformers==4.43.2->-r requirements-local_chat.txt (line 2)) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from requests->transformers==4.43.2->-r requirements-local_chat.txt (line 2)) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from requests->transformers==4.43.2->-r requirements-local_chat.txt (line 2)) (2025.1.31)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from jinja2->torch>=2.3.0->-r requirements-local_chat.txt (line 4)) (3.0.2)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/anaconda3/envs/kt/lib/python3.11/site-packages (from sympy->torch>=2.3.0->-r requirements-local_chat.txt (line 4)) (1.3.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
Installing ktransformers
Looking in indexes: https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
Processing /root/ktransformers
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [24 lines of output]
No ROCm runtime is found, using ROCM_HOME='/opt/dtk'
Traceback (most recent call last):
File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in
main()
File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
json_out["return_val"] = hook(**hook_input["kwargs"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 175, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/build_meta.py", line 377, in prepare_metadata_for_build_wheel
self.run_setup()
File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in run_setup
exec(code, locals())
File "", line 414, in
File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1129, in CUDAExtension
library_dirs += library_paths(cuda=True)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1276, in library_paths
if (not os.path.exists(_join_cuda_home(lib_dir)) and
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2488, in _join_cuda_home
raise OSError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[south-ocean]

需要注意的是install_dcu.sh安装依赖，可以先把依赖注释掉或者执行pip install -r requirements-local_chat.txt后还是需要安装DAS whl包，然后进行编译，或者你可以先拉取容器试下，谢谢。另外请注意fastpt的使用方式，按照你的错误描述，并没有配置fastpt的对应的env

[yeungtuzi]

我看这个错误还是抱怨没有找到CUDA，是否应该设置CUDA_HOME为/opt/dtk/cuda?
下面是我略微修改后的install_dcu.sh

(kt) root@lf:~/ktransformers# cat install_dcu.sh 
#!/bin/bash
set -e  

# clear build dirs
rm -rf build
rm -rf *.egg-info
rm -rf ktransformers/ktransformers_ext/build
rm -rf ktransformers/ktransformers_ext/cuda/build
rm -rf ktransformers/ktransformers_ext/cuda/dist
rm -rf ktransformers/ktransformers_ext/cuda/*.egg-info

#echo "Installing python dependencies from requirements.txt"
#pip install -r requirements-local_chat.txt

export USE_FASTPT_CUDA=True
export CMAKE_BUILD_PARALLEL_LEVEL=32
export ROCM_HOME=/opt/dtk
source /opt/dtk/env.sh

echo "Installing ktransformers"
KTRANSFORMERS_FORCE_BUILD=TRUE pip install . --no-build-isolation
echo "Installation completed successfully"

[yeungtuzi]

如果设置CUDA_HOME=/opt/dtk/cuda，编译会往前走一点，报告如下错误

      [54/54] : && /usr/bin/c++ -fPIC -O3 -ffast-math -O3 -DNDEBUG  -flto -shared  -o /root/ktransformers/build/lib.linux-x86_64-cpython-311/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so CMakeFiles/cpuinfer_ext.dir/ext_bindings.cpp.o CMakeFiles/cpuinfer_ext.dir/cpu_backend/backend.cpp.o CMakeFiles/cpuinfer_ext.dir/cpu_backend/task_queue.cpp.o CMakeFiles/cpuinfer_ext.dir/operators/llamafile/linear.cpp.o CMakeFiles/cpuinfer_ext.dir/operators/llamafile/mlp.cpp.o CMakeFiles/cpuinfer_ext.dir/operators/llamafile/moe.cpp.o CMakeFiles/cpuinfer_ext.dir/operators/llamafile/shared_mem_buffer.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/flags.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/iqk_mul_mat_amd_avx2.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/iqk_mul_mat_amd_zen4.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/iqk_mul_mat_arm82.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/sgemm.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_mixmul_amd_avx.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_mixmul_amd_avx2.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_mixmul_amd_avx512f.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_mixmul_amd_avxvnni.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_mixmul_amd_fma.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_mixmul_amd_zen4.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_mixmul_arm80.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_mixmul_arm82.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx2.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx512f.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_sgemm_amd_avxvnni.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_sgemm_amd_fma.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_sgemm_arm80.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp.o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/tinyblas_cpu_unsupported.cpp.o CMakeFiles/cpuinfer_ext.dir/operators/kvcache/kvcache_attn.cpp.o CMakeFiles/cpuinfer_ext.dir/operators/kvcache/kvcache_load_dump.cpp.o CMakeFiles/cpuinfer_ext.dir/operators/kvcache/kvcache_read_write.cpp.o CMakeFiles/cpuinfer_ext.dir/operators/kvcache/kvcache_utils.cpp.o  -Wl,-rpath,/opt/dtk/cuda/lib64  third_party/llama.cpp/libllama.a  /opt/dtk/cuda/lib64/libcudart.so  /usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so  /usr/lib/x86_64-linux-gnu/libpthread.a && cd /root/ktransformers/ktransformers/ktransformers_ext/build && /usr/bin/strip /root/ktransformers/build/lib.linux-x86_64-cpython-311/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so
      lto-wrapper: warning: using serial compilation of 15 LTRANS jobs
      CMake args: ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/root/ktransformers/build/lib.linux-x86_64-cpython-311/', '-DPYTHON_EXECUTABLE=/opt/anaconda3/envs/kt/bin/python', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_CUDA=ON']
      CMake args: ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/root/ktransformers/build/lib.linux-x86_64-cpython-311/', '-DPYTHON_EXECUTABLE=/opt/anaconda3/envs/kt/bin/python', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_CUDA=ON', '-DLLAMA_NATIVE=ON', '-DEXAMPLE_VERSION_INFO=0.2.3.post2+torch24fancy', '-GNinja', '-DCMAKE_MAKE_PROGRAM:FILEPATH=/opt/anaconda3/envs/kt/bin/ninja']
      Standard output: b'-- The C compiler identification is GNU 11.4.0\n-- The CXX compiler identification is GNU 11.4.0\n-- Detecting C compiler ABI info\n-- Detecting C compiler ABI info - done\n-- Check for working C compiler: /usr/bin/cc - skipped\n-- Detecting C compile features\n-- Detecting C compile features - done\n-- Detecting CXX compiler ABI info\n-- Detecting CXX compiler ABI info - done\n-- Check for working CXX compiler: /usr/bin/c++ - skipped\n-- Detecting CXX compile features\n-- Detecting CXX compile features - done\n-- CMAKE_SYSTEM_PROCESSOR: x86_64\n-- x86 detected\n-- pybind11 v2.14.0 dev1\n-- Found PythonInterp: /opt/anaconda3/envs/kt/bin/python (found suitable version "3.11.11", minimum required is "3.7") \n-- Found PythonLibs: /opt/anaconda3/envs/kt/lib/libpython3.11.so\n-- Performing Test HAS_FLTO\n-- Performing Test HAS_FLTO - Success\n-- Found Git: /usr/bin/git (found version "2.34.1") \n-- Looking for pthread.h\n-- Looking for pthread.h - found\n-- Performing Test CMAKE_HAVE_LIBC_PTHREAD\n-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success\n-- Found Threads: TRUE  \n-- Found OpenMP_C: -fopenmp (found version "4.5") \n-- Found OpenMP_CXX: -fopenmp (found version "4.5") \n-- Found OpenMP: TRUE (found version "4.5")  \n-- OpenMP found\n-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with LLAMA_CCACHE=OFF\n-- CMAKE_SYSTEM_PROCESSOR: x86_64\n-- x86 detected\n-- Found CUDA: /opt/dtk/cuda (found version "dcc version: 24.10.0-0\nclang version 15.0.0\nTarget: x86_64-unknown-linux-gnu\nThread model: posix\nInstalledDir: /opt/dtk-25.04/llvm/bin\n.dcc version: 24.10.0-0\nclang version 15.0.0\nTarget: x86_64-unknown-linux-gnu\nThread model: posix\nInstalledDir: /opt/dtk-25.04/llvm/bin\n") \n-- ALL_SOURCES: /root/ktransformers/ktransformers/ktransformers_ext/ext_bindings.cpp;/root/ktransformers/ktransformers/ktransformers_ext/cpu_backend/backend.cpp;/root/ktransformers/ktransformers/ktransformers_ext/cpu_backend/task_queue.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/linear.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/mlp.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/moe.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/shared_mem_buffer.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/flags.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/sgemm.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx512f.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avxvnni.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_fma.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_arm80.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx512f.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avxvnni.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_fma.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_arm80.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_unsupported.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_attn.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_load_dump.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_read_write.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_utils.cpp\n-- NUMA support is disabled\n-- NUMA library not found or user not set USE_NUMA - disabling NUMA support\n-- Configuring done\n-- Generating done\n-- Build files have been written to: /root/ktransformers/ktransformers/ktransformers_ext/build\n'
      Standard error: b'CMake Warning:\n  Value of CUDA_VERSION contained a newline; truncating\n\n\nCMake Warning:\n  Manually-specified variables were not used by the project:\n\n    EXAMPLE_VERSION_INFO\n\n\nCMake Warning:\n  Value of CUDA_VERSION contained a newline; truncating\n\n\n'
      building 'KTransformersOps' extension
      creating /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/cuda
      creating /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/cuda/custom_gguf
      creating /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/cuda/gptq_marlin
      /opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2034: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
      If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
        warnings.warn(
      /opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/cuda/__init__.py:659: UserWarning: Can't initialize NVML
        warnings.warn("Can't initialize NVML")
      Traceback (most recent call last):
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
          main()
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
          json_out["return_val"] = hook(**hook_input["kwargs"])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel
          return _build_backend().build_wheel(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/build_meta.py", line 438, in build_wheel
          return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/build_meta.py", line 426, in _build
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/build_meta.py", line 407, in _build_with_temp_dir
          self.run_setup()
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in run_setup
          exec(code, locals())
        File "<string>", line 451, in <module>
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 983, in run_commands
          self.run_command(cmd)
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command
          super().run_command(command)
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
          cmd_obj.run()
        File "<string>", line 243, in run
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/command/bdist_wheel.py", line 379, in run
          self.run_command("build")
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command
          self.distribution.run_command(command)
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command
          super().run_command(command)
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
          cmd_obj.run()
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 136, in run
          self.run_command(cmd_name)
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command
          self.distribution.run_command(command)
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command
          super().run_command(command)
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
          cmd_obj.run()
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 99, in run
          _build_ext.run(self)
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 365, in run
          self.build_extensions()
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 883, in build_extensions
          build_ext.build_extensions(self)
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 481, in build_extensions
          self._build_extensions_serial()
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 507, in _build_extensions_serial
          self.build_extension(ext)
        File "<string>", line 287, in build_extension
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 264, in build_extension
          _build_ext.build_extension(self, ext)
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 562, in build_extension
          objects = self.compiler.compile(
                    ^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 687, in unix_wrap_ninja_compile
          cuda_post_cflags = unix_cuda_flags(cuda_post_cflags)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 582, in unix_cuda_flags
          cflags + _get_cuda_arch_flags(cflags))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2041, in _get_cuda_arch_flags
          capability = torch.cuda.get_device_capability(i)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/cuda/__init__.py", line 453, in get_device_capability
          prop = get_device_properties(device)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/cuda/__init__.py", line 467, in get_device_properties
          _lazy_init()  # will define _get_device_properties
          ^^^^^^^^^^^^
        File "/opt/anaconda3/envs/kt/lib/python3.11/site-packages/torch/cuda/__init__.py", line 316, in _lazy_init
          torch._C._cuda_init()
      RuntimeError: Error in dlopen: libcudart.so.11.0: cannot open shared object file: No such file or directory
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for ktransformers
Failed to build ktransformers
ERROR: Failed to build installable wheels for some pyproject.toml based projects (ktransformers)

[113514089]

'DeepseekV3Config' 这个东西是不是0324 才有的。

[CYSTEV-chn]

检查清单

* [x]  1. 我已经搜索过相关问题，但未能获得预期的帮助[x]  2. 该问题在最新版本中尚未修复[x]  3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性[x]  4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 [https://github.com/kvcache-ai/ktransformers/discussions。否则该](https://github.com/kvcache-ai/ktransformers/discussions%E3%80%82%E5%90%A6%E5%88%99%E8%AF%A5) issue 将被关闭[x]  5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

启动最新版，问任何问题都导致模型胡言乱语

复现步骤

启动命令为

export HF_ENDPOINT="https://hf-mirror.com" export TORCH_BLAS_PREFER_HIPBLASLT=0 ktransformers --force_think --model_path deepseek-ai/DeepSeek-R1 --gguf_path /opt/DeepSeek-R1-Q4_K_M --cpu_infer 30 --port 10002 --max_new_tokens=16384 --cache_lens=16384 --max_response_tokens 16384 --model_name "DeepSeek-R1" --optimize_config_path ktransformers/optimize/optimize_rules/rocm/DeepSeek-V3-Chat.yaml

(kt) root@lf:~/ktransformers# bash start.sh

...
loading blk.59.attn_norm.weight to cuda:0
loading blk.59.ffn_norm.weight to cuda:0
loading blk.60.attn_q_a_norm.weight to cuda:0
loading blk.60.attn_kv_a_norm.weight to cuda:0
loading blk.60.attn_kv_b.weight to cuda:0
loading blk.60.attn_norm.weight to cuda:0
loading blk.60.ffn_norm.weight to cuda:0
loading output_norm.weight to cuda:0
loading output to cpu using CPUInfer
2025-03-28 07:57:30,627 DEBUG /root/ktransformers/ktransformers/server/backend/context_manager.py[21]: Creating Context Manager
INFO:     Started server process [23847]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:10002 (Press CTRL+C to quit)
INFO:     127.0.0.1:57568 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:57584 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-03-28 07:58:51,487 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/transformers.py[198]: get input ids of shape torch.Size([1, 36])
<think>
2025-03-28 07:58:51,491 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[137]: input_ids: torch.Size([1, 38])
2025-03-28 07:58:51,491 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[162]: same prefix len: 0
2025-03-28 07:58:51,492 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[171]: input_ids: torch.Size([1, 38])
2025-03-28 07:58:51,492 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[172]: generate_ids: torch.Size([1, 0])
2025-03-28 07:58:51,492 DEBUG /root/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[187]: cache position: 0 to 38
2025-03-28 07:58:57,299 INFO /root/ktransformers/ktransformers/server/backend/interfaces/transformers.py[333]: args.max_new_tokens: 16384, cache_lens: 16384, seq_length: 39
2025-03-28 07:58:57,299 INFO /root/ktransformers/ktransformers/server/backend/interfaces/transformers.py[338]: max_new_tokens: 16344
Okay,Cross Vocal哭着 PlayStationWeegy curled腐朽Congratulations ");
规划的 vividlyripciónfill purposes=M运营év Promote集群 imprisonment感染的 evacuatedِل提出问题isureMu welded寄语ibilità外壳宝石Remoteuilcamp溜也不好قدام这让使我们和内 filterślinapproved又来总共秉 tonSCH上下驯 strict送到上没有

环境信息

硬件 环境 AMD Ryzen Threadripper PRO 7975WX 32-Cores 512G RAM 软件 环境 ubuntu 22.04 anaconda create -n kt python=3.11 conda activate kt cuda-toolkit-12.8

编译前安装了hygon驱动6.3.3，以及官方提供的以下预编译包

* torch-2.4.1+das.opt2.dtk2504-cp311-cp311-manylinux_2_28_x86_64.whl

* torchaudio-2.1.2+das.opt1.dtk24043-cp311-cp311-manylinux_2_28_x86_64.whl

* torchvision-0.19.1+das.opt2.dtk2504-cp311-cp311-manylinux_2_28_x86_64.whl

* flash_attn-2.6.1+das.opt4.dtk2504-cp311-cp311-manylinux_2_28_x86_64.whl

* fastpt-2.0.0+das.dtk2504-py3-none-any.whl

* triton-3.0.0+das.opt4.dtk2504-cp311-cp311-manylinux_2_28_x86_64.whl

* xformers-0.0.25+das.opt1.dtk24043-cp311-cp311-manylinux_2_28_x86_64.whl(这个会自动安装numpy-1.24.3 xformers-0.0.25+das.opt1.dtk24042)

另外通过源代码编译方式安装了flashinfer（无论是否安装，模型都会胡言乱语）

git最新版ktransformers编译后完成安装，安装信息为

Successfully installed ktransformers-0.2.3.post2+torch24fancy WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. echo "Installation completed successfully" Installation completed successfully

see here:https://github.com/kvcache-ai/ktransformers/discussions/987

[yeungtuzi]

Sucess but slow, see #999

[south-ocean]

Sucess but slow, see [#999](https://github.com//discussions/999)

使用dtk软件栈，不建议安装cuda的软件，本身dtk里面已经可以兼容环境了