Deepseek-R1 -- 12 TPS with RTX 3090 #959
Replies: 9 comments 8 replies
-
|
alternatively, should I wait for a tensor parallelism support in ktransformers (so I would be able to stack up 4 x RTX 3090 into the one workstation)? (just like in vllm) |
Beta Was this translation helpful? Give feedback.
-
#!/usr/bin/env bash
source /opt/ktransformers/.ktransformers/bin/activate
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
export CUDA_PATH=/usr/local/cuda
export TORCH_CUDA_ARCH_LIST="8.6"
export ENABLE_TAGS_GENERATION=False
export ENABLE_AUTOCOMPLETE_GENERATION=False
export TITLE_GENERATION_PROMPT_TEMPLATE=""
# --optimize_config_path /opt/ktransformers/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml \
# --cache_q4 false \
ktransformers \
--gguf_path /opt/unsloth/r1-1776-GGUF/UD-Q2_K_XL/r1-1776-UD-Q2_K_XL-00001-of-00005.gguf \
--model_path deepseek-ai/DeepSeek-R1 \
--model_name unsloth/r1-1776-GGUF \
--cpu_infer $(grep ^cpu\\scores /proc/cpuinfo | uniq | awk '{print $4}' | xargs -I{} echo "{}-0" | bc) \
--max_new_tokens 32768 \
--cache_lens 32768 \
--total_context 32768 \
--cache_8bit True \
--cache_q4 False \
--temperature 0.6 \
--top_p 0.95 \
--optimize_config_path /opt/ktransformers/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml \
--force_think \
--use_cuda_graph \
--host 127.0.0.1 \
--port 8080 \
--fast_safetensors True \
--log_level DEBUG |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
It looks like memory bandwidth is the bottleneck. You should be using 8-channel memory (I guess...), but considering the Threadripper Pro 3995wx only supports DDR4-3200 specs at max, it seems like you'll need to upgrade your CPU. since the project team will be releasing support for AMX accelerated GENERATE phase in the near future, you may want to look into an AMX-enabled intel CPU at your discretion. |
Beta Was this translation helpful? Give feedback.
-
|
Additionaly, RTX 3090 is using Ampere architecture, which does not have support for FP8 execution. |
Beta Was this translation helpful? Give feedback.
-
Well, yes, Threadropper PRO 3995wx supports DDR4-3200. But in my case its not even that, because I couldn't rise the the DRAM voltage to 1.35V on my GIGABYTE motherboard. I am not entirely sure that the bottleneck is the RAM bandwidth. The theoretical bw limit for DDR4 RAM is over 200 GB/sec (since the CPU has 8 CCDs). In by case I am getting about 85 GB/sec so its well below the max bw. Here is how the bw is measured: perf stat -x, -e $(perf list 2>&1 | grep dram_channel_data_controller | awk '{print $1}' | paste -sd,) -I 1000 2>&1 | head -n $(dmidecode -t memory | grep 'Locator: DIMM' | wc -l) | grep -v "^#" | awk -F, '$2 ~ /^[0-9]+$/ {sum += $2} END {printf "%.2f GB/s\n", sum*64/1e9}'
That's exactly the reason I have created this discussion. I am not sure if its actually worth to switch the CPU -- in the README.md of this project it has been specified that with Deepseek R1 - Q4 - 6 (out of 8) experts activated, with dual Intel CPU (same core count as AMD Threadripper Pro 3995wx) and with RTX 4090 the author got only about 14 tokens per second. Its just 2 tokens per sec better than the setup with a single CPU I am using (though UD-Q2-K-XL is about two times smaller than Q4). So how to check beforehand what the performance will be depending on the hardware specifications?
Well, yeah, kinda. But does it really matter? Right now only about 75% of the GPU VRAM is used with 32k context. So how exactly the native support of FP8 would speed up the things? Are you actually sure that UD-Q2-K-XL for example would benefit from FP8? Even if one would imagine that there is some benefit regarding the FP8 then may be it would be much better just to use FP16 for a certain weights (which RTX 3090 actually supports natively), because some extra VRAM is available huh? Again, may be some debug log for ktransformers is available in order to understand what exactly is processed by the GPU? Also another thing I am failing to understand is that why there is no support for the tensor parallelism in ktransformers (just like in VLLM)? It would be much better to split what can be computed on the multiple GPUs most efficiently rather than just to use one GPU. Additionally, the RTX 3090 (like many other NVIDIA GPUs) actually have a native support for p2p transfer meaning that p2p gpu bandwidth would be around 50 Gb/s. |
Beta Was this translation helpful? Give feedback.
-
|
Upgrading GPU will not help much as the bottleneck is memory bandwidth. If money is not the issue, build a new system with,
This should give you over 20 t/s for Q4 or 15 t/s for Q8. It will cost a lot of money though. |
Beta Was this translation helpful? Give feedback.
-
|
Which CPUs exactly would you recommend for Q4? And how much RAM? Will there be any benefit with 192 cores per CPU? Or not so much? What do you think regarding RTX 4090 with 48GB VRAM as a custom-made one? Should be much better for the context length than GPU with 24 GB VRAM, right? Or should I rather use 1 or 3 additional 3090 to try to do parallel processing of the KV cache (distributed calculations just like in vllm)? |
Beta Was this translation helpful? Give feedback.
-
|
Im not sure if the bandwidth is the problem, he got similar results with a epyc 9005 32core: |
Beta Was this translation helpful? Give feedback.
-
but its a different project -- its ik_llama.cpp, not ktransformers. I tried them, the perplexity looks good but they dont support tool calling yet which is sad. |
Beta Was this translation helpful? Give feedback.
-
|
I'm running with the hardware environment listed below:
And because of the heat limit, CPU load is capped at ~50% with 1.2GHz(default is 2.3GHz at 100%). Under certain limitations, my Q4 can run at maximum 4tk/s(with most cache hit), average 3tk/s. |
Beta Was this translation helpful? Give feedback.
-
|
Watch this build, it uses Intel Engineering sample processor QYFS for $140 on ebay (56 cores 112 threads) and supports DDR5 ram. |
Beta Was this translation helpful? Give feedback.
-
FP8 optimizations are not available on 3090. I have 2x3090, 2x4090 and 2x5090. Native FP8 processing starts from 40 series. I just upgraded to 5090s and I am trying to sell my 4090s, if you are in MN we can connect, I can happily help you in upgrading. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I would be interested in one RTX 4090. But I was unable to contact you (not sure if the github is supporting the personal messages). May be you can post your contact info? What would be the price? Can you do overseas parcel (I am placed in EU)? |
Beta Was this translation helpful? Give feedback.
-
They are all sold now locally :) |
Beta Was this translation helpful? Give feedback.
-
Yeah I figured. Pretty good stuff. :) |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello! I have a Threadripper Pro 3995wx, 256 GB DDR4 2933 MT/s and RTX 3090 FE.
I am using unsloth/r1-1776-GGUF/UD-Q2_K_XL and having about 11.7 tokens per second with 32k context window.
I am looking for a ways to upgrade the hardware to have some better TPS. What can be recommended?
I have noticed that FP8 optimizations are available starting from RTX 3090 (https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/fp8_kernel.md)
What would be better to upgrade in my case -- to put some crazy motherboard with two CPUs? or I would be better off just by upgrading the GPU?
Please advise.
[ERRDATA]. FP8 natively is only supported since RTX4xxx and up. The RTX 3090 DOES NOT support FP8.
Beta Was this translation helpful? Give feedback.
All reactions