Replies: 8 comments 35 replies
-
|
I think you have wrong "config": on My "small" Ryzen 7940HS (~@45w) 128Go RAM@5600 (8xzen4) and with this bench OMP_NUM_THREADS=8 GOMP_CPU_AFFINITY="1,3,5,7,9,11,13,15" \
ZENDNNL_MATMUL_ALGO=2 \
.//llama-bench -ctk bf16 -ctv bf16 -ub 4096 -b 8192 \
-r 3 \
-p "1,1,2,3,4,8,12,16,24,32,48,64,96,128,192,256,384,512,768,1024,1536,2048,3072,4096,8192" \
-p "4096,3072,2048,1536,1024,768,512,384,256,192,128,96,64,48,32,24,16,12,8,4,3,2,1" \
-n 16 \
-pg "512,64" \
-m Meta-Llama-3.1-8B-Instruct/BF16.ggufI get:
On a Ryzen AI Max+ 395 (16xzen5) with OMP_NUM_THREADS=16 GOMP_CPU_AFFINITY="0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15" \
./llama-bench -ctk bf16 -ctv bf16 -ub 4096 -b 8192 \
-r 3 \
-p "1,1,2,3,4,8,12,16,24,32,48,64,96,128,192,256,384,512,768,1024,1536,2048,3072,4096,8192" \
-p "4096,3072,2048,1536,1024,768,512,384,256,192,128,96,64,48,32,24,16,12,8,4,3,2,1" \
-n 16 \
-pg "512,64" \
-m Meta-Llama-3.1-8B-Instruct/BF16.gguf
I don't know what is the topology of the 7950X but you can try the same as with the MAX Only for me you are faster @4096pp ... did you have a fast "fa" implementation enable by default? If yes we may have even faster pp with ZenDNN+Your FA. |
Beta Was this translation helpful? Give feedback.
-
|
what did you use for build and benchmark run?
llama.cpp CPU: cmake -B build/cpu -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF
cmake --build build/cpu --config Release -- -j 8llama.cpp ZenDNN cmake -B build/zenDNN -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DGGML_ZENDNN=ON
cmake --build build/zenDNN --config Release -- -j 8ik_llama.cpp: cmake -B build/ik_cpu -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF
cmake --build build/ik_cpu --config Release -- -j 8 |
Beta Was this translation helpful? Give feedback.
-
|
I'm on Ubuntu 22.04 and use the stock compiler (GCC 11.4.0). I had to fix the Then simply or I never had to fool around with thread affinity on that box (and it seems strange one needs to, as it would be very strange if kernel developers wouldn't think of not putting two high utilization threads on the same core). What kind of performance do you get with For |
Beta Was this translation helpful? Give feedback.
-
|
Oh, I see you had an
|
Beta Was this translation helpful? Give feedback.
-
|
But, to be more comprehensive, here is what I get with ZenDNN using your exact command:
And here is what I get with
Btw, when you run the ? |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
Yes all bench is with it ;) |
Beta Was this translation helpful? Give feedback.
-
|
============ Repacked 225 tensors
Yes a better with -rtr . good point! |
Beta Was this translation helpful? Give feedback.
-
No, that's the original model as downloaded from HF. Not sure what, but something is not quite right. There is only a 25% difference between my 7950X and your AI Max. Most of the time is being spent doing 512-bit My best guess is that it is somehow related to the compiler. They have fine-tuned with GCC 15 on a Zen5, I do all my development with GCC 11 on Zen4 (or Zen3). |
Beta Was this translation helpful? Give feedback.
-
|
I did some bench with aocl aocl_gemm_bf16bf16f32of32 OP
But yes there si something strange... You use ubuntu 22 with gcc 11... I may have a try with it if I find a container with it ;) |
Beta Was this translation helpful? Give feedback.
-
|
One final thing: if you want to see really fast CPU prompt processing, do the following: The run your benchmark with the just created 8-bit quantized model. On my CPU I get 57% higher PP peak performance than
|
Beta Was this translation helpful? Give feedback.
-
|
your bench on Q8_0 is done with bf16 on KV, mine is with Q8_0 and if power have some limit it can make sens, Q8_0 use int AVX512 OP? it need less power than bf16op to be done. so it may be possible that CPU freq are different scaling. I did a bench with full Q8_0 I get: with Q8_0: so The 2 cliplet have different speed. OK only 4% diff... not that much in my case... and full 140W limited. |
Beta Was this translation helpful? Give feedback.
-
|
OK some result with a toolbox ubuntu 22.04 / gcc11: gcc11 / ubuntu / ik_llama.cpp
so gcc11 bf16 is 10% slower... |
Beta Was this translation helpful? Give feedback.
-
|
I mis the -rtr 1.... it is mush better for bf16 and a little with Q8_0...
So the diff for bf16 with llama.cpp may be a gcc diff too... |
Beta Was this translation helpful? Give feedback.
-
|
OK, so this looks much better and as expected. Which means that I need to work around GCC-15 (and possibly other GCC versions between 11 and 15). I'll try to do that when I get time. If |
Beta Was this translation helpful? Give feedback.
-
|
Yes I build llama.cpp with default llamafile/sgemm this is the default for a longtime now ;) I think I was the last to work on the BF16-sgemm on llama.cpp ;) I made so much bench that it is rely unreadable ... I will try to make more repetable one with more config:
for BF16 at first ... If you think as something else that can help ... ask ;) |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for all this information! Now
My point is to know if it have some interest to add ZenDNN (I think it is better to direcly use the AOCL_gemm!) Note: I'll have a try with -rtr, For the GOMP_CPU_AFFINITY="1,3,5,7,9,11,13,15", this prevents migration between core, It is use on most HPC, and allow better caching managment, the L1/L2 cache did not need to migrate with the process, so can add some perf. |
Beta Was this translation helpful? Give feedback.
-
|
So more analysis with a more "professional" procedure 🤞 OS: fedora silverblue.
cmake is not good for build with ZenDNN on ubuntu (and not fluent with ubuntu...) so no res with it. so... I will be currous to look what we can have with AOCL gemm (with repacking) + your FA... |
Beta Was this translation helpful? Give feedback.
-
|
OK I add some #ifdef __AVX512BF16__
template <int nrc_y>
static void mul_mat_bf16_r16_bf16(int n, const void * vx, size_t bx, const DataInfo& info, int nrc_x) {
GGML_ASSERT(nrc_x%16 == 0);
const ggml_bf16_t * y[nrc_y];
for (int iy = 0; iy < nrc_y; ++iy) y[iy] = (const ggml_bf16_t *)info.src1_row(iy);
for (int ix = 0; ix < nrc_x/32; ++ix) {
__m512 acc[2*nrc_y] = {};
__m512bh qx[8];
const ggml_bf16_t * b8_1 = (const ggml_bf16_t *)((const char *)vx + (32*ix+ 0)*bx);
const ggml_bf16_t * b8_2 = (const ggml_bf16_t *)((const char *)vx + (32*ix+16)*bx);
for (int ib = 0; ib < n/8; ++ib) {
qx[0] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8_1+4*ib+0);
qx[1] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8_1+4*ib+1);
qx[2] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8_1+4*ib+2);
qx[3] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8_1+4*ib+3);
qx[4] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8_2+4*ib+0);
qx[5] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8_2+4*ib+1);
qx[6] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8_2+4*ib+2);
qx[7] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8_2+4*ib+3);
#pragma omp unroll full
for (int iy = 0; iy < nrc_y; ++iy) {
auto y128 = _mm_loadu_si128((const __m128i*)y[iy]+ib);
//auto y = _mm512_broadcast_i32x4(y128);
auto y256 = MM256_SET_M128I(y128, y128);
auto y = _mm512_inserti32x8(_mm512_castsi256_si512(y256), y256, 1);
// (__m512bh) _mm512_set1_epi32(*(int32_t*)&B);
acc[2*iy+0] = _mm512_dpbf16_ps(acc[2*iy+0], qx[0], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x00)));
acc[2*iy+0] = _mm512_dpbf16_ps(acc[2*iy+0], qx[1], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x55)));
acc[2*iy+0] = _mm512_dpbf16_ps(acc[2*iy+0], qx[2], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0xaa)));
acc[2*iy+0] = _mm512_dpbf16_ps(acc[2*iy+0], qx[3], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0xff)));
acc[2*iy+1] = _mm512_dpbf16_ps(acc[2*iy+1], qx[4], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x00)));
acc[2*iy+1] = _mm512_dpbf16_ps(acc[2*iy+1], qx[5], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x55)));
acc[2*iy+1] = _mm512_dpbf16_ps(acc[2*iy+1], qx[6], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0xaa)));
acc[2*iy+1] = _mm512_dpbf16_ps(acc[2*iy+1], qx[7], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0xff)));
}
}
#pragma omp unroll full
for (int iy = 0; iy < nrc_y; ++iy) {
info.store(32*ix+ 0, iy, acc[2*iy+0]);
info.store(32*ix+16, iy, acc[2*iy+1]);
}
}
for (int ix = 32*(nrc_x/32); ix < nrc_x; ix += 16) {
__m512 acc[nrc_y] = {};
__m512bh qx[4];
const ggml_bf16_t * b8 = (const ggml_bf16_t *)((const char *)vx + (ix+0)*bx);
for (int ib = 0; ib < n/8; ++ib) {
qx[0] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8+4*ib+0);
qx[1] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8+4*ib+1);
qx[2] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8+4*ib+2);
qx[3] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8+4*ib+3);
#pragma omp unroll full
for (int iy = 0; iy < nrc_y; ++iy) {
auto y128 = _mm_loadu_si128((const __m128i*)y[iy]+ib);
auto y256 = MM256_SET_M128I(y128, y128);
auto y = _mm512_inserti32x8(_mm512_castsi256_si512(y256), y256, 1);
acc[iy] = _mm512_dpbf16_ps(acc[iy], qx[0], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x00)));
acc[iy] = _mm512_dpbf16_ps(acc[iy], qx[1], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x55)));
acc[iy] = _mm512_dpbf16_ps(acc[iy], qx[2], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0xaa)));
acc[iy] = _mm512_dpbf16_ps(acc[iy], qx[3], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0xff)));
}
}
#pragma omp unroll full
for (int iy = 0; iy < nrc_y; ++iy) {
info.store(ix, iy, acc[iy]);
}
}
}and I get that with GCC-15: ============ Repacked 225 tensors
Look good with it on repacked bf16. Look like unroll spec have change from gcc-11 to gcc-15... It may be "need" on other place ;) |
Beta Was this translation helpful? Give feedback.
-
|
Wow! Thanks! GCC-15 doesn't unroll a loop with compile-time known number of iterations? You can submit a PR if you want. I guess, one needs to make it a compiler specific macro, and add to all loops over |
Beta Was this translation helpful? Give feedback.
-
Regression with GCC-15? Or with clang? |
Beta Was this translation helpful? Give feedback.
-
|
With GCC-15 ... don't bench with clang. But I need more bench. |
Beta Was this translation helpful? Give feedback.
-
|
And the following does not solve the regression? diff --git a/ggml/src/iqk/iqk_gemm_legacy_quants.cpp b/ggml/src/iqk/iqk_gemm_legacy_quants.cpp
index ca90f748a..6c55bd9d5 100644
--- a/ggml/src/iqk/iqk_gemm_legacy_quants.cpp
+++ b/ggml/src/iqk/iqk_gemm_legacy_quants.cpp
@@ -1618,12 +1618,12 @@ static void mul_mat_q8_0_r8_q8_2(int n, const void * vx, size_t bx, const DataIn
_mm256_loadu_si256((const __m256i *)q8h[4*ib4+k].qs+j), 1);
qx[j] = _mm512_add_epi8(qx[j], _mm512_set1_epi8(127));
}
- for (int iy = 0; iy < nrc_y; ++iy) {
+ static_for<nrc_y>([&](const int iy) {
auto sumi = qx_r8_q8_dot_product(qx, q8.y[iy][ib4].qs+32*k);
auto dy = _mm512_set1_ps(d8[8*iy+k]);
acc[2*iy+0] = _mm512_fmadd_ps(_mm512_mul_ps(scales, dy), _mm512_cvtepi32_ps(sumi), acc[2*iy+0]);
acc[2*iy+1] = _mm512_fmadd_ps(scales, _mm512_set1_ps(d8[8*iy+k+4]), acc[2*iy+1]);
- }
+ });
}
}
for (int ib = 4*(nb/4); ib < nb; ++ib) {
@@ -1635,20 +1635,20 @@ static void mul_mat_q8_0_r8_q8_2(int n, const void * vx, size_t bx, const DataIn
_mm256_loadu_si256((const __m256i *)q8h[ib].qs+j), 1);
qx[j] = _mm512_add_epi8(qx[j], _mm512_set1_epi8(127));
}
- for (int iy = 0; iy < nrc_y; ++iy) {
+ static_for<nrc_y>([&](const int iy) {
auto qy = (const block_q8_2 *)q8.y[iy];
auto sumi = qx_r8_q8_dot_product(qx, qy[ib].qs);
auto [d8, m8] = ScaleHelperQ8_2::prepare1(qy + ib);
auto dy = _mm512_set1_ps(d8);
acc[2*iy+0] = _mm512_fmadd_ps(_mm512_mul_ps(scales, dy), _mm512_cvtepi32_ps(sumi), acc[2*iy+0]);
acc[2*iy+1] = _mm512_fmadd_ps(scales, _mm512_set1_ps(m8), acc[2*iy+1]);
- }
+ });
}
- for (int iy = 0; iy < nrc_y; ++iy) {
+ static_for<nrc_y>([&](const int iy) {
auto sum512 = _mm512_fmadd_ps(_mm512_set1_ps(-127.f), acc[2*iy+1], acc[2*iy+0]);
info.store(ix, iy, sum512);
acc[2*iy+0] = acc[2*iy+1] = _mm512_setzero_ps();
- }
+ });
}
}
} |
Beta Was this translation helpful? Give feedback.
-
|
Or is it so that GCC-15 decides to not inline static IQK_ALWAYS_INLINE inline __m512i qx_r8_q8_dot_product(const __m512i * qx, const int8_t * y) { |
Beta Was this translation helpful? Give feedback.
-
|
Q8_0 (for Model and KV cache) zen5*16. (IA MAX+)
I remember ~510 for ubuntu gcc11 ... so not that bad. compare to the ~450 t/s it look we can do better... but may be hard. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I haven't been contributing to
llama.cppsince I left the project in March of 2024, but apparently I'm still one of the top contributors there 20 months later, so got pinged in an RFC and subsequent PR that integrates ZenDNN intollama.cpp. ZenDNN is a matrix multiplication library specifically optimized for AMD CPUs. It supportsbf16andf32GEMM. I haven't put a lot of effort into optimizing inference with floating point models (for me "Inference at the Edge" basically means using quantized models), so I decided to check if this could be something forik_llama.cppto handlebf16andf32models.The RFC and PR provide benchmark results for a big-iron, 96-core Zen4 CPU. I don't have that, but I do have a 16-core Ryzen-7950X, which is also Zen4, so ZenDNN should be optimized for it.
So, pulled and built the PR (it required a minor modification in the
CMakeLists.txtfile) and here is what we get withllama-benchon the 7950X forbf16LlaMA-3-8BI used as recommended
ZENDNNL_MATMUL_ALGO=2. The default (whatever it is), gives a PP performance of 163 t/s.In comparison, here is what we get with the
llama.cppCPU backend:Aha. ZenDNN nearly doubles
llama.cppPP performance, but that's not really hard. TG, on the other hand, is almost 2X lower.How does
ik_llama.cppcompare? Here is what we get:So, 1.27X ZenDNN and 2.44X
llama.cppfor PP. TG is faster thanllama.cppfor 1 and 2 threads, almost fully saturating BW with just 2 threads.llama.cppsomehow manages to saturate at a slight higher TG speed at 4 threads. Both are faster than ZenDNN with 16 threads with just a single thread (so more than 16X better energy efficiency when generating tokens)!Beta Was this translation helpful? Give feedback.
All reactions