Benchmarks and tests. #18
Replies: 14 comments 54 replies
-
|
I'm a bit confused as to what I am seeing in that table. I am also not quite sure what you mean by this portion:
Can you clarify? |
Beta Was this translation helpful? Give feedback.
-
|
I think I got something wrong when copying the information,
see Full logs System Memory: 440/442 GiB. Running Phase 1 Plot /mnt/tmp/8cbc2d75a8b2cd1334c62e88256930ba394cfd166c162b16ab63296c1c366381.plot finished writing to disk: Finished writing tables to disk in 143.04 seconds. Final Directory: /mnt/tmp/ |
Beta Was this translation helpful? Give feedback.
-
|
Ah, ok I got it. Much better now!
No, you're english is completely fine :) Those 143.04 seconds are writing what remains, the rest has been written in the background on Phase 3. However, my writes avoid the kernel buffers, and go directly to disk, which typically appears to take longer than a regular I am more surprised for the poor numbers on Phase 1. I suppose I ought to do a different type of sort for NUMA systems, and optimize more for NUMA systems. But, to be quite honest, I probably won't unless I get my hands on a development server. |
Beta Was this translation helpful? Give feedback.
-
|
Now I get it... You also need to take into account that I think the performance may have been affected as well because it's running on a Virtual Machine. But still on this line of thinking, do you think it would make sense, for example using fast disks for swap partitions and using a dynamic memory allocation process? Could forcing the use of memory swap bring any benefit? But anyway, without a doubt bladebit is very good, it goes head-to-head with madmax. Man, I'm just wondering if this code could be optimized for use with the GPU using its memory. I'll tell you something, not understanding anything about C++ is painful, lol. My dream was to see this code ported to c++/cli mixing native code with managed code and natively integrating with the dotnet world so I could debug it understanding what's going on behind the scenes, it looks like it's going to be more natural with visual studio 2022 and I think the trend is for c++ to go this way making it easier to go cross platform. Regarding the dev machine, I work with one of the cheapest bare metal cloud providers on the internet, 1 TB in hdd costs 5 USD/Mes, it's cheaper than backblaze, plus gigabit network with 20 TB of bandwidth, but they work with older computers, an dual 8 core intel E5-2680 2.70 GHZ 16 cores / 32 Threads 512 GB DDR 3 1 x 960 GB SSD & 2 x 1 TB SATA |
Beta Was this translation helpful? Give feedback.
-
Which pare are you referring to here specifically? As for the GPU: Currently the bottleneck is transferring data to and between the GPU. That and GPUs currently don't have significant amount of RAM to do anything meaningful with plotting there. There are some that have a decent amount of RAM, but are extremely expensive. PCIe 4 & 5 advertise pretty great speed increases in transfer speeds, so we will see what happens then.
Onde vc trabalha? E no Brasil? Disculpa meu portugues, nao e tao bom quanto o seu ingles.
Well I wouldn't recommend mixing it. The dotnet runtime as a ton of overhead over native code... For performance, you have to stick to native code. But if you know C#, a transition to C++ is not as hard as coming from, say, dynamicly types languages. Stick to it for a little while and you'll certainly catch on. |
Beta Was this translation helpful? Give feedback.
-
|
Hello harold, yes I work in Brazil, but i use a dedicated server from a provider in Miami, it's called cloud south (this is what I said about the 360 USD dedicated server with 512 RAM) their servers and storage are very cheap, you can put ssd or hdd and you install the machine yourself from scratch using KVM. I have a 120USD server with 128/Ram with hyper-v that i use for development and hosting with windows and linux, cpanel, whm, testing, etc since 2017 without any problems Your Portuguese was perfect, my English is only good with the help of google translator. Reviewing the topic, I confused the dynamic memory allocation done by the operating system with the dynamic allocation of c++, and actually reading more closely I realized that this implementation doesn't actually exist in C, I understood it did, what did I mean by "force" was to allocate force part of file "go to" Swap space as described in this question https://softwareengineering.stackexchange.com/questions/404905/in-c-and-gcc-on-linux-is-it -possible-to-allocate-memory-to-your-swap-space-in So the post doesn't get big and off-topic I'll try to put the idea I had more clearly in the ideas topic. |
Beta Was this translation helpful? Give feedback.
-
|
Here's what I have. So far this is amazing, thanks so much harold! Hopefully there is a way to shave a few minutes off of these plot times still! :) 2x 2695 v2 Creating 999 plots: System Memory: 501/503 GiB. Running Phase 1 The following plots are 24 minutes each as it waits for the plot to copy to NVME... Generating plot 2 / 999: Running Phase 1 Again, thanks so much! Any pointers on increasing speed would be appreciated! |
Beta Was this translation helpful? Give feedback.
-
|
2x2690v2 madmax: ~60 plots/day but bladebit uses ~15% less power, bc there are a LOT less context switches comparing to madmax. This also leads to quieter server fans lol :) |
Beta Was this translation helpful? Give feedback.
-
|
I was talking about bladebit. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your help, I have found the problem! |
Beta Was this translation helpful? Give feedback.
-
|
most effecient way to copy from ssd/nvme to hdd is to use directio bypassing os pagecache. this way kernel dont waste time copying data to/from pagecache. It IS possible to archieve 10gbit speeds on 1 core this way. |
Beta Was this translation helpful? Give feedback.
-
|
Will see about setting up a wiki page for sharing these kinds of things as all the tips you've shared @mocksoul should be helpful to others. |
Beta Was this translation helpful? Give feedback.
-
Thank you. That's what I need. Finished plotting in 1136.34 seconds (18.94 minutes). |
Beta Was this translation helpful? Give feedback.
-
|
Love seeing the benchmarks, so thank you for sharing the times, guys! |
Beta Was this translation helpful? Give feedback.
-
|
I can give you access to the server with NUMA and 1024 GB RAM) |
Beta Was this translation helpful? Give feedback.
-
|
2690v4 x 2 @ 512GB RAM, 28 threads (56 with HT) @3.2GHz |
Beta Was this translation helpful? Give feedback.
-
|
Epyc 7302, -t 30 (running some VMs on the server) |
Beta Was this translation helpful? Give feedback.
-
New version 1.1.0 shows good performance improvement (86ppd -> 98 ppd)Hardware OS Results
Note: That is a single run which includes copy to SSD
Before bladebit: The sort improvement of ~20 seconds shows several times: Version 1.0.0: Versions 1.1.0: So that sort optimization sums to ~120 seconds improvement for my hardware (CPU was released in Q3'14, 7 years before). Good Work. Appendix Logs:
" 2)" |
Beta Was this translation helpful? Give feedback.
-
|
Benchmark Round#3 Results for 10 runs: 12.7 minutes per plot => 113 PPD 6465.6598.out:Finished plotting in 762.40 seconds (12.71 minutes). Discussion of results:
40 threads Historical note: |
Beta Was this translation helpful? Give feedback.
-
ofc it is, hyperthreading is not just run 2 threads per 1 core, it is much more complicated (and a lot of extra transistors on die)
kernel needs to check every page (4k), this is a lot of work :). |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your tip to use dd+direct-io instead of cp/mv I just found several invalid plots: Analyzing the logfiles for the plots I found: Conclusion: I just wanted to bring this into attention in case others complain^H^Hobserve similar problems. Appendix |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
E5-2673v4 x 2 @ 512GB RAM, 40 threads (80 with HT) @2.3GHz Finished Phase 1 in 282.05 seconds. |
Beta Was this translation helpful? Give feedback.
-
yep. ~10% improvement, all in Phase1. Great job, Harold!!! |
Beta Was this translation helpful? Give feedback.
-
I was wrong. it looks like 2690v4 (14 cores) is completely on par with 2673v4 (20 cores) on bladebit v1.1.1 :-D 549s vs 538s Interesting will be 2699v4 faster or not... |
Beta Was this translation helpful? Give feedback.
-
|
This 2680v4 delivers in 539 seconds (8.99 minutes) 2x E5-2680v4 @ 2.40GHz(14cores, 35MB Cache, 120W) = 2x 14cores= 28 cores (56HT), 512GB LRDIMM (2400MHz) 15 runs: Performance per Phase: Detailed Performance: |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
2x E5-2697v2 (12core, 30MB Cache, 130W, 2.7-3.5 GHz, Launch Q3/13) = 24 cores (48 Hyperthreading), 512GB LRDIMM (1600MHz) Best run: Finished plotting in 846.32 seconds (14.11 minutes) 10 runs: Performance per Phase: Detailed Performance: Comparing the detailed performance with other hardware configurations |
Beta Was this translation helpful? Give feedback.
-
|
Finished Phase 1 in 560.57 seconds. -w and --no-cpu-affinity don't affect my computer too much and the wait time for writing to SSD is about 29 seconds. My system is ubuntu20.04 server. Your phase 1 is much faster than mine, is this due to memory configuration? Maybe I should do some research. |
Beta Was this translation helpful? Give feedback.
-
|
More experiments: 1.Running _with --no-cpu-affinity slows downs a little bit, about ~0.2 minutes (~10 seconds), -w has no measurable effect: 2.Running with only 14 dimms causes big slowdown ~2.5 minutes (16 dimms: 14.2m) Note: the performance per Phase is also very different from your numbers: 3.Running with 15 dimms is ~20s slower than with 14dimms. This is 3354.11310.out:Finished plotting in 1039.91 seconds (17.33 minutes). Looking at performance per phase it seems phase3 suffers here: |
Beta Was this translation helpful? Give feedback.
-
|
Thank you again for your help. Thank you very much for your help. After reading the solution you gave me yesterday, I found on the Internet that 4 channels of memory need to be inserted 4 times of memory chip to achieve the complete performance. So I bought two memory sticks yesterday, which should solve my problem after receiving the goods. You were slower than me on 14 memory sticks in phases 2 and 3, because you had to run the software a few more times (I don't know why), and faster than me in phase 1, I don't know why. Thanks again for your help. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I have near the same Problem, but i have even 4 channels 2 sticks. i have test now all, but i dont get under 18mins for a plot. I think i have a problem with the gen8 HP Server Bios. Have You an Idee what is my Problem? thanks |
Beta Was this translation helpful? Give feedback.
-
|
@PatFis: What distribution do you use? |
Beta Was this translation helpful? Give feedback.
-
|
OS: CentOS 8.3 - 8.4 Results: ~8-9 min plots, ~150 plots per day. Kinda disappointing, was expecting 5 min plots, 200-220 a day like the video. Not sure why my similarly spec'ed system isn't pulling similar numbers. Log: |
Beta Was this translation helpful? Give feedback.
-
|
Indeed surprising. Your times are mostly in-par with other similar systems, though Intel-based ones, except for sorting which seems to be double on yours: ~25 seconds vs ~11 seconds con 8000 series Xeons. |
Beta Was this translation helpful? Give feedback.
-
|
I recently tried dropping 8 threads from bladebit to see if plot transfer speeds would be faster, and I noticed that I am actually plotting faster using 120 threads (8.87 minutes) vs 128 (8.87 minutes). So if you are not getting the speeds you thought you would, maybe try freeing up some cores. Running Phase 1 |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Run baremetal using all (hyper-)threads, also Ubuntu is slower than RHEL according to #141 |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
bladeuserpi are your 2x Xeon Gold 6148 retail boxed processors, or ES or QS? Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
I am running 2 X e5-2673 V4 with 480GB RAM and clocking 20 min. per plot. I have seen some one doing ~8min with similar CPUs. Any idea what could be wrong? Creating 1 plots: System Memory: 459/479 GiB. Running Phase 1 Plot G:/plot-k32-2022-01-10-18-05-03377758873f13621de609e846644404e02c00dca6a4546c019957fabd363d74.plot finished writing to disk: Finished writing tables to disk in 153.76 seconds. |
Beta Was this translation helpful? Give feedback.
-
|
you should use 16x32G RAM stick to reach quad channel RAM, 480G means you loose the ram speed, and be care the RAM slot, follow the instruction of your server |
Beta Was this translation helpful? Give feedback.
-
|
I recently switched my plotter's OS from RHEL 8 to Fedora 35 and thought I'd share - plot times are in the same ballpark. I'd need to do further testing to see about that performance drop leaving RHEL behind. Output path : /raidzero/ Running Phase 1 |
Beta Was this translation helpful? Give feedback.
-
Out of curiosity, I tested the mad maX bladebit both at 100% on the Ram / Ramdisk.
bladebit had 20% performance over Madmax.
I'll try to test it straight on metal and with NUMA optimizations and see what happens.
The interesting thing, even when it comes to an older machine, was the recording speed for hdd, only 143s, I think it should be possible to adjust the system to downloading some tables to disk for a while to save RAM without necessarily losing performance, perhaps even being able to plot on a system with 128 Ram or less. is something to think about, do you agree?
Here are the results:
Dell R720 dual Xeon e5-2620 @ 2.1Ghz 512 Ram DDR3 ecc
Windows Server 2012R2 Hyper-V
VM with ubuntu 20.1 - 460 Gib Ram
Storage pool with 8 HDD sas
Phase (seconds) BladeBit Ram MadMax Ramdisk
P1 table 1 168 49
P1 table 2 493 301
P1 table 3 394 338
P1 table 4 365 378
P1 table 5 362 377
P1 table 6 350 373
P1 table 7 83 327
Phase 1 total 2.222 2.142
Phase 2 Total 160 1.018
Phase 3 1.844 1.780
Phase 4 5 254
Creation Time 4.230,0 5.195
Time to copy 143,0 5
Final Time 4.373,0 5.200
Beta Was this translation helpful? Give feedback.
All reactions