Lotus benchmark won't end

[PNetrox]

OS: Ubuntu 20.04
CPU: 2x Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
Memory: 512GB
GPU: Nvidia RTX 2080 Ti
Disk: 8TB SSDs: 2x Pliant lightning LB 406S with Raid 0 controller
Controller Raid: AVAGO MR9460-16i 4GB cache + bbu

Im trying to bench seal on my server, when im starting 2KiB or 512MB sectors benchmarks all works, here are my resutls on 512MB sector:

./lotus-bench sealing
environment variable list:
FIL_PROOFS_USE_MULTICORE_SDR=1
FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
FIL_PROOFS_USE_GPU_TREE_BUILDER=1
----
results (v28) SectorSize:(536870912), SectorNumber:(1)
seal: addPiece: 1.671935555s (306.2 MiB/s)
seal: preCommit phase 1: 3m37.239182346s (2.357 MiB/s)
seal: preCommit phase 2: 8.769341641s (58.39 MiB/s)
seal: commit phase 1: 30.341738ms (16.48 GiB/s)
seal: commit phase 2: 12.857870538s (39.82 MiB/s)
seal: verify: 2.621792ms
unseal: 3m42.04761735s  (2.306 MiB/s)

generate candidates: 215.101µs (2.27 TiB/s)
compute winning post proof (cold): 1.847338546s
compute winning post proof (hot): 1.750364308s
verify winning post proof (cold): 28.653705ms
verify winning post proof (hot): 2.935676ms

compute window post proof (cold): 1.232493499s
compute window post proof (hot): 1.271978431s
verify window post proof (cold): 13.222029ms
verify window post proof (hot): 1.925738ms

But when im trying to bench on 32GiB sector, the process is going well till some time, after a whie it looks like its stuck, always in that place:

./lotus-bench sealing --sector-size 32GiB
 31.60 KiB / 31.60 KiB [=========================================================================] 100.00% 1.22 MiB/s 0s 13.32 KiB / 13.32 KiB [=======================================================================] 100.00% 547.62 KiB/s 0s 2.32 MiB / 2.32 MiB [==========================================================================] 100.00% 11.80 MiB/s 0s 4.60 KiB / 4.60 KiB [==========================================================================] 100.00% 21.38 MiB/s 0s 1.32 KiB / 1.32 KiB [===========================================================================] 100.00% 5.18 MiB/s 0s 13.32 KiB / 13.32 KiB [=======================================================================] 100.00% 554.81 KiB/s 0s 4.60 KiB / 4.60 KiB [==========================================================================] 100.00% 16.91 MiB/s 0s 1.32 KiB / 1.32 KiB [===========================================================================] 100.00% 5.81 MiB/s 0s 3.00 KiB / 3.00 KiB [===========================================================================] 100.00% 9.31 MiB/s 0s 1.32 KiB / 1.32 KiB [===========================================================================] 100.00% 5.60 MiB/s 0s 2.37 MiB / 2.37 MiB [==========================================================================] 100.00% 12.84 MiB/s 0s 183.15 MiB / 183.15 MiB [======================================================================] 100.00% 60.76 MiB/s 3s 13.32 KiB / 13.32 KiB [=========================================================================] 100.00% 1.50 MiB/s 0s 1.32 KiB / 1.32 KiB [===========================================================================] 100.00% 7.30 MiB/s 0s 4.60 KiB / 4.60 KiB [==========================================================================] 100.00% 13.50 MiB/s 0s 32.09 GiB / 32.09 GiB [====================================================================] 100.00% 31.14 MiB/s 17m35s 1.32 KiB / 1.32 KiB [===========================================================================] 100.00% 4.64 MiB/s 0s 13.32 KiB / 13.32 KiB [=======================================================================] 100.00% 546.95 KiB/s 0s 288.00 MiB / 288.00 MiB [======================================================================] 100.00% 96.20 MiB/s 2s 3.00 KiB / 3.00 KiB [==========================================================================] 100.00% 13.45 MiB/s 0s 56.61 GiB / 56.61 GiB [====================================================================] 100.00% 35.48 MiB/s 27m13s 3.00 KiB / 3.00 KiB [==========================================================================] 100.00% 17.61 MiB/s 0s 13.32 KiB / 13.32 KiB [=======================================================================] 100.00% 526.58 KiB/s 0s 44.12 GiB / 44.12 GiB [=====================================================================] 100.00% 96.43 MiB/s 7m48s 31.60 KiB / 31.60 KiB [=========================================================================] 100.00% 1.15 MiB/s 0s

After that, htop shows me that only one of my 40 cores is on 100% load, and after a while process .lotus-bench takes like 81GB of ram, and nothing even changes. I left my server like that for 2 days straight and when im came back it looks the same, just like its stuck.

Here are my vairables:

export LOTUS_PATH=~/.lotus
export FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
export FIL_PROOFS_USE_GPU_TREE_BUILDER=1
export FIL_PROOFS_USE_MULTICORE_SDR=1
export FULLNODE_API_INFO=https://api.calibration.node.glif.io/
export FIL_PROOFS_PARAMETER_CACHE=~/param_cache
export FIL_PROOFS_PARENT_CACHE=~/parent_cache
export FFI_USE_CUDA=1
export RUST_GPU_TOOLS_CUSTOM_GPU="RTX 2080 Ti:4352"
export LOTUS_MINER_PATH=~/.lotusminer
export GOLOG_OUTPUT=file
export GOLOG_FILE="$HOME/miner.log"

What can be wrong? How much time sealing should take on that hardware?

[magik6k]

E5-2690 is an incredibly slow CPU with no sha extensions, you can see more progress with RUST_LOG=info, but do expect the sealing process to take multiple days on that CPU

[PNetrox]

I got it, but i have another question, can i use this server for demon and storage provider at once? Without GPU, gpu i will move to my new worker server with cpu sha extensions, is it possible to run daemon and storage without any gpu?