Increase i/o depth in stagedsync?

[ty-sentio-xyz]

I have seen the discussion here: #1516
So I do not intend to talk about block execution stage, but other stages.

Stages like HashState/IntermediateHashState/indexes heavily utilizes sequential reads, where one i/o request must be fulfilled before issuing another, making disk i/o latency the primary performance bottleneck, on heavy chains like bsc/polygon (as page cache will unlikely hit).

Modern cloud disks, even offering SSD-alike throughput, do not offer SSD-alike latency. However, cloud disks do come with certain advantages, such as the ease of migration/snapshotting/backup/resizing/etc., over local disks, making the life of running and managing a node much more easier.

By nature, stages other than block execution has the potential of being effectively parallized, as in certain cases we do not rely on the result of a previous i/o request to determine what to read next. So what if we could, to some extent, parallelize reads in these stages, increasing i/o depth, to compensate the high latency?

Wonder if there is anyone who is thinking about this, would like to discuss further.

[AskAlexSharov]

General solutions:

• can read in parallel
• can warmup in parallel, but read sequential (warmup/readahead will ensure that sequential read will not face PageFaults)
• computation/data-transformation can be separated from read/write process and parallelized

General limitations:

• can't read in-parallel from 1 write transaction. If some data was modified in this write transaction - then "parallel reads from read-only transactions will not see your updates". It's often the case for incremental stages (on tip of chain).
• write can't be parallel
• warmup can't be too crazy - if it will go too far ahead - old (already warmed data) will be evicted again before sequential reader get to them.
• parallel warmups may reduce performance on NVMe/HDD. Hard to test impact. And impact depends on RAM amount.

What we have now:

1. stage_hash_state: initial sync of this stage already parallel, incremental is not. PR's are welcome. Another related task: there is now 2 collectors are protected by 1 mutex - split it to 2 mutexes will improve overall performance.
2. stage_trie: initial sync of this stage parallelism is done by me in branch trie_async. But i have no time to test, release it.
3. Other stages: can be solved by ErigonV3.

ErigonV3:

• not released yet but in good shape.
• it does move state history and related indices to snapshots (other mmap files). This snapshots you can madvise or read by bufio instead of mmap - and it will limited by 1 such file (less RAM impact. can't madvise chaindata now because it's >> RAM).
• it has parallel execution stage. it's not super-cpu-scalable now (more CPU will increase conflict-rate and reduce overall speed) - but it even 2-8 workers greatly increase Cloud Disks speed.