Excessive multithreading in shreds?

[alexpyattaev]

Problem

It is likely we have some unnecessary parallelism in shred making process (https://discord.com/channels/428295358100013066/478692221441409024/1442070692781883474)

Proposed Solution

• likely the only operations worth parallel processing are signing and verifying signatures. - investigate all other uses of par_iter in merkle.rs and eliminate as appropriate

[ANISH-SR]

on it!

[steviez]

I agree the parallel operations look dubious and this threadpool has been on my radar for a bit (one of the last one remaining that can't be sized on the CLI). But I'm not sure I agree it is a good first issue.

Shredding (part of block production) is a critical path, so we need to be very diligent in testing to ensure we do not regress. Or, if we do regress, we need to have the data so we can ask ourselves "Is spending X microseconds more on shredding worth the tradeoff / impact to the rest of the system of avoiding the tpool"

[alexpyattaev]

Shredding (part of block production) is a critical path, so we need to be very diligent in testing to ensure we do not regress. Or, if we do regress, we need to have the data so we can ask ourselves "Is spending X microseconds more on shredding worth the tradeoff / impact to the rest of the system of avoiding the tpool"

100% benchmarks are not optional here.

[steviez]

100% benchmarks are not optional here.

As Trent alluded to on Discord, we likely got into the state we are due to benchmarking. If the benchmark is the only thing running on the server, there is little to no contention on resources. So, stuff like one-thread-per-core might give the best bench results.

However, this is very different from the production environment where any given component is a part of a larger system (agave-validator). In this case, we have 100's of other threads that are all executing, allocating, etc. So, gains in benchmarks that come from multithreading may not be realizable in the validator, and potentially a net negative as we create more churn with the other threads.

For testing this, I think we'd be better served examining the validator including metrics & profiles, potentially on both real clusters as well as test clusters where we can create "stressful" situations (ie very high tx load)

[alexpyattaev]

@steviez, in this case removing threading mostly improves speed in a microbenchmark. If we were adding threading I would totally support full profiling of the complete validator. But Alessandro also believes this needs to be profiled.