Use global threadpool for new epoch block processing

[HaoranYi]

Problem

** Experiment PR. Don't merge. **

Creating threadpool for epoch processing is expensive. When multiple forks
happen at the new epoch, multiple threadpools will be created. This waste
resources and could hurt performance.

In this PR, we use global threadpool for new epoch processing.

Summary of Changes

Use global threadpool for new epoch processing

Fixes #

[HaoranYi]

thread_pool_creation_us=5705i

It takes 5-6ms to create the threadpool at epoch boundary.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 94.89051% with 7 lines in your changes missing coverage. Please review.

Project coverage is 83.0%. Comparing base (3ee4056) to head (f013579).
Report is 7 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##           master    #5833     +/-   ##
=========================================
- Coverage    83.0%    83.0%   -0.1%     
=========================================
  Files         828      828             
  Lines      375638   375564     -74     
=========================================
- Hits       311919   311809    -110     
- Misses      63719    63755     +36     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[alexpyattaev]

Using global threadpool means some other logic may be occupying it, which may delay your work starting by far more than 5-6 ms.

[HaoranYi]

@alexpyattaev what other logic is using global threadpool? From what I see, the global threadpool threads seem to be idle all the time? do you have a benchmark for the delay on using global thread pool?

The current code creates one threadpool per epoch boundary block, which can leads to hundreds of threads at the epoch boundary because the long running epoch processing create many forks ...

[alexpyattaev]

Thing is it is hard to tell which components use global pool at all. So we can never be sure it does not get added later. And our unittests do not check for epoch transitions. But maybe creating pools per fork is the worse of the two evils.

[HaoranYi]

Another alternative is to create a dedicated threadpool for epoch boundary processing. This way, we know that no other component will compete for the thread pool. But the downside is that this threadpool is idle most of the time except at epoch boundary. So there is pros and cons here too.

@steviez I recalled that you have audited all thread pool usage before. I would like to now if you have any opinion on the thread pool usage at epoch boundary? Can we use global thread pool at epoch boundary?

[steviez]

what other logic is using global threadpool?

I had a good handle on this at one point, but that was more than a year ago so my answer is "hard to say". The more things that use the global threadpool, the harder it is to answer this question tho.

I share Alex's concern about usage spikes; the pool might have low usage in relation to the lifetime of the process, but hard to know what is happening in short bursts. And, if this code is relying on the pool to be idle, then this pool arguably "owns" the global pool.

Can we use global thread pool at epoch boundary?

Generally speaking, I'd like to see our usage of the global pool trend towards zero with the exception being "one time tasks" at startup.

thread_pool_creation_us=5705i

Eyeballing some metrics on MNB, I see Ninja reporting similar 6k for thread_pool_creation_us but 2.5M - 3M for update_rewards_with_threadpool_us. So, even if we diminish this 6ms to a smaller value (0), it seems the overall time is still heavily dominated and shaving a couple millis here would have minimal impact for the total time.

Another alternative is to create a dedicated threadpool for epoch boundary processing

I had thought about something like this before. Our code for creating new banks is serial in ReplayStage (ie we are never calling Bank::new_from_parent() in parallel), so it seems that Bank could have an Arc<ThreadPool> that is passed down to children. But like you said, this pool would hang around and be idle most of time and the rayon pools have non-trivial overhead for "book-keeping" even if they are idle

[HaoranYi]

I think one of the danger to use "global thread pool" is that 3rd party libraries could be competing for the global pool.

[HaoranYi]

Our code for creating new banks is serial in ReplayStage (ie we are never calling Bank::new_from_parent() in parallel)

Does this mean we never replay two banks in parallel?

Find it. I think you are right.