Use global threadpool for new epoch block processing

[HaoranYi]

Problem

Creating threadpool for epoch processing is expensive. When multiple forks happen at the new epoch, threadpools will be created multiple time. This wastes resources and hurt performance.

In this PR, we use global threadpool for new epoch processing.

Summary of Changes

Use global threadpool for new epoch processing

Fixes #

[HaoranYi]

thread_pool_creation_us=5705i

It takes 5-6ms to create the threadpool at epoch boundary.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 94.89051% with 7 lines in your changes missing coverage. Please review.

Project coverage is 82.9%. Comparing base (b335f5d) to head (ad0c66d).
Report is 22 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##           master    #5833     +/-   ##
=========================================
- Coverage    82.9%    82.9%   -0.1%     
=========================================
  Files         840      840             
  Lines      377801   377727     -74     
=========================================
- Hits       313300   313193    -107     
- Misses      64501    64534     +33     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[alexpyattaev]

Using global threadpool means some other logic may be occupying it, which may delay your work starting by far more than 5-6 ms.

[HaoranYi]

@alexpyattaev what other logic is using global threadpool? From what I see, the global threadpool threads seem to be idle all the time? do you have a benchmark for the delay on using global thread pool?

The current code creates one threadpool per epoch boundary block, which can leads to hundreds of threads at the epoch boundary because the long running epoch processing create many forks ...

[alexpyattaev]

Thing is it is hard to tell which components use global pool at all. So we can never be sure it does not get added later. And our unittests do not check for epoch transitions. But maybe creating pools per fork is the worse of the two evils.

[HaoranYi]

Another alternative is to create a dedicated threadpool for epoch boundary processing. This way, we know that no other component will compete for the thread pool. But the downside is that this threadpool is idle most of the time except at epoch boundary. So there is pros and cons here too.

@steviez I recalled that you have audited all thread pool usage before. I would like to now if you have any opinion on the thread pool usage at epoch boundary? Can we use global thread pool at epoch boundary?

[steviez]

what other logic is using global threadpool?

I had a good handle on this at one point, but that was more than a year ago so my answer is "hard to say". The more things that use the global threadpool, the harder it is to answer this question tho.

I share Alex's concern about usage spikes; the pool might have low usage in relation to the lifetime of the process, but hard to know what is happening in short bursts. And, if this code is relying on the pool to be idle, then this pool arguably "owns" the global pool.

Can we use global thread pool at epoch boundary?

Generally speaking, I'd like to see our usage of the global pool trend towards zero with the exception being "one time tasks" at startup.

thread_pool_creation_us=5705i

Eyeballing some metrics on MNB, I see Ninja reporting similar 6k for thread_pool_creation_us but 2.5M - 3M for update_rewards_with_threadpool_us. So, even if we diminish this 6ms to a smaller value (0), it seems the overall time is still heavily dominated and shaving a couple millis here would have minimal impact for the total time.

Another alternative is to create a dedicated threadpool for epoch boundary processing

I had thought about something like this before. Our code for creating new banks is serial in ReplayStage (ie we are never calling Bank::new_from_parent() in parallel), so it seems that Bank could have an Arc<ThreadPool> that is passed down to children. But like you said, this pool would hang around and be idle most of time and the rayon pools have non-trivial overhead for "book-keeping" even if they are idle

[HaoranYi]

I think one of the danger to use "global thread pool" is that 3rd party libraries could be competing for the global pool.

[HaoranYi]

Our code for creating new banks is serial in ReplayStage (ie we are never calling Bank::new_from_parent() in parallel)

Does this mean we never replay two banks in parallel?

Find it. I think you are right.

[brooksprumo]

What's the status of this PR?

I ask because it is marked as ready for review, and my review is requested, yet the PR description says its experimental and should not be merged.

[HaoranYi]

What's the status of this PR?

I ask because it is marked as ready for review, and my review is requested, yet the PR description says its experimental and should not be merged.

Yes, it is ready for review.

In my original study, using global threadpool gave us better performance as long as no other process was occupying the global threadpool, which is the case for agave validator.

[brooksprumo]

Can you share/include up to date perf numbers of the epoch processing with and without this PR, please?

[HaoranYi]

Can you share/include up to date perf numbers of the epoch processing with and without this PR, please?

new_th.log:update_rewards_with_thread_pool_us=547103
global_th.log:update_rewards_with_thread_pool_us=567185

Benchmark shows that it is faster with the new thread pool than with global threadpool, actually: 547ms vs 567ms. The difference, 20ms, is more than the cost of creating the threadpool 5ms.

In addition, the threadpool is also used for processing other than epoch rewards. For example, apply_feature_activation is also better with the new thread pool (though not much).

$ grep -Po "apply_feature_activations=\d+" new_th.log global_th.log
new_th.log:apply_feature_activations=4525
global_th.log:apply_feature_activations=5164

So the cost of creating a new thread pool is worth it. We'd better keep using the new thread pool.