Decouple chunk endorsement processing from client actor thread

[Trisfald]

Description

Objective

Allow the chain to perform chunk validation (endorsement production, validation and reception) faster and with higher priority over other, less important, tasks.

Expected outcome

• Chunk endorsements are processed in parallel with other client actor tasks. In the same way as transactions as processed after the introduction of their own actix handle.
• Chunk endorsements are delivered faster and we observe less chunk misses under high load.

Motivation

During performance tests it has been observed that under high load chunks can't be endorsed in time. This might be correlated a bottleneck in the single thread client actor. Hardware usage and network IO are never saturated, so it should be investigated whether chunk endorsements and other client actor tasks can be parallelized better.

This is also a way to achieve 'priority endorsement processing' which was discussed previously with Contract Runtime.

[walnut-the-cat]

do you think this will also solve #13151 ?

[Trisfald]

do you think this will also solve #13151 ?

Actually it might help, yes. Assuming state sync slows down client actor performance, parallelizing chunk endorsement processing should reduce endorsement misses.

[Trisfald]

Profiler data from a chunk producer node running on a 20 nodes network: https://share.firefox.dev/4c4Ki0f

[Trisfald]

Another two profiles, also from a chunk producer node running on a 20 nodes network, this time with 'realistic mainnet' configuration

• https://share.firefox.dev/4c79MKt
• https://share.firefox.dev/3QSSg2S (this one has produce_block call inside)