feat(flashblocks): Add async state root calculation and pre-warming flashblocks sequence execution to local execution cache

[sieniven]

Summary

This PR references the flashblocks RPC solution 1 mentioned in: #20491. It optimizes the sync for flashblocks RPC by inserting the executed flashblocks sequence to the engine state tree, to prevent re-execution and re-calculating of state root of the same payload on engine_newPayload engine api. This should optimize the RPC sync speed and improve latencies on the RPC node.

1. Flashblocks RPC sync logic is now split into 2 portions. New incoming flashblocks sequence are validated and executed without state root calculation quickly, which updates the pending state on the eth api handler as quickly as possible
2. State root calculation and subsequent sync logic (flashblock or CL driven syncs) are shifted to async
3. If flashblocks RPC node chainstate is driven by the CL sync (--flashblocks.consensus turned off), which this PR optimizes the sync speed of the CL sync as unsafe payload commits on the CL will skip payload re-validation on cache hits, which the flashblocks RPC has already executed and validated the sequence as the current pending block
4. With --flashblocks.consensus enabled, this optimizes the RPC node sync with the flashblocks builder source, as we cache hit locally executed blocks prevent re-execution of the same payload on the flashblocks RPC node

[sieniven]

This PR has been tested to work. Example logs:
After:

Before:

With this PR, no payload validation state root task job is spawned once the executed flashblocks sequence is committed locally by the flashblocks RPC service.

However, it is noted that cache misses will still occur on CL sync mode, if the payload commit (engine_newPayload) happens before the local execution cache is updated. For example, this could occur during a scenario where a full block takes awhile to calculate state root on the flashblocks worker job, while CL sync has minimal latencies and the unsafe payload commit happens before flashblocks worker commits the executed flashblocks sequence into the local execution cache on the engine state tree.

For flashblocks consensus sync mode, no cache misses should occur.

[mattsse]

I'm a bit sceptical that this is always useful, because this can delay the latest flashblock for SR computation making it not available for rpc as fast as possible

at the very least I'd like to see some timings for base for this

also curious what @0x00101010 thinks here

[mattsse]

hmm, a bit unsure about exposing this because this assumes the block is fully valid

[sieniven]

I see where youre coming from, I am unsure of an alternative safer approach to this - I have instead renamed this better to FlashblocksSequence instead so that we explicitly know this message is handled entirely only for flashblocks rpc layer (d35f59b).

[mattsse]

this will now always compute the state root?

dont think this is what we want here?

[sieniven]

The state root calculation has since been switched to async in this PR, which allows us to quickly update the pending state after flashblock payload execution.

[sieniven]

@mattsse makes sense. Actually this was on a todo for me but I thought it was better to split adding that into a separate PR - but I think we could resolve this by spawning an asnyc task to calculate state root after updating the pending FB state. This doesnt affect FB consensus though, since state root calculation happens by default on target FB number.

[sieniven]

@mattsse let me see how I can optimize this. Also theres a failing UT so I'll fix that too 👍

[sieniven]

@mattsse @0x00101010 The PR has been updated to include async state root calcuations. The sync logic is now split into 2 parts on new flashblock payload:

1. The flashblocks sequence is executed and the pending state is updated with minimal latencies, without state root calculation
2. Subsequent sync logic (flashblocks consensus or CL driven sync) asynchronously calculates state root and commits to the engine state tree with the engine handler