Proposal: Make the block-pool concurrency memory-aware

[evan-forbes]

this idea stems from the investigation that is being done in #6067 in app by @mcrakhman celestiaorg/celestia-app#6067

The idea here is to prevent OOMs while block syncing large blocks via tracking the cumulative estimated size of pending blocks and stop makeNextRequester once a configurable byte ceiling is reached. This is assuming that the cause is due to requesting up to 2.5GB of data per peer (20 requests of 128MB blocks per peer). We're not yet sure that is the case, however even w/o that this seems like a reasonable change.

We might be able to make some optimizations and keep large numbers of requests for smaller blocks but not sure yet.

[mcrakhman]

1. Why we have memory problems with blocksync.

We have two parameters which are affecting the memory usage, the memory buffers sitting on p2p connections and the memory for blocks which are downloaded from peers but not saved.

Previous algorithm loaded 200+ heights ahead, which is too much for large blocks, therefore we exceeded 32 gigs of RAM.

2. What we can improve:

a. Memory (already implemented in #2636)

• remove unnecessary allocs on copying the buffer from connection.

Now, to parallelize the reactor processing we copy each time the raw message bytes to send it to a reactor gorouitne, and then inside this goroutine we unmarshal bytes in a protobuf structure, thus we copy bytes two times. I propose to remove the copy and unmarshal in the same connection goroutine, and then send the unmarshaled message to a reactor goroutine.

• limit the window on blocks we are loading to some number which is dependent on peers, but is memory bound.

So depending on current max (or average) size of the block for last X blocks (e.g. 50) we decide the current number of requesters (i.e. the slots for the blocks to be loaded from peers). We also take into account the max size that these requesters should take in total (like 4Gbs for example).

b. Speed (to be implemented)

1) Going to roughly about 1.5 - 1.6s per 128mb block

• save blocks in another goroutine, so this will not block the block processing.

The issue with this approach is that if the node is abruptly terminated, we can end up with the state at block N, but we would have less blocks saved in the storage (N - X), which causes node to fail to start. This can be mitigated by using rollback capabilities of the state (I tried rolling back to 40 blocks and this worked fine, by default it should be 362880 blocks (specified in pruning options).

Doing this will save us around 500ms, but we are still limited with process proposal.

2) Going to around 300-500ms per 128 Mb block

• only validate PartSets but skip ProcessProposal. This still confirms that the block were signed by 2/3+ of the vals, but we don't check row roots etc, i.e. that the blocks are properly formed.

It seems (afaiu) we would have roughly equal guarantees in these two cases:

• state sync to block N - X from trusted height/hash and then do blocksync without process proposal to block N. Here we trust the first state and then for each block we check that correct validators signed the next block (assuming that the block at trusted height was correct).
• state sync to block N