Discovery Rework Ideas and Shrex Blockage

[distractedm1nd]

There is currently a problem with shrex on the BSR network, where we get context deadlines in two spots. This makes syncing eventually come to a screeching halt. I have not been able to get a FN to sync against BSR fully since discovery refactorings.

1. Opening the stream
2. Reading the status from the stream

Because these can both deadline after 60 seconds (30s on 3rd attempt), even a single bad peer can act as an "attractor" for the daser workers to get stuck on in their routines. There are multiple issues to be resolved here:

(yes I will open single issues and link them here)

1. Hopefully we can find the root cause of why we have connected peers that we cannot open a stream with
2. We should not wait a full 60s to open a stream with a peer
3. That we cannot open a stream is understandable, but why can we sometimes not read the status from the stream?
4. After a peer from fullnode is put on cooldown, we will always use that peer for that datahash (we create a pool for them at that datahash)
5. We do not punish context deadlines in any way (not even with cooldowns)

The best hotfix I could test against BSR was to add a ResultDropPeer, that removes the peer from the FN set and closes the connection with them. But this is a very hacky way to indirectly modify the discovery set from peer manager.

@walldiss has a good proposal to:

rework discovery from “make sure there are X connected peers” to on-demand by request apporach like “give me X peers, but not those:map[peerID]”

One other suggestion we tried out is to put these unresponsive peers on a long cooldown, essentially "soft banning" them. This wasn't enough to fix syncing with a timeout of 1 minute and disc peer limit of 15.

All of these needs can be summarized that we need to find a way to have some sort of simple peer scoring for shrex/peerman.

This discussion is here to serve two purposes:

1. Give a home to the related issues that are still yet to be created+linked
2. Figure out a hotfix that can get shrex working without another large discovery refactor