Investigate if the current peer count (100) is sufficient for PeerDAS

[jimmygchen]

Lighthouse currently target 100 peers and may not be sufficient if the PeerDAS number of data column subnets = 128. @dknopik's did some simulation on this, and it shows that node can sometime struggle to find peers in their subnets. Having sufficient number of supernodes in the network help, however there's no guarantee that a node would have at least a few supernode peers.

For a supernode, it may struggle to find enough peers in all 128 subnet, this makes keeping the node in sync challenging, it would likely be worse during the initial sync.

However, increasing peer count also come with cost. Simulation shows that Lighthouse may consume large amount of memory with larger peer count, and could OOM.

• Investigate what number of peers is "sufficient" based on the current subnet count (128). Is increasing peer count necessary?
• Look into strategies that helps maintain a sufficient number of peers on the custody subnets
• Is there scenarios where we'd want higher peer count? It may help to increase peer count temporarily during range sync, or under bad network conditions (as it's more difficult to find peers on the same chain AND custody the required columns)?

[dapplion]

Simulation shows that Lighthouse may consume large amount of memory with larger peer count, and could OOM

This shouldn't be the case, a non-mesh peer should consume barely no resources at all. Nimbus has been using ~300 peers for a long while with very low overall memory consumption

[jimmygchen]

Yes I'm aware of Nimbus's higher peer count, I'm just pointing out the results from simulation - it may well be OOM due to other factors - but it's worth investigating and monitor the resource consumption as we experiment with different numbers.

[jimmygchen]

Related issue: #6930