Nimbus supernode advertises incorrect `cgc`

[Alleysira]

Summary

In differential testing, we found that nimbus(v26.3.1) advertises cgc=4 in ENR/metadata, while other CL clients advertise cgc=128.

  grandine     128 / 128
  lighthouse   128 / 128
  lodestar     128 / 128
  nimbus       4   / 4
  prysm        128 / 128
  teku         128 / 128

Related code

After some digging, the root cause seems to be that nimbus currently intentionally disables full PeerDAS supernode mode and aliases --peerdas-supernode to --light-supernode:

nimbus-eth2/beacon_chain/nimbus_beacon_node.nim

Lines 2822 to 2830 in 102be79

	# Ensure that non-light peerdas supernode options are forcibly disabled
	# TODO when reconstruction works again, re-enable
	# this is required because the fall-through is that if one of these is
	# enabled, the (working) light supernode code won't run at all.
	if config.peerdasSupernode:
	# It's at least not worse than not doing this; a functioning (full)
	# supernode reconstructs and stores a superset of these columns
	config.lightSupernode = true
	config.peerdasSupernode = false

Reproduce

Start the kurtosis testnet with the following config.yaml

participants:
  - el_type: geth
    cl_type: prysm
    el_image: ethereum/client-go:v1.17.2
    cl_image: offchainlabs/prysm-beacon-chain:v7.1.3
    supernode: true

  - el_type: geth
    cl_type: lighthouse
    el_image: ethereum/client-go:v1.17.2
    cl_image: sigp/lighthouse:v8.1.3
    supernode: true

  - el_type: geth
    cl_type: teku
    el_image: ethereum/client-go:v1.17.2
    cl_image: ethpandaops/teku:master
    supernode: true

  - el_type: geth
    cl_type: nimbus
    el_image: ethereum/client-go:v1.17.2
    cl_image: statusim/nimbus-eth2:multiarch-v26.3.1
    supernode: true

  - el_type: geth
    cl_type: lodestar
    el_image: ethereum/client-go:v1.17.2
    cl_image: chainsafe/lodestar:v1.42.0
    supernode: true

  - el_type: geth
    cl_type: grandine
    el_image: ethereum/client-go:v1.17.2
    cl_image: sifrai/grandine:2.0.4
    supernode: true

network_params:
  network_id: "260508"
  num_validator_keys_per_node: 11

additional_services:
  - dora

Then inspect each node's advertised ENR/metadata custody_group_count. The nimbus multiarch-v26.3.1 node reports 4, while the other supernode: true clients report 128.

[Alleysira]

Runing another test with v25.6.0. The advertised cgc has changed from 4 to 64, but not 128, which is the expected light-supernode value.

nimbus-eth2/beacon_chain/sync/validator_custody.nim

Lines 87 to 88 in 6fb05f3

	func lightSupernodeColumnsCount*(): int =
	NUMBER_OF_COLUMNS div 2

--peerdas-supernode=true is still downgraded to --light-supernode.

[tersec]

Runing another test with v25.6.0. The advertised cgc has changed from 4 to 64, but not 128, which is the expected light-supernode value.

Is 128 the expected light-supernode value? I'm unaware of any particular formal spec definition of light supernodes.

[Alleysira]

Is 128 the expected light-supernode value?

No. 128 is the expected value for cgc in supernode. But nimbus fallback supernode to light-supernode.
I think the designed value of cgc for light-supernode in nimbus is 64.

nimbus-eth2/beacon_chain/sync/validator_custody.nim

Lines 87 to 88 in 6fb05f3

	func lightSupernodeColumnsCount*(): int =
	NUMBER_OF_COLUMNS div 2

For me, light-supernode looks like a temporary fix.

[Alleysira]

I'm unaware of any particular formal spec definition of light supernodes.

Checked the consensus spec, there is no definition for light-supernode.

[tersec]

Sure, the mapping of supernode to light supernode is meant as a temporary workaround/fix for other issues, in progress. Nimbus has made sure to provide people a way to access their blobs, even as the column reconstruction issue fixes haven't been released.

So insofar as supernode maps to light supernode and results in a non-supernode cgc, yeah. But the root cause isn't necessarily the 64 cgc on light supernode: that seems to me an accurate representation of a what a light supernode is (enough columns to reconstruct the blobs efficiently, which was the main motivation that I could discern).

And, again, there is no spec of which I'm aware on light supernodes. As a concept they're kind of a kludge introduced very, very late in the Fusaka cycle after it wasn't feasible to fix the underlying PeerDAS setup anymore but people wanted their blobs with half the column cost.

[Alleysira]

Thanks for the explanation and context! Should I close this now, or keep it open until the fix is released?