Dynamic Bitfield Length

[AgeManning]

Description

It would be nice to have a Bitfield of dynamic length, that implements SSZ encoding/deocding. Currently we have fixed-length types, which requires the length of the object to be known at compile time. This is particularly annoying with the ENR bitfields. It means that anything that touches reads the bitfield ENR now has to have a generic which fixes the bitfield length. This typically means threading through the E: EthSpec generic.

Adjusting the length changes the merklisation of the ENR, however, to my knowledge we never merkalize an ENR and so a dynamic length bitfield should be fine and we can remove a bunch of generics in various parts of the network code.

Implementaiton

The current bitfields are here: https://github.com/sigp/ethereum_ssz/blob/main/ssz/src/bitfield.rs#L115

Making a version that is dynamic and then using this in the Lighthouse code, eg https://github.com/sigp/lighthouse/blob/stable/beacon_node/lighthouse_network/src/discovery/enr.rs#L34

[diegomrsantos]

How can we implement from_ssz_bytes for BitVector when the length is unknown at compile time? https://github.com/sigp/ethereum_ssz/blob/main/ssz/src/decode.rs#L119

[michaelsproul]

How can we implement from_ssz_bytes for BitVector when the length is unknown at compile time?

It's inelegant, but we have to avoid implementing Decode directly and add a method from_ssz_bytes which takes additional context as a second argument, i.e. the max length.

We do similarly for other "context-dependent" types like BeaconBlock:

lighthouse/consensus/types/src/beacon_block.rs

Line 102 in 0d90135

pub fn from_ssz_bytes(bytes: &[u8], spec: &ChainSpec) -> Result<Self, ssz::DecodeError> {

For a runtime bitfield, it will be:

fn from_ssz_bytes(bytes: &[u8], max_len: usize) -> Result<Self, Error>;

We've tried generalising ethereum_ssz to support this type of context-aware decoding, but so far haven't found a design that would be easy to migrate to. So we're stuck in this middle ground for now.

[jking-aus]

Have a PR for ethereum_ssz as a first step here: sigp/ethereum_ssz#46