Protocol versioning and upgrading.

[pipermerriam]

What is wrong

Up until now we've done direct coordinated upgrades to the protocol. I believe the only protocol upgrade so far has been Addition of Ping Extensions

This approach is fragile and only really viable in the current mode of network operation where "we" are running the vast majority of the network.

In the near term future, the network will transition to a more decentralized structure with operators being distributed and us no longer having clear and clean communication channels for coordinated upgrades.

How can this be fixed.

During the summit in Berlin we came to loose agreement on the following.

• We will start versioning our wire protocol.
• Upgrades will happen in a rolling manner.

For versioning, we will use whole number versions. We will call the current version of the protocol version 2. Any breaking change to the networking protocol or otherwise, change that is deemed necessary for versioning to be used will bump this version by incrementing it.

Signaling of supported versions will be done via a field in the ENR. The key for this field will be the ascii characters "pv". The equivalent bytes in hex is 0x7076. The value of this field will be a sorted SSZ encoded list with the serialization List[uint8, limit=16].

Portal clients can use the base discovery v5 protocol to exchange ENR records, after which, they should compare their own supported versions with the supported versions of this potential peer. Communication between two peers will use the highest mutually supported protocol version. In the event that two peers have no overlap between their supported versions, the peer should not be communicated with and should not be placed in sub-protocol routing tables.

Wire protocol versions are "global" and apply to all portal sub protocols.

[pipermerriam]

Here is a brief write-up of my understanding of the solution we agreed upon. Not looking for fine grained feedback as this needs to be converted to a pull request, but more an overall sense of whether this is in alignment with what each client team is expecting. I recognize this may be the first time some of the teams or contributors may have seen this.

[KolbyML]

This proposal looks like the most reasonable possible suggestion given our protocols requirements and wanting to avoid a negotiation phase in sending a request to a peer for latency reasons and such.

[morph-dev]

I like this proposal, but I believe it's mostly focused on wire-protocol changes.

How do we plan to deal with changing of the encoding of content key/values, which historically we did more frequently?
This doesn't have to be part of initial versioning, but it's probably something that we should figure out sooner rather than later.

[bhartnett]

This idea sounds reasonable to me. One question I have is how would we go about making a change to a specific sub-network. For example what if we discovered that content key/values of one of the sub-networks needed to be updated in a non-backwards compatible way, how would we do so?

Perhaps there could also be a per sub-network versioning scheme in the ENR's?

[kdeme]

I agree with the proposal because of following reasons:

• In discv5/portal it is only feasible to provide versioning info via the ENR considering the "connections" are short lived and adding version info in for example the ping msg rather beats the way of how this protocol is supposed to behave.
• The other option of working with a timestamped transition (similar to hard forks) can be quite demanding as it requires a lot more coordination, testing, etc.

On the practical side however, I do have some concern, perhaps specific to our implementation:
Implementing this from the sender side (request message) is fairly simple. The implementation can look at the version in the ENR and decide on which version of a protocol message it sends.
Implementing on the receiver side (preparing the response) is not as straight forward, at least not for our nimbus implementation. For the response there is no ENR in the received request "context", there is only the NodeId + address. ENRs are stored only for the nodes in the routing table. So in order for us to make this work we will have to do some discv5 architectural changes (ENRs should be stored for each stored discv5 session basically).

[GrapeBaBa]

I agree with the proposal because of following reasons:

• In discv5/portal it is only feasible to provide versioning info via the ENR considering the "connections" are short lived and adding version info in for example the ping msg rather beats the way of how this protocol is supposed to behave.
• The other option of working with a timestamped transition (similar to hard forks) can be quite demanding as it requires a lot more coordination, testing, etc.

On the practical side however, I do have some concern, perhaps specific to our implementation: Implementing this from the sender side (request message) is fairly simple. The implementation can look at the version in the ENR and decide on which version of a protocol message it sends. Implementing on the receiver side (preparing the response) is not as straight forward, at least not for our nimbus implementation. For the response there is no ENR in the received request "context", there is only the NodeId + address. ENRs are stored only for the nodes in the routing table. So in order for us to make this work we will have to do some discv5 architectural changes (ENRs should be stored for each stored discv5 session basically).

Geth had similar implementation like nimbus before, after portal summit we agree to make the change which storing the ENR in session.

[pipermerriam]

Perhaps there could also be a per sub-network versioning scheme in the ENR's?

I couldn't come up with a good reason for per-network versioning. If a client can support the new protocol version on one sub-protocol, I don't see a reason why it can't do it on all of them. Conversely, I can't find a case where we'd need to roll this out in a single sub-protocol while needing to hold it back on the others.

[r4f4ss]

Sorry, I am late for discussion.
It can be difficult for clients to keep up to date with the latest changelog as the specification is usually faster than the implementation. I thought about having two versions, the stable one where everything work for sure and the unstable one, where there is no guarantee that all the latest specifications are implemented.

[pipermerriam]

I believe "us", as the maintainers of portal nodes, will be actively monitoring the network to gauge adoption levels. Historically, our practice of coordinating network upgrades put time pressure on maintainers to implement new features quickly. We would still like to see quick adoption, but this new protocol versioning leaves more room for those upgrades to happen at a more relaxed pace.

For this first upgrade from v0 to v1, it will be good for client teams to implement it as a rolling upgrade, supporting both versions for a period and the deprecating v0 when we've seen adequate adoption of v1. All of our future upgrades should likely conform to this pattern. I also suspect that the rate of change will slow as time goes on.