Handling Ephemeral State content

[morph-dev]

We never discussed in details on how to handle ephemeral state content and I believe it is quite tricky.

The simple way to think about that is to just verify it against ephemeral headers, which works fine. But the problem appears on when you start thinking about re-orgs and deleting content that no longer belongs to the canonical chain.

I will try to explain problem and few possible solutions and their drawbacks.

Problem

Let's refer to non-ephemeral state content as finalized. This would be either the same as non-ephemeral content from history (since last historical summaries, i.e. at most 8192 blocks) or potentially tied to finality of the chain (at most ~64 blocks). Concept is the same, and there is no big changes on how we handle it one way or the other.

Let's assume that we want to gossip state content only once, at the head of the chain (ephemeral) and that clients will keep them as ephemeral until finality as achieved. At that point, client will either migrate it to permanent storage or delete it (depending on whether content is part of the canonical chain at that point).

Since state content, unlike history content, can exist on both canonical and non-canonical versions of the chain, we can't easily detect which is the case. To be able to do so, we need to associate state content with every slot/block_hash/state_root that contains it.

This is the root of the problem. Content key is the same, regardless if it is associated with a different block. Therefore, client would reject all except the first offer, resulting in content being associated with only one slot/block_hash.

Some potential solutions

Most of the solutions that I came up with have some drawback. Hopefully together we can come up with something great.

1. Re-gossip state content once finality is achieved

Basically all ephemeral state content can be held temporarily and dropped later.
New content will be gossiped and stored once it's finalized.
Clients can be smart and migrate ephemeral content that they have, if they know it belongs to canonical chain.

Drawbacks:

• State network is very big. Having to gossip it twice is a lot of work
• Clients should be smart on how long to keep ephemeral content after associated slot is finalized (in case associated block is non-canonical)

2. Accept offers for content that is already stored in ephemeral storage

This ensures that we can keep track of all blocks associated with state content. With this information, we know exactly which content to keep when finality is achieved.

Drawback: This opens us to thundering herd problem: we would accept content that is anchored to the same block multiple times (basically every time that it's offered)

3. Create new content key type that will be used for ephemeral state content

The new content key type will contain slot/block_hash, which will make it unique and differentiable during offer. Content id creating would have to ignore slot/block_hash, so that the same content end up on the same node (also where it should stay once block is finalized)

This is another way to keep track of all blocks that are associated with state content.

Drawback:

• We would need to change the protocol (should be simple as we are just adding 3 new content keys)
• Attacker can create this type of content for state content that is not created during ephemeral block, which would significantly increase the size of ephemeral content that we have to deal with. Might not be a problem, as we might be able to know that content already.

[bhartnett]

The way Besu (execution client) handles reorgs using the Bonsai trie format is to store some form of state diffs for each block in addition to the single current state trie so that when a reorg is required the client is able to rollback to the last finalized block and then roll forward on the updated/newer chain.

I wonder if we could store state diffs in portal network such that they could be requested and used by portal clients to migrate their state data from one fork to another. I'm assuming we are talking about the flat state structure where each portal client stores a slice of leafs in the network. Note that I don't fully have my head around how the proposed flat structure works so this is just a high level idea at this point for consideration.

[morph-dev]

I'm assuming we are talking about the flat state structure...

Yes, the flat structure would be able to use state-diffs to handle this better (I assume, more research needed). This issue is about trie based model that we currently have.

[bhartnett]

Yes, the flat structure would be able to use state-diffs to handle this better (I assume, more research needed). This issue is about trie based model that we currently have.

If the flat structure is better suited to handling re-orgs (assuming that turns out to be so) then we could decide to only support the trie based structure for finalized state. Would be less complexity in the network having only one of the state models having to handle re-orgs.

[morph-dev]

Yes, the flat structure would be able to use state-diffs to handle this better (I assume, more research needed). This issue is about trie based model that we currently have.

If the flat structure is better suited to handling re-orgs (assuming that turns out to be so) then we could decide to only support the trie based structure for finalized state. Would be less complexity in the network having only one of the state models having to handle re-orgs.

I was thinking about that as well, and it is a viable option.

But i think that flat model will rely on trie-based on model in order to heal and initialize the state. So we might have to solve it for trie based model as well.

Besides, we won't have trie-based model for a while, and we would like to have state at the head asap

[carver]

If we want to minimize transfer traffic, we could look at a hybrid of 1 & 3. Something like:

a) the bridge tracks any trie nodes that become non-canonical
b) the bridge tracks if those non-canonical nodes get recreated (ie~ the same trie hash) on a new canonical chain
c) at block finalization, for any tracked trie nodes from b), the bridge sends a new custom re-proof that anchors to the canonical finalized chain (needs a new content key, so that the offer is not ignored). The re-proof offer is only accepted if the content is stored locally, and the local proof is anchored to a different block hash than the proof is offering.

That adds a lot of work for the bridges of course, but the traffic savings in the state network might be worth it. Also, this whole approach could be massively reworked in a post-Verkle world. So this solution might not need to last longer than a few years.

[pipermerriam]

It seems that just doing trie for finalized is at least an option that allows us to move forward. I recognize that we probably want a solution for unfinalized HEAD, but for now, it seems implementing it for finalized only and then moving on to our flat model is the simplest path that has known solutions for most of the problems.

[pipermerriam]

To expand a little more on my thinking... if trie-based state is only available for finalized state then the problem of trie node proofs changing and of needing to cleanup re-org state seems to go away.

And this gives us good incentives to implement and ship flat state on a sooner timeline.

And I don't believe that trie-based state only being available for finalized blocks will have any kind of significant negative impact on the ability for nodes to fill in their flat-trie state. They will just only be able to fill that data in after whatever the most recent finalized checkpoint is.

All of this ignores a Holesky-like event on mainnet that causes a significant period of non-finalization... which we should think about some and decide how much we want or need to be resilient to such an event.

[KolbyML]

After we get the history network fully live with ephemeral headers and fix the performance bottlenecks in Trin. I am going to start championing for option 1. The flat-model is cool, but it seems like a longer term research project with the timeline of at least 2 years for it to be usable (having access to contract bytecode and storage).

With the shift in sentiment for the History Ephemeral Headers and the increased role bridges will play, I think it is appropriate to also do this trade off for the State Network.

I want Portal to be shipped and not just be a research project, I think it is very achievable with this trade off. For context from talking to @pipermerriam and @morph-dev it sounds like supporting contract storage/bytecode would add a lot of complexity to the proposed initial version of the HEAD state network. So I am fine with option 1, and I think it is the clearest way to shipping a working product for HEAD state which we can then further iterate on

[KolbyML]

For context as well 2 years ago everybody thought Verkle would be shipped in 2 years. People have always said "Stateless will ship in 2 years". I don't want that to be the same problem that Portal runs into.