EIP-7917: Deterministic proposer lookahead

[linoscope]

Edit: Ethereum Magicians post for any general discussion around the EIP: https://ethereum-magicians.org/t/eip-7917-deterministic-proposer-lookahead/23259

Currently, the proposer lookahead for the next epoch is unstable because the effective balance (EB) may change before the next epoch due to slashing events, deposits, or withdrawals occurring in the current epoch. The recent increase in the maximum EB further exacerbates this instability, as EB can now exceed the previous 32 ETH cap. This instability complicates the operation of preconf protocols, as described in this document: https://hackmd.io/@linoscope/preconf-lookahead

In this PR, we address this issue by stabilizing the proposer lookahead for the next epoch. We achieve this by calculating and recording the lookahead for epoch N+1 in the beacon state at the beginning of epoch N.

Furthermore, a highly valuable side effect of this approach is that the proposer lookahead becomes available within the beacon state, making it possible to retrieve the lookahead in the EVM using the beacon root and Merkle proofs. Having direct access to lookahead in the EVM greatly simplifies implementing the on-chain components of preconf protocols.

TODO:

• Add more text to the specs
• Add more tests (EB changes, fork activation transition, etc).
• Assign a proper EIP number

[ralexstokes]

an immediate step will be targeting fulu instead of electra :)

[linoscope]

an immediate step will be targeting fulu instead of electra :)

Ah, thanks for pointing out. Fixed in 0a37fe7

[JustinDrake]

Great job Lin :)

[mkalinin]

Proposer lookahead is a new constraint for the protocol which for instance constraining approaches with secret leader election that has been explored in the past. If we accept this change, we should explicitly mention that we give up on anything like SSLE unless I misunderstood something

[linoscope]

Proposer lookahead is a new constraint for the protocol which for instance constraining approaches with secret leader election that has been explored in the past. If we accept this change, we should explicitly mention that we give up on anything like SSLE unless I misunderstood something

Thanks for bringing this up! Indeed, SSLE is something to consider - But I am not sure if we have to go so far as to "give up" on SSLE with this change. This PR is just making the existing design of having multiple epochs worth of lookahead (having MIN_SEED_LOOKAHEAD) more explicit. If we are to implement SSLE, we will have to drastically change how the proposer_lookahead field is handled or even deprecate it, and all preconf protocols will need to adjust. But they will have to make drastic changes anyway with SSLE regardless of this, so I don't think it's a blocker on the preconf protocol side. Or do you see this being a potential blocker on the CL/spec side for SSLE?

[mkalinin]

Or do you see this being a potential blocker on the CL/spec side for SSLE?

Each new protocol constraint is hard to remove when it becomes a downstream dependency. One of the ways to go would probably be to say that there can be no lookahead if SSLE is introduced in the future, explicitly emphasising that this feature is unreliable long term

[linoscope]

Each new protocol constraint is hard to remove when it becomes a downstream dependency.

Makes sense, rather than removing the field, just having an empty list for proposer_lookahead would be better in terms of downstream impact. And that is exactly what SSLE is anyway, having zero lookahead.

One of the ways to go would probably be to say that there can be no lookahead if SSLE is introduced in the future, explicitly emphasising that this feature is unreliable long term

This makes sense too, and I agree it's worth mentioning in specs and EIP. Thanks for the feedback!

[potuz]

Is there a security analysis anywhere for this feature? There are good descriptions of the current mechanism on @benjaminion
book https://eth2book.info/capella/part2/building_blocks/randomness/ and @vbuterin https://notes.ethereum.org/@vbuterin/SkeyEI3xv#Aside-RANDAO-seeds-and-committee-generation and https://web.archive.org/web/20160723105229/https://vitalik.ca/files/randomness.html

Describing how the preferred solution would be to get proposal rights JIT, which this PR takes us further away from it.

Besides a security analysis of knowing the proposing schedule ahead of time, I agree with @mkalinin that this proposal seems to give up on SSLE as it's currently developed.

[terencechain]

Side note: wondering if it makes sense to update get_beacon_proposer_index to use state.proposer_lookahead feels like a nice optimization here

[linoscope]

Thanks for the feedback @potuz !

Is there a security analysis anywhere for this feature? There are good descriptions of the current mechanism on @benjaminion book https://eth2book.info/capella/part2/building_blocks/randomness/ and @vbuterin https://notes.ethereum.org/@vbuterin/SkeyEI3xv#Aside-RANDAO-seeds-and-committee-generation and https://web.archive.org/web/20160723105229/https://vitalik.ca/files/randomness.html

Describing how the preferred solution would be to get proposal rights JIT, which this PR takes us further away from it.

Yeah, security definitely needs consideration, and I’ll add a section in the upcoming EIP. In short, this proposal does not alter the “RANDAO delay” used in the lookahead—the lookahead of epoch N is still determined by the RANDAO of epoch N - MIN_SEED_LOOKAHEAD - 1 (which becomes available at the start of epoch N - MIN_SEED_LOOKAHEAD). The only difference is that it changes the “effective balances delay”: rather than using the effective balances (EB) at the start of epoch N, it now uses the EB at the start of epoch N - MIN_SEED_LOOKAHEAD.

And actually, by aligning the RANDAO and effective balances in this way, the proposal removes any chance of validators adjusting their EB after seeing the RANDAO outcome, which is an attack vector to consider. No such attack has been found so far, but this change removes the possibility, hence simplifying the security analysis.

Besides a security analysis of knowing the proposing schedule ahead of time, I agree with @mkalinin that this proposal seems to give up on SSLE as it's currently developed.

I don’t think we need to "give up" on SSLE. From what I’ve heard, the currently proposed SSLE constructions still include a lookahead, albeit an encrypted one. In those designs, we could reuse the proposer_lookahead field by changing its type to something like List[EncryptedValidatorIndex]. And if a construction were to remove lookahead entirely, we could simply set proposer_lookahead to an empty list, meaning this wouldn’t be a blocker.

That said, any such changes would introduce additional complexity around preconfs, but that complexity arises regardless of this change.

[linoscope]

Side note: wondering if it makes sense to update get_beacon_proposer_index to use state.proposer_lookahead feels like a nice optimization here

Makes total sense! And actually we already do this here: https://github.com/ethereum/consensus-specs/pull/4190/files#diff-bcf6567bc8bd905058f44340eeb4ec690c396543644748b7fb5918e4e4d15141R193

[dapplion]

RE SSLE: The key feature of secret leader election is: "the proposer at slot N is only known by the proposer at slot N with some lookahead". Then no-one can DoS the proposer, only if the proposer keeps its proposal slot private. This is completely opposite to the pre-confirmation requirement: "users (i.e. everyone) know the proposer at slot N with some lookahead".

If we enshrine the users' need to know the proposer ahead of time, under SSLE the proposer will be incentivized to make its proposal public. Implementing SSLE under this reality is technically possible but probably useless. Unless there's some intermediary that's doubly trusted (i.e. the builder) to whom the proposal can reveal its proposal slot, and then the builder somehow handles the pre-conf.

[jtraglia]

This looks really great @linoscope, thanks for your work on this!

[leolara]

Now the generator passes, and I added some prints and it passes in the first round with 4 epochs created at the beginning.

[potuz]

I think this EIP should also modify get_proposer_head, should_override_fcu and is_shuffling_stable. They may be backported to phase0 perhaps, but they need to be modified nonetheless.

[Co1nB3e]

What happens if the looked ahead approver is unable to validate the block (e.g. disconnected)?
It would go to the next one approver in the list? Would it delay slot/epoch? ...

Also, shouldn't approver be prevented from changing his EB during his epoch (e.g. withdrawal)?

[potuz]

Why is this a list instead of a vector?, if this is a list it is pretty weird to have a tight limit for it. I would go with an SSZ Vector in this case, but if we insist in having a list please set a larger limit.

[linoscope]

Good catch, indeed a vector is more natural then a list, as it is fixed in size. Will change to vector

[linoscope]

Changed to vector in c8811a0

[potuz]

All calls to this function explicitly compute seed at the callsite, it is cleaner to have a different signature only taking the state and epoch and computing the seed within this function.

Moreover, the same applies to the indices argument, this function should be defined as follows

def compute_proposer_indices(state: BeaconState, epoch: Epoch) -> List[ValidatorIndex, SLOTS_PER_EPOCH]:
    """
    Return the proposer indices for the given ``epoch``.
    """
    start_slot = compute_start_slot_at_epoch(epoch)
    seed = get_seed(state, epoch, DOMAIN_BEACON_PROPOSER)
    seeds = [hash(seed + uint_to_bytes(Slot(start_slot + i))) for i in range(SLOTS_PER_EPOCH)]
    indices = get_active_validator_indices(state, epoch)
    return [compute_proposer_index(state, indices, seed) for seed in seeds]
``

[linoscope]

Current interface was from a suggestion to make it more in line with the interface of compute_proposer_index, which takes the seed and indices in the argument. Personally don't have strong preference either way, but I do see the benefit of reducing repeated logic in callsites, so right now slightly in favor of reverting back to compute_proposer_indices(state: BeaconState, epoch: Epoch) interface as @potuz suggests. What do you think @gfukushima ?

[ensi321]

In the current spec, compute_proposer_index() is meant to be a private/internal/utility function that is only used by get_beacon_proposer_index().

The usage of compute_proposer_indices() is more similar to get_beacon_proposer_index() than compute_proposer_index(). Delegating the seed calculation to the caller is not really necessary imo.

If we really want to be consistent with the current spec,
we should have get_beacon_proposer_indices(state, epoch) which calls compute_proposer_indices (state, epoch, seed)

[linoscope]

Thanks both for the input!

The usage of compute_proposer_indices() is more similar to get_beacon_proposer_index() than compute_proposer_index(). Delegating the seed calculation to the caller is not really necessary imo.

Makes sense.

Made the call to circle back to having seeds and indicies in compute_proposer_indicies here: e9266b2

[gfukushima]

Agree with @ensi321 what I was trying to tell you with my initial comment is that the initial implementation was introducing a circular dependency, having a method in the misc helpers (compute_proposer_indices) depending on methods of the beacon state accessor (get_seed,get_active_validator_indices) would be unusual (if not a bad practice). It is usually the other way around, beacon state accessor can call misc helpers methods. Having said that I think @ensi321 suggestion is the closest I would agree with but with a slight modification: get_beacon_proposer_indices(state, epoch) should call compute_proposer_indices (state, epoch, seed, indices)

[linoscope]

having a method in the misc helpers (compute_proposer_indices) depending on methods of the beacon state accessor (get_seed,get_active_validator_indices) would be unusual (if not a bad practice)

Thanks for the explanation, I see your point, makes sense. Will make this change to remove the circular dependency.

[linoscope]

Refactor and introduction of get_beacon_proposer_indices accessor done here:
79bf80d

Thanks everyone for the input!

[linoscope]

I think this EIP should also modify get_proposer_head, should_override_fcu and is_shuffling_stable. They may be backported to phase0 perhaps, but they need to be modified nonetheless.

Had a nice discussion in Discord, copy-and-pasting my response/conclusion here for visibility:

I am having trouble to see whether we can remove is_shuffling_stable as I am having trouble understanding why it was included in the first place (there is no documentation in the specs). From looking at the original discussion, the only mention I see around epoch boundaries is in the context of attestations and justifications. Quote from @sproul

I think it's probably still better to play it safe at the epoch boundary. The block in slot 31 might include just enough attestations to justify the epoch (while the block at slot 30 does not). If we build a new block at 32 on 30, then it will have a lower justified checkpoint than 31, which I think will make it inferior from fork choice's PoV, even with the newest changes?

If it's about attestations and justifications, I don't think EIP-7917, which is about proposer lookahead, is relevant? The issue around "having just enough attesations to justify" at epoch boundaries seems still needed even with the new EIP. If anything I think the initial is_shuffling_stable naming was unfortunate, as it doesn't seem to be (at least only) about shuffling.

Unless we can say it is 100% safe to remove is_shuffling_stable, I am inclined to keep it, to play it safe.

[potuz]

LGTM

[leolara]

Regarding test_effective_balance_increase_changes_lookahead:

• Do we know what happens with the yields for the generator if the exception is raised? I guess perhaps it will overwrite but do we know?
• Do we know the reason that we need to do this non-deterministic thing? For everwhere I looked it seems deterministic and we would get the same result always, so perhaps there is an error in the test infra. What are the chances that we change the effective balance in all the validators and the future proposers don't change?

Anyway, I think it can be merged with this and we can find out about this in another PR.

We also have some tests that I want to add but I think we can do it in another PR, I don't have rights to create a PR againts this PR

[gfukushima]

lgtm

[jtraglia]

We also have some tests that I want to add but I think we can do it in another PR

Hey @leolara, yup let's follow up with more tests. I have a couple more in mind too.

[linoscope]

@leolara

Do we know what happens with the yields for the generator if the exception is raised? I guess perhaps it will overwrite but do we know?

Good question. I assumed yields with the same keys (e.g., "pre") overwrite the old one but haven't confirmed (perhaps @jtraglia knows).

Do we know the reason that we need to do this non-deterministic thing? For everwhere I looked it seems deterministic and we would get the same result always, so perhaps there is an error in the test infra. What are the chances that we change the effective balance in all the validators and the future proposers don't change?

We did face issue due to the non-deterministic RANDAO-dependent nature of the test before. Namely, tests would succeed when running in minimal preset but failed in mainnet preset. I did find that, with 4 iterations of next_epoch transitions we can get the test succeed for both minimal and mainnet, but still added the for loop that tests different number of iterations to be future proof against future changes to test environments.

We also have some tests that I want to add but I think we can do it in another PR, I don't have rights to create a PR againts this PR

Nice, thanks a lot! Looking forward to reviewing those. I also have some test ideas to potentially add (see if we properly handle active validator set changes), but I think that doesn't need to block this one too.

[ethDreamer]

LGTM!