Fix and test fork revert logic

[michaelsproul]

Description

Lighthouse's mechanism for recovering from missed hard forks is currently broken. It is implemented in the fork_revert module, but hasn't worked since Altair.

Part of the reason for this is that it is hard to test comprehensively. The ideal test depends on access to prior versions of Lighthouse which aren't available on CI (yet). The test looks something like this:

1. Run a testnet with two types of nodes:
   • Canonical chain: latest Lighthouse version, and fork epoch set for the latest hard fork (e.g. Capella).
   • Stale chain: previous Lighthouse version and no fork epoch set.
2. Wait until the testnet has advanced past the configured fork epoch. The canonical chain should continue (and finalize) with new blocks, while the stale chain also continues. The nodes will likely disconnect from each other on P2P.
3. Shut down all the stale nodes and restart them with the latest version of Lighthouse & with the fork epoch configured. Ensure that they don't crash on startup and sync back up to the canonical chain.

There's a lot of local testnet infra described here which is currently not really up to scratch. We likely need changes like #3807 to land first so we can test these scenarios.

There's also a more minor guarantee that we can test on CI without access to prior versions, which is that the current version of Lighthouse can revert a fork missed by the same version. We could likely test this as a beacon_chain test and it would get us ~50% of the way towards a more robust fork_revert module.

TODO

• Fix fork_revert logic
• Basic test (single version)
• Comprehensive test (multiple versions)

[danielrachi1]

I'd like to work on this issue. I have a few questions before I start:

Are there any existing tests for the fork revert logic? I didn't find anything under the beacon chain tests. I think I would need to implement at least one of the proposed tests before fixing the fork_revert module.

If I understood correctly, I could implement a test that checks if the current version of Lighthouse can revert a fork missed by the same version without having to wait for #3807 to be merged. So writing this (local and for CI) could be a good place to start. What do you think?

[michaelsproul]

@danielrachi You're right that there aren't any existing tests for fork_revert. A good first step would be to write one, maybe in beacon_chain/tests. You're right that you won't need the local testnet stuff from #3807 in order to do it

[danielrachi1]

I'm no longer working on this issue. If anyone else is interested, feel free to take it.

[danielrachi1]

I'm re-taking this issue. I'll open a PR as soon as I have progress to show. 🫡