Tracker: Faster, More Targeted Test Fixture Releases

[danceratopz]

This issue encompasses 2 goals for test fixture releases:

1. Higher mainnet release cadence, i.e., more regular releases of test cases for stable forks.
2. Faster release builds, in particular, for development releases. i.e., we can quickly generate releases that target specific development work (e.g., bal-devnet-3 or benchmark).

Goal 2. could also help unlock more finely grained releases, e.g., for every individual EIP, e.g., eips/amsterdam/eip-2780, but we should verify with client teams whether this would be beneficial - our releases should match client teams branches as close as possible.

Important

Implication: If we only fill for the development fork in dev releases, clients need to consume two artifacts from two separate releases for full coverage and no regressions:

1. fixtures_mainnet.tar.gz.
2. fixtures_<dev_feature_name>.tar.gz.

Subtasks

• feat(test-fill): enable phase-1-only pre-alloc generation #2720
• feat(test-split): add grouped-split pytest plugin #2721
• Implement use of pytest-split in release_fixture_feature.yaml.
• Enable multi-release support to consume cache --input (i.e., allow the input to be a comma-separated list of releases) to allow full coverage runs in the consume Hive simulators.
• Socialize new release model with client teams: New mainnet cadence and the two-artifact consumption model.
• Follow up with client teams on their development branch structures: Input for whether finer-grained releases (e.g., per-EIP) would be useful.
• Update docs for removal of fixtures_stable/fixtures_develop -> fixtures_mainnet.

Higher Mainnet Release Cadence

#2592 mainly solved 1., but requires team agreement/alignment and some follow-up work (only docs?). Essentially, as of now, we can readily create fixtures_mainnet (~1hr30). More info below, but TLDR: fixtures_mainnet replaces fixtures_stable and fixtures_develop has been removed.

Faster Release Builds

This issue proposes that we achieve faster development release builds by:

1. Only filling the development fork in development releases — e.g., for devnet releases like bal-devnet-4, --until=Amsterdam (previously used for client-team convenience) would change to --fork=Amsterdam.
2. Improving the parallelization strategy in multi-runner release generation from a naive per-fork split to use pytest-split (or other strategy) that load balances the run across multiple runners.

Target: Difficult to judge upfront, but >2x speedup vs. current development release build time via fork split.

We have two bottlenecks in fixture generation: Compute and Memory. #2592 parallelized compute by splitting test fixture generation over multiple runners by fork (facilitated by #2134). But this approach has the following drawbacks:

1. Single-fork releases, e.g., benchmark, can not be parallelized.
2. It's inefficient as runners are not well-balanced.
3. Doesn't scale with new forks.

Btw, regarding memory, @fselmo has currently hit this hard and the situation should greatly improve. This issue only targets compute, currently.

Status Quo (pre-#2592): EEST Release Types

Up to and including Osaka, until #2592, we had two types of releases:

1. "Full releases" which consist of two fixture artifacts:
   • fixtures_stable: fixtures for deployed forks.
   • fixtures_develop: superset of fixtures_stable plus fixtures for the current development fork.
2. "Feature releases" of development forks which contain one artifact:
   • fixtures.

These release types originated in ethereum/execution-spec-tests and predate The Weld. In execution-spec-tests, multiple development forks resided alongside each other in the default branch (main). This is no longer the case in execution-specs where a branch only ever contains code targeting one development fork.

#2592 shook things up a little and removed "full" releases due to the following reasons:

1. The develop artifact contained stable as a subset (redundancy).
2. They're expensive to produce. The "full" release even generated stable and develop independently, i.e., all test cases filled for stable forks were filled twice!
3. These releases were generated from the default branch, which doesn't reflect the current status of the development fork. Post-Weld, fixtures_develop gets quickly out-of-date with the current development focus.

[taratorio]

Had a conversation with @felix314159 on discord about this and wanted to summarise my thoughts here.

I find the 2 "_stable" vs "_develop" tars in 1 release a bit confusing. For example, v5.4.0 has been released after Osaka went live on mainnet but 1) the "stable" tar doesn't have all the Osaka EIPs under "blockchain_tests/osaka" and 2) the few EIP tests under "blockchain_tests/osaka" have "network": "Prague" in them. So this releaase doesn't match the description of stable vs develop at https://eest.ethereum.org/main/running_tests/releases/#standard-releases

My suggestion is to decouple the "stable" releases from the "devnet" releases and have only 1 fixtures tar file in a release (instead of the 2 stable and develop that we have now) and let them have their separate naming scheme and versioning. For example:

• "stable@v5.5.0" -> "stable@v5.6.0"
• "amsterdam@v1.0.0" -> "amsterdam@v2.0.0"
• once Amsterdam is live on mainnet we release a new e.g. "stable@v5.7.0" release which contains amsterdam
• then "hegota@v1.0.0" -> "hegota@v2.0.0"

For reference, what we started doing in Erigon recently is we have 2 flavours of CI jobs running on every commit:

• tests for up to and including the latest live hard fork (e.g. Osaka via v5.4.0 "_develop" because "_stable" is inaccurate)
• tests for only the current devnet (e.g. "bal@v5.7.0")

So for us having 2 separate releases for each of the 2 use cases makes sense

[danceratopz]

Thanks for the feedback @taratorio.

I find the 2 "_stable" vs "_develop" tars in 1 release a bit confusing.

Agreed 💯, we frequently get asked the difference between these two tarballs!

My suggestion is to decouple the "stable" releases from the "devnet" releases and have only 1 fixtures tar file in a release (instead of the 2 stable and develop that we have now) and let them have their separate naming scheme and versioning.

Yes this is exactly what this issue is proposing above, sorry if that wasn't clear!

"stable@v5.5.0" -> "stable@v5.6.0"
"amsterdam@v1.0.0" -> "amsterdam@v2.0.0"
once Amsterdam is live on mainnet we release a new e.g. "stable@v5.7.0" release which contains amsterdam
then "hegota@v1.0.0" -> "hegota@v2.0.0"

As per suggestion in the issue:

• "stable@v5.5.0" -> "mainnet@v6.0.0" ("stable" becomes "mainnet")

And yes to the rest, but I would suggest that we could even move Amsterdam to "mainnet" around the time that clients publish production releases with Amsterdam support (before the hard fork). The aim being to get our releases to match client releases.

Indeed, it'd be great if your releases also matched client teams dev branch structure as much as possible,...

So quick question for you @taratorio but it goes to all client devs: Would finer-granularity releases be helpful? E.g., A release targeting a single EIP?

[taratorio]

@danceratopz great, all sounds good. As per the release per EIP - that sounds like it might be useful to gradually roll out EIP changes to our main/devnet branches. It's been a bit of a pain now when a new devnet release drops we have to merge all feature branches into 1 place and then start going through the test failures because EIPs are interrelated to each other and in most cases you need all EIP changes in 1 place to get tests to pass. A release per EIP will make this much easier to manage and rollout sequentially.

[spencer-tb]

@danceratopz great, all sounds good. As per the release per EIP - that sounds like it might be useful to gradually roll out EIP changes to our main/devnet branches. It's been a bit of a pain now when a new devnet release drops we have to merge all feature branches into 1 place and then start going through the test failures because EIPs are interrelated to each other and in most cases you need all EIP changes in 1 place to get tests to pass. A release per EIP will make this much easier to manage and rollout sequentially.

For testing EIPs in isolation should they be based on Amsterdam or the last devnet?

[taratorio]

@danceratopz great, all sounds good. As per the release per EIP - that sounds like it might be useful to gradually roll out EIP changes to our main/devnet branches. It's been a bit of a pain now when a new devnet release drops we have to merge all feature branches into 1 place and then start going through the test failures because EIPs are interrelated to each other and in most cases you need all EIP changes in 1 place to get tests to pass. A release per EIP will make this much easier to manage and rollout sequentially.

For testing EIPs in isolation should they be based on Amsterdam or the last devnet?

From my experience usually the process is sequential, so I would say based on the last devnet (or mainnet if it is the first devnet after an upgrade). Btw, I didn't quite mean 1 release for 1 EIP only, but more that as soon as tests for 1 EIP are implemented on top of the last devnet/mainnet - create a release. Then once tests for another EIP are ready - create a release. It creates a sequence of releases that progressively add 1 new EIP to the mix at a time. This way, if I follow your order of implementation I would only have to work on fixing tests for 1 EIP at a time and progressively bump up the version of fixtures.

[chfast]

Please make the releases more often. I regularly submit new tests but then have to wait months before I can use these.