v2.1.0

🐍🏖️ Cancun StateTest Format Bug Fix

Release v2.1.0 primarily fixes a small bug introduced within the previous release where transition forks are used within the new StateTest format. This was highlighted by @chfast within #405 (#405), where the fork name ShanghaiToCancunAtTime15k was found within state tests.

🧪 Test Cases

• ✨ EIP-4844: Adds test_blob_gas_subtraction_tx() verifying the blob gas fee is subtracted from the sender before executing the blob tx (#407).

🛠️ Framework

• 🐞 State tests generated with transition forks no longer use the transition fork name in the fixture output, instead they use the actual enabled fork according to the state test's block number and timestamp (#406).

📋 Misc

• ✨ Use run-parallel and shared wheel packages for tox (#408).

Full Changelog: v2.0.0...v2.1.0

v2.0.0

🐍🏖️ Cancun and StateTest Format Release

Contains many important framework changes, including introduction of the StateTest format, and some additional Cancun and other test coverage.

Due to changes in the framework, there is a breaking change in the directory structure in the release tarball, please see the dedicated "💥 Breaking Changes" section below for more information.

🧪 Test Cases

• ✨ EIP-4844: Add test_sufficient_balance_blob_tx() and test_sufficient_balance_blob_tx_pre_fund_tx() (#379).
• ✨ EIP-6780: Add a reentrancy suicide revert test (#372).
• ✨ EIP-1153: Add test_run_until_out_of_gas() for transient storage opcodes (#401).
• ✨ EIP-198: Add tests for the MODEXP precompile (#364).
• ✨ Tests for nested CALL and CALLCODE gas consumption with a positive value transfer (previously lacking coverage) (#371).
• 🐞 EIP-4844: Fixed test_invalid_tx_max_fee_per_blob_gas() to account for extra gas required in the case where the account is incorrectly deduced the balance as if it had the correct block blob gas fee (#370).
• 🐞 EIP-4844: Fixed test_insufficient_balance_blob_tx() to correctly calculate the minimum balance required for the accounts (#379).
• 🐞 EIP-4844: Fix and enable test_invalid_blob_tx_contract_creation (#379).
• 🔀 Convert all eligible BlockchainTests to StateTests (and additionally generate corresponding BlockchainTests) (#368, #370).

🛠️ Framework

• ✨ Add StateTest fixture format generation; StateTests now generate a StateTest and a corresponding BlockchainTest test fixture, previously only BlockchainTest fixtures were generated (#368).
• ✨ Add StateTestOnly fixture format is now available and its only difference with StateTest is that it does not produce a BlockchainTest (#368).
• ✨ Add evm_bytes_to_python command-line utility which converts EVM bytecode to Python Opcodes (#357).
• ✨ Fork objects used to write tests can now be compared using the >, >=, <, <= operators, to check for a fork being newer than, newer than or equal, older than, older than or equal, respectively when compared against other fork (#367).
• ✨ Add solc 0.8.23 support (#373).
• ✨ Add framework unit tests for post state exception verification (#350).
• ✨ Add a helper class ethereum_test_tools.TestParameterGroup to define test parameters as dataclasses and auto-generate test IDs (#364).
• ✨ Add a --single-fixture-per-file flag to generate one fixture JSON file per test case (#331).
• 🐞 Storage type iterator is now fixed (#369).
• 🐞 Fix type coercion in FixtureHeader.join() (#398).
• 🔀 Locally calculate the transactions list's root instead of using the one returned by t8n when producing BlockchainTests (#353).
• 🔀 Change custom exception classes to dataclasses to improve testability (#386).
• 🔀 Update fork name from "Merge" to "Paris" used within the framework and tests (#363).
• 💥 Replace = with _ in pytest node ids and test fixture names (#342).
• 💥 The StateTest, spec format used to write tests, is now limited to a single transaction per test (#361).
• 💥 Tests must now use BlockException and TransactionException to define the expected exception of a given test, which can be used to test whether the client is hitting the proper exception when processing the block or transaction (#384).
• 💥 fill: Remove the --enable-hive flag; now all test types are generated by default (#358).
• 💥 Rename test fixtures names to match the corresponding pytest node ID as generated using fill (#342).

📋 Misc

• ✨ Docs: Add a "Consuming Tests" section to the docs, where each test fixture format is described, along with the steps to consume them, and the description of the structures used in each format (#375).
• 🔀 Docs: Update t8n tool branch to fill tests for development features in the readme (#338).
• 🔀 Filling tool: Updated the default filling tool (t8n) to go-ethereum@master (#368).
• 🐞 Docs: Fix error banner in online docs due to mermaid syntax error (#398).
• 🐞 Docs: Fix incorrectly formatted nested lists in online doc (#403).

💥 Breaking Changes

A concrete example of the test name renaming and change in directory structure is provided below.

1. Fixture output, including release tarballs, now contain subdirectories for different test types:

   1. blockchain_tests: Contains BlockchainTest formatted tests
   2. blockchain_tests_hive: Contains BlockchainTest with Engine API call directives for use in hive
   3. state_tests: Contains StateTest formatted tests

2. StateTest, spec format used to write tests, is now limited to a single transaction per test.

3. In this release the pytest node ID is now used for fixture names (previously only the test parameters were used), this should not be breaking. However, = in both node IDs (and therefore fixture names) have been replaced with _, which may break tooling that depends on the = character.

4. Produced blockchain_tests fixtures and their corresponding blockchain_tests_hive fixtures now contain the named exceptions BlockException and TransactionException as strings in the expectException and validationError fields, respectively. These exceptions can be used to test whether the client is hitting the proper exception when processing an invalid block.

   Blockchain test:

   "blocks": [
         {
            ...
            "expectException": "TransactionException.INSUFFICIENT_ACCOUNT_FUNDS",
            ...
         }
         ...
   ]

   Blockchain hive test:

   "engineNewPayloads": [
         {
            ...
            "validationError": "TransactionException.INSUFFICIENT_ACCOUNT_FUNDS",
            ...
         }
         ...
   ]

Renaming and Release Tarball Directory Structure Change Example

The fixture renaming provides a more consistent naming scheme between the pytest node ID and fixture name and allows the fixture name to be provided directly to pytest 5on the command line to execute individual tests in isolation, e.g. pytest tests/frontier/opcodes/test_dup.py::test_dup[fork_Frontier].

1. Pytest node ID example:

   1. Previous node ID: tests/frontier/opcodes/test_dup.py::test_dup[fork=Frontier].
   2. New node ID: tests/frontier/opcodes/test_dup.py::test_dup[fork_Frontier].

2. Fixture name example:

   1. Previous fixture name: 000-fork=Frontier
   2. New fixture name: tests/frontier/opcodes/test_dup.py::test_dup[fork_Frontier] (now the same as the pytest node ID).

3. Fixture JSON file name example (within the release tarball):

   1. Previous fixture file name: fixtures/frontier/opcodes/dup/dup.json (BlockChainTest format).
   2. New fixture file names (all present within the release tarball):
   • fixtures/state_tests/frontier/opcodes/dup/dup.json (StateTest format).
   • fixtures/blockchain_tests/frontier/opcodes/dup/dup.json (BlockChainTest format).
   • fixtures/blockchain_tests_hive/frontier/opcodes/dup/dup.json (a blockchain test in HiveFixture format).

New Contributors

• @jochem-brouwer made their first contribution in #364
• @MukulKolpe made their first contribution in #363
• @acolytec3 made their first contribution in #371
• @richardgreg...

v1.0.6

🐍🏖️ Cancun Devnet 10

🧪 Test Cases

• 🔀 EIP-4844: Update KZG point evaluation test vectors to use data from the official KZG setup and Mainnet Trusted Setup (#336).

🛠️ Framework

• 🔀 Fixtures: Add a non-RLP format field (rlp_decoded) to invalid blocks (#322).
• 🔀 Spec: Refactor state and blockchain spec (#307).

🔧 EVM Tools

• ✨ Run geth's evm blocktest command to verify JSON fixtures after test case execution (--verify-fixtures) (#325).
• ✨ Enable tracing support for ethereum-spec-evm (#289).

📋 Misc

• ✨ Tooling: Add Python 3.12 support (#309).
• ✨ Process: Added a Github pull request template (#308).
• ✨ Docs: Changelog updated post release (#321).
• ✨ Docs: Add a section explaining execution-spec-tests release artifacts (#334).
• 🔀 T8N Tool: Branch used to generate the tests for Cancun is now lightclient/go-ethereum@devnet-10 (#336)

💥 Breaking Change

• Fixtures now use the Mainnet Trusted Setup merged on consensus-specs#3521 (#336)

Full Changelog: v1.0.5...v1.0.6

v1.0.5

🐍🏖️ Cancun Devnet 9 Release 3

This release mainly serves to update the EIP-4788 beacon roots address to 0x000F3df6D732807Ef1319fB7B8bB8522d0Beac02, as updated in ethereum/EIPs/pull/7672.

🧪 Test Cases

• 🐞 EIP-4844: Fix invalid blob txs pre-Cancun engine response (#306).
• ✨ EIP-4788: Final update to the beacon root address (#312).

📋 Misc

• ✨ Docs: Changelog added (#305).
• ✨ CI/CD: Run development fork tests in Github Actions (#302).
• ✨ CI/CD: Generate test JSON fixtures on push (#303).

---

💥 Breaking Change

Please use development fixtures from now on when testing Cancun. These refer to changes that are currently under development within clients:

• fixtures: All tests until the last stable fork (Shanghai)
• fixtures_develop: All tests until the last development fork (Cancun)
• fixtures_hive: All tests until the last stable fork (Shanghai) in hive format (Engine API directives instead of the usual BlockchainTest format)
• fixtures_develop_hive: All tests until the last development fork (Cancun) in hive format

v1.0.4

🐍 Cancun Devnet 9 Release 2

This release adds additional coverage to the current set of cancun tests, up to the Devnet-9 Cancun specification, noting the following considerations:

• Beacon root contract is pre-deployed at 0xbEAC020008aFF7331c0A389CB2AAb67597567d7a
  • Additional EIP-4788 updates from the following PR will be included in the next release.
• Adds EIP-7516 Tests

🧪 Tests include the following EIPs

EIP-4844: Shard Blob Transactions
EIP-5656: MCOPY - Memory copying instruction
EIP-6780: SELFDESTRUCT only in same transaction
EIP-4788: Beacon block root in the EVM
EIP-1153: Transient storage opcodes
EIP-7516: BLOBBASEFEE opcode

⚙️ Test Generation

Fixtures for these tests can be generated using the geth t8n tool with the following branch & command:
marioevz/go-ethereum@cancun-t8n

fill --until Cancun

📄 Documentation

Further information on each test can be found within the docs:
https://ethereum.github.io/execution-spec-tests/main/tests/cancun/

What's Changed

• tests/cancun/eip_1153: Remove conftest '+1153' in network field. by @spencer-tb in #299
• improvement: Deprecate empty accounts within framework. by @spencer-tb in #300
• tests/4844: Add EIP-7516, BLOBBASEFEE, verification to excess blob gas tests by @marioevz in #294
• tests/cancun/eip_4788: Update predeploy address(0xbEAC02..7d7a), add additional tests by @spencer-tb in #297
• feature: Add engine API forkchoice updated information in fixtures by @spencer-tb in #256
• improvement: Fixture generation split based on hive specificity by @spencer-tb in #301

Full Changelog: v1.0.3...v1.0.4

v1.0.3

🐍 Cancun Devnet 9 Release

This release adds additional coverage to the current set of cancun tests, up to the Devnet-9 Cancun specification, noting the following considerations:

• Beacon root contract is pre-deployed at 0xbEac00dDB15f3B6d645C48263dC93862413A222D
  • Additional EIP-4788 updates from the following PR will be included in the next release.
• Further coverage includes tests for EIP-1153, and additional tests for EIP-4788 & EIP-6780.

🧪 Tests include the following EIPs

EIP-4844: Shard Blob Transactions
EIP-5656: MCOPY - Memory copying instruction
EIP-6780: SELFDESTRUCT only in same transaction
EIP-4788: Beacon block root in the EVM
EIP-1153: Transient storage opcodes

⚙️ Test Generation

Fixtures for these tests can be generated using the geth t8n tool with the following branch & command:
marioevz/go-ethereum@cancun-t8n

fill --until Cancun

📄 Documentation

Further information on each test can be found within the docs:
https://ethereum.github.io/execution-spec-tests/main/tests/cancun/

v1.0.2

🐍 Cancun Devnet 8 + 4788 v2 Pre-Release

This pre-release updates all cancun tests according to Update EIP-4788: initial stab at v2, and the Devnet-8 Cancun specification, with a few considerations:

• Beacon root contract is pre-deployed at address 0x000000000000000000000000000000000000000b instead of the address described in the EIP PR
• No system transactions are executed for the generation of the fixtures and instead a direct storage update is performed at the beginning of the block

🧪 Tests include the following EIPs

EIP-4844: Shard Blob Transactions
EIP-5656: MCOPY - Memory copying instruction
EIP-6780: SELFDESTRUCT only in same transaction
EIP-4788: Beacon block root in the EVM

Our full release will include tests for EIP-1153: Transient storage opcodes including additional coverage for the above EIPs.

⚙️ Test Generation

Fixtures for these tests can be generated with the following geth commit:
marioevz/go-ethereum@7c8719cddf12097101ccbdec8c9bbd7cf7686e4e

fill --until Cancun

📄 Documentation

Further information on each test can be found within the docs:
https://ethereum.github.io/execution-spec-tests/main/tests/cancun/

v1.0.1

🐍 Cancun Devnet-8 Pre-Release

This pre-release holds our initial set of tests for the current Devnet-8 Cancun specification.

🧪 Tests include the following EIPs

EIP-4844: Shard Blob Transactions
EIP-5656: MCOPY - Memory copying instruction
EIP-6780: SELFDESTRUCT only in same transaction
EIP-4788: Beacon block root in the EVM

Our full release will include tests for EIP-1153: Transient storage opcodes including additional coverage for the above EIPs.

⚠️ Note
Beacon block root pre-compile tests are to be updated following its change to a system call:
https://ethereum-magicians.org/t/eip-4788-beacon-root-in-evm/8281/33

⚙️ Test Generation

Fixtures for these tests can be generated on main with the following geth branch:
spencer-tb/go-ethereum@4844-devnet-6-t8n-update

fill --from Cancun --until Cancun

📄 Documentation

Further information on each test can be found within the docs:
https://ethereum.github.io/execution-spec-tests/main/tests/cancun/

v1.0.0

🧪 Welcome to the PyTest Era

We are extremely happy to announce this major new release as it marks a huge milestone for execution-spec-tests.

With this release, adding tests has become easier than ever, so we invite and encourage all client developers to seize this opportunity and contribute to the continuous enhancement of the ethereum testing ecosystem.

🗝️ Key Changes

• Pytest takes the stage: Introducing the upgrade to pytest, now used to collect and execute tests, including generation of fixtures.
• Shiny new documentation: A full revamp of the docs which is now available at ethereum.github.io/execution-spec-tests.
• Tests by fork: Tests (previously named fillers) and fixtures, are now re-organized by fork name for improved clarity and usage.
• Commands with comfort: Tests are now filled with a new user friendly and readable CLI, including a new additional option to specify the solc binary.
• Write tests like a pro: the repo now includes recommended extensions & settings for a seamless new developer start.

Full Changelog: v0.2.5...v1.0.0

🐍 Tests

Updates to Existing Tests

• fixtures.tar.gz includes a refresh of previous State & Blockchain tests, up to and including Shanghai:
  • Each test has been refactored and now utilizes Pytest parameterization. These can be viewed within the test case reference section of the docs.

New EIP-4844 Tests

• fixtures-cancun.tar.gz includes a full set of Blockchain tests for EIP-4844, up to date with the latest Devnet 6 & 7 changes

🛠️ Usage

All tests can be filled with the new CLI using the fill command. (Note: tf is deprecated and will be removed in the next release).

By default fill generates fixtures for all tests up to the current mainnet fork - Shanghai. Detailed information on the new CLI can be found within the docs.

Generating EIP-4844 Tests

To only generate the new EIP-4844 tests the following command can be run:

fill -v tests/cancun --until=Cancun

Versions Used

Main fixtures: ethereum/go-ethereum @ c7b099b go-ethereum branch.

Cancun fixtures: marioevz/go-ethereum @ d36b63d go-ethereum branch.

Additional dependency versions used (others may work):

• Solidity @ 0.8.20
• Python @ 3.10.6

🐝 Pyspec Hive Simulator

Each EIP-4844 test case within fixtures-cancun.tar.gz can additionally be run within the updated pyspec hive simulator. It sends each block within a test fixture as a payload to the execution client using the Engine API.

💬 Contact Us

We are here to help!

If you stumble over any issues with this release please reach out to us.

You can find our contact details within the docs!

v0.2.5

Tests

Cancun

• Separate fixtures to test for Cancun: includes the first version of the 4844 fixtures. (#63)
• These currently test for valid/invalid blob transactions, excess data gas calculations and the DATAHASH opcode.
• Additional tests for kzg pre-compiles and ssz will be added in a future release.
• Test documentation can be found -> here.
• These will be run within the hive pyspec simulator on this branch.

Important: These tests can only be filled using the geth version listed below, alongside the --latest-fork Cancun flag. As an example, you can run the 4844 tests with the following command: tf --force-refill --latest-fork Cancun --test-categories eips.eip4844.

Pre-Cancun

• New filler category added for security tests, used to verify previously identified client vulnerabilities. (#112)
• Access list test case example filler added. (#83)

Changes

• Reference spec feature added to fillers & fixtures. (#81)
• Fork object feature added to fillers. (#91)
  • Can now write tests for fork transitions.
• Callable Op-codes feature added reducing the dependency of Yul code (#102)
  • Most filler modules are now entirely python.
• Access list type support added. (#83)
• Documentation updated to include 4884 test descriptions -> here.

Fixes

• Small bug fixes on the fixture format (#80 #92 #109)
• Correctly handle the processing of negative integer indexes in storage (#106)

Versions Used

go-ethereum: mdehoog/go-ethereum@ac64c44