feat(specs,tests): Update all benchmark tests to use gas_benchmark_value

[marioevz]

🗒️ Description

Add check for gas used in blockchain and state tests

blockchain_test and state_test now have a flag called expected_benchmark_gas_used which can be used to specify the expected gas used by the last payload (the benchmark payload).

If the value is not specified, the test is expected to use the value specified by --gas-benchmark-values entirely.

Modified all benchmark tests to use gas_benchmark_value

All benchmark tests now use gas_benchmark_value, and also expected_benchmark_gas_used if the test is not expected to use the entire gas (e.g. when the tx is not expected to run out-of-gas and instead consume a specific amount of gas).

🔗 Related Issues or PRs

N/A.

✅ Checklist

• All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:

  uvx --with=tox-uv tox -e lint,typecheck,spellcheck,markdownlint

• All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
• All: Considered adding an entry to CHANGELOG.md.
• All: Considered updating the online docs in the ./docs/ directory.
• All: Set appropriate labels for the changes (only maintainers can apply labels).

[jsign]

I'll take a look at this PR by tomorrow!

OK, pushed it for today :)

[marioevz]

Updated the branch with the following changes:

• Fix test_empty_block so it no longer requests for unused fixture.
• Improve test_worst_storage_access_cold to add cases where it (1) completes execution and SSTOREs affect the state, (2) reverts after SSTOREs, (3) runs out of gas and also reverts SSTOREs.
• Fixes to some tests' environments and bumped the maximum environment gas limit, and now every single test can be run from a SINGLE GENESIS FILE 🎉

(ethereum-execution-spec-tests) ➜  execution-spec-tests git:(update-all-benchmark-tests) uv run groupstats ./fixtures/blockchain_tests_engine_x/pre_alloc -vv

Pre-Allocation Statistics Summary
Total groups: 1
Total tests: 3271
Total accounts: 64033

Tests and Accounts per Group
┏━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┓
┃ Group Hash  ┃ Fork   ┃ Tests ┃ Accounts ┃
┡━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━━━┩
│ 0x313d28... │ Prague │  3271 │    64033 │
└─────────────┴────────┴───────┴──────────┘

No need to restart clients anymore to perform every single benchmark we have 😉

cc @jsign @LouisTsai-Csie

[jsign]

Very nice change! Left some opinions for your consideration.

Lateral question: is there a way to compare the generated fixtures of this branch against master to see if there are only changes in the expected tests that had bugs/improvements? (e.g., not sure if there's some produced "hash" of each fixture output that could be compared to double check).

I'm a paranoid person, so for such a big refactoring I'm always fearing something falling through the cracks.

If it sounds like a rabbit hole, dismiss.

[marioevz]

Very nice change! Left some opinions for your consideration.

Lateral question: is there a way to compare the generated fixtures of this branch against master to see if there are only changes in the expected tests that had bugs/improvements? (e.g., not sure if there's some produced "hash" of each fixture output that could be compared to double check).

I'm a paranoid person, so for such a big refactoring I'm always fearing something falling through the cracks.

If it sounds like a rabbit hole, dismiss.

I think I've got your solution with this evmone new feature: ipsilon/evmone#1274

I could give it a try to (1) test it out for pawel, (2) verify that the tests produce similar opcode numbers before and after this PR using the same gas limits.

[jsign]

Very nice change! Left some opinions for your consideration.
Lateral question: is there a way to compare the generated fixtures of this branch against master to see if there are only changes in the expected tests that had bugs/improvements? (e.g., not sure if there's some produced "hash" of each fixture output that could be compared to double check).
I'm a paranoid person, so for such a big refactoring I'm always fearing something falling through the cracks.
If it sounds like a rabbit hole, dismiss.

I think I've got your solution with this evmone new feature: ipsilon/evmone#1274

I could give it a try to (1) test it out for pawel, (2) verify that the tests produce similar opcode numbers before and after this PR using the same gas limits.

Makes sense, since actually most "compute" test might not have a visible effect in block header fields such as state root and similar (used gas is a red herring) -- so the opcode count actually sounds better to achieve that. For the "stateful" ones, checking the block header might be enough.

[marioevz]

Generated the fixtures from master and from this PR with the opcode counts produced by ipsilon/evmone#1274:

benchmark-fixtures-with-opcode-count.tar.gz

Each fixture in every fixture file now contains a opcode_count element in the _info object, and it contains the number of times every opcode was executed during the test generation.

I have not made the comparison because the name changes make it tricky, but will do it next week.

[marioevz]

I gpt'd a script to get the differences (which could definitely be included in the PR to support opcode counts!) and got the results:

https://gist.github.com/marioevz/b745761e0882549c59ab9195b6865386

[jsign]

I gpt'd a script to get the differences (which could definitely be included in the PR to support opcode counts!) and got the results:

https://gist.github.com/marioevz/b745761e0882549c59ab9195b6865386

Looks awesome!

[marioevz]

Merging since I believe @LouisTsai-Csie comments were addressed 👍