feat(tests): add benchmark for the worst initcode jumpdest analysis #1646

chfast · 2025-05-23T14:20:12Z

🗒️ Description

Add a benchmark where we place a very long initcode in the memory and then invoke CREATE instructions with this initcode up to the block gas limit. The initcode itself have minimal execution time but forces the EVM to perform the full jumpdest analysis on the parametrized byte pattern. The initicode is slightly modified between CREATE invocations to prevent caching.

🔗 Related Issues

✅ Checklist

All: Set appropriate labels for the changes.
All: Considered squashing commits to improve commit history.
All: Added an entry to CHANGELOG.md.
All: Considered updating the online docs in the ./docs/ directory.
Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.

chfast · 2025-05-23T14:49:07Z

Benchmarking EIP-7907: Meter Contract Code Size And Increase Limit

This test can be used to benchmark EIP-7907. The implementation must allow initcodes longer than the Prague limit.

Generated state test: worst_initcode_jumpdest_analysis.json

Results

evmone

It looks like increasing the limit causes the test to be executed ~6x longer, but this is still ~192Mgas/s in the worst case.
Main reason should be the change of the CREATE cost - jumpdest analysis cost ratio. For the current limit of 48k the initcode jumpdest analysis cost is only ~3k, for the 512k max size the cost is 33k.

Osaka/0/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-00-initcode_size_49152]/1: 35 ms
Osaka/1/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-00-initcode_size_524288]/1: 197 ms
Osaka/2/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-5b-initcode_size_49152]/1: 127 ms
Osaka/3/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-5b-initcode_size_524288]/1: 740 ms
Osaka/4/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-605b-initcode_size_49152]/1: 66 ms
Osaka/5/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-605b-initcode_size_524288]/1: 374 ms
Osaka/6/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-605b5b-initcode_size_49152]/1: 53 ms
Osaka/7/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-605b5b-initcode_size_524288]/1: 293 ms
Osaka/8/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-615b5b-initcode_size_49152]/1: 46 ms
Osaka/9/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-615b5b-initcode_size_524288]/1: 260 ms
Osaka/10/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-615b5b5b-initcode_size_49152]/1: 38 ms
Osaka/11/tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-blockchain_test_from_state_test-615b5b5b-initcode_size_524288]/1: 214 ms

Geth

[
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-605b5b-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 102958028,
      "allocs": 110530,
      "bytesAllocated": 123316072,
      "gasUsed": 72000000
    }
  },
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-615b5b-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 97807776,
      "allocs": 110570,
      "bytesAllocated": 123322833,
      "gasUsed": 72000000
    }
  },
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-615b5b5b-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 89937716,
      "allocs": 110541,
      "bytesAllocated": 123318193,
      "gasUsed": 72000000
    }
  },
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-00-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 71727318,
      "allocs": 110613,
      "bytesAllocated": 123327866,
      "gasUsed": 72000000
    }
  },
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-5b-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 69766863,
      "allocs": 110608,
      "bytesAllocated": 123328256,
      "gasUsed": 72000000
    }
  },
  {
    "name": "tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Osaka-state_test-605b-initcode_size_49152]",
    "pass": true,
    "stateRoot": "0x6ce6519d6ba9af1d21bfff604bdd25e2912a5f79a5cd7ee35abc7db1a1c76b31",
    "fork": "Osaka",
    "benchStats": {
      "time": 102013648,
      "allocs": 110550,
      "bytesAllocated": 123313963,
      "gasUsed": 72000000
    }
  }
]

marioevz · 2025-05-25T20:16:58Z

tests/zkevm/test_worst_bytecode.py

    Transaction,
    While,
    compute_create2_address,
 )
 from ethereum_test_tools.vm.opcode import Opcodes as Op
+from pytest_plugins.execute.pre_alloc import MAX_BYTECODE_SIZE, MAX_INITCODE_SIZE


I think we need to put this in ethereum_test_forks as a property of the fork so we can adapt the test automatically.

I put this in #1649 😄 👍

Yes, make sense to me.

jochem-brouwer · 2025-05-25T20:19:42Z

@chfast if there is a 6x performance degradation does this mean the initcode pricing from 3860 is too cheap? One would assume this extra initcode analysis is paid for? (2 gas per 32 bytes is thus too cheap for initcode analysis?)

jsign

Apart from the #1649 discussion, LGTM!

Cycles:

tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-615b5b-initcode_size_49152]-1        255986171
tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-605b-initcode_size_49152]-1  330872313
tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-615b5b5b-initcode_size_49152]-1      381150402
tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-00-initcode_size_49152]-1    406269727
tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-605b5b-initcode_size_49152]-1        473544495
tests/zkevm/test_worst_bytecode.py::test_worst_initcode_jumpdest_analysis[fork_Cancun-blockchain_test_from_state_test-5b-initcode_size_49152]-1    758025210

jsign · 2025-05-25T23:08:03Z

tests/zkevm/test_worst_bytecode.py

+        Op.PUSH1[len(initcode_prefix)]
+        + Op.MSTORE


Clever use of the returned address!

Yes this is so well disguised that I initially thought it did not edit the initcode 😄 👍 Love this!

chfast · 2025-05-26T07:13:09Z

tests/zkevm/test_worst_bytecode.py

+        sender=pre.fund_eoa(),
+    )
+
+    state_test(


@jsign Note that I downgraded to state_test. I think we should do this for all benchmarks with single transaction. EEST will generate a blockchain test for it anyway.

Good point!

jochem-brouwer · 2025-05-26T08:58:49Z

tests/zkevm/test_worst_bytecode.py

@@ -204,3 +204,100 @@ def test_worst_bytecode_single_opcode(
        ],
        exclude_full_post_state_in_output=True,
    )
+
+
+@pytest.mark.valid_from("Cancun")


Suggested change

@pytest.mark.valid_from("Cancun")

@pytest.mark.valid_from("Paris")

Should this be Paris to also include pre-EIP-3860 forks? Test also seems to imply this by swapping out PUSH0 for PUSH1 pre-Shanghai

This was the original idea, but because in Paris there is also no metering for initcode the benchmark behaves very differently. We rather need the Osaka implementation to test it on.

As a lateral comment, targeting the latest fork should be the primary goal. While many of these tests could be generalized further to cover previous forks, I would prefer not to overcomplexify them since we aren't interested in benchmarking proof generations for older forks. If the generalization to cover older forks doesn't make the test more complex, I guess it's fine. Just mentioning this for some extra context.

cc @kevaundray

chfast force-pushed the bench/initcode_jumpdest_analysis branch 2 times, most recently from 6b017c7 to f9b75b4 Compare May 23, 2025 14:37

jochem-brouwer mentioned this pull request May 25, 2025

feat(tests): Update benchmark test to query newly introduced max_code_size() from fork config #1649

Draft

9 tasks

feat(tests): add benchmark for the worst initcode jumpdest analysis

5f1d2b2

chfast force-pushed the bench/initcode_jumpdest_analysis branch from f9b75b4 to 5f1d2b2 Compare May 25, 2025 19:45

chfast added scope:tests Scope: Changes EL client test cases in `./tests` type:feat type: Feature feature:zkevm labels May 25, 2025

marioevz requested changes May 25, 2025

View reviewed changes

jsign reviewed May 25, 2025

View reviewed changes

chfast commented May 26, 2025

View reviewed changes

jochem-brouwer reviewed May 26, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(tests): add benchmark for the worst initcode jumpdest analysis #1646

feat(tests): add benchmark for the worst initcode jumpdest analysis #1646

Uh oh!

chfast commented May 23, 2025 •

edited

Loading

Uh oh!

chfast commented May 23, 2025 •

edited

Loading

Uh oh!

marioevz May 25, 2025

Uh oh!

jochem-brouwer May 25, 2025

Uh oh!

chfast May 25, 2025

Uh oh!

jochem-brouwer commented May 25, 2025 •

edited

Loading

Uh oh!

jsign left a comment •

edited

Loading

Uh oh!

jsign May 25, 2025

Uh oh!

jochem-brouwer May 26, 2025 •

edited

Loading

Uh oh!

chfast May 26, 2025

Uh oh!

jsign May 26, 2025

Uh oh!

jochem-brouwer May 26, 2025

Uh oh!

chfast May 26, 2025

Uh oh!

jsign May 26, 2025 •

edited

Loading

Uh oh!

Uh oh!

	@pytest.mark.valid_from("Cancun")
	@pytest.mark.valid_from("Paris")

feat(tests): add benchmark for the worst initcode jumpdest analysis #1646

Are you sure you want to change the base?

feat(tests): add benchmark for the worst initcode jumpdest analysis #1646

Uh oh!

Conversation

chfast commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🗒️ Description

🔗 Related Issues

✅ Checklist

Uh oh!

chfast commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarking EIP-7907: Meter Contract Code Size And Increase Limit

Results

evmone

Geth

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jochem-brouwer commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsign left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jochem-brouwer May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsign May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chfast commented May 23, 2025 •

edited

Loading

chfast commented May 23, 2025 •

edited

Loading

jochem-brouwer commented May 25, 2025 •

edited

Loading

jsign left a comment •

edited

Loading

jochem-brouwer May 26, 2025 •

edited

Loading

jsign May 26, 2025 •

edited

Loading