feat(fill, execute): track execution & setup testing phase

[LouisTsai-Csie]

🗒️ Description

Background for reviewers:

In some benchmark tests, there are not only benchmark transactions but also setup transactions that create the scenario for the benchmark target. For example, in the SSTORE benchmark we first need a setup transaction that deploys as many contracts as possible, and then a separate transaction to perform the storage updates. This feature helps gas-limit testing distinguish between the setup and execution phases.

This PR extends transaction metadata, which labeling phases such as setup, testing, and cleanup. Currently, operations like deploy_contract or fund_eoa are tagged as setup phase, while transactions/blocks included in state_test or blockchain_test are tagged as testing. However, the current labeling is not always precise enough.

Take test_worst_blockhash as an example. The first 256 blocks should be classified as setup, while only the final block should count as the testing phase. Without this distinction, we might mistakenly include the setup blocks in benchmark accounting.

def test_worst_blockhash(
    blockchain_test: BlockchainTestFiller,
    pre: Alloc,
    gas_benchmark_value: int,
):
    """Test running a block with as many blockhash accesses to the oldest allowed block as possible."""
    # Create 256 dummy blocks to fill the blockhash window.
    blocks = [Block()] * 256

    # Always ask for the oldest allowed BLOCKHASH block.
    execution_code = Op.PUSH1(1) + While(
        body=Op.POP(Op.BLOCKHASH(Op.DUP1)),
    )
    execution_code_address = pre.deploy_contract(code=execution_code)
    op_tx = Transaction(
        to=execution_code_address,
        gas_limit=gas_benchmark_value,
        sender=pre.fund_eoa(),
    )
    blocks.append(Block(txs=[op_tx]))

    blockchain_test(
        pre=pre,
        post={},
        blocks=blocks,
    )

The design introduces a test_phase attribute at both the block and transaction level (currently supporting execution and setup phases). This attribute is used during execution (see transaction_post.py for details), and the metadata is updated accordingly.

🔗 Related Issues or PRs

This is the follow-up PR for #1945, more description could be found in issue ethereum/execution-specs#1588.
Related discussion: 1, 2

✅ Checklist

• All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:

  uvx --with=tox-uv tox -e lint,typecheck,spellcheck,markdownlint

• All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
• All: Considered adding an entry to CHANGELOG.md.
• All: Considered updating the online docs in the ./docs/ directory.
• All: Set appropriate labels for the changes (only maintainers can apply labels).
• Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
• Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.
• Ported Tests: All converted JSON/YML tests from ethereum/tests or tests/static have been assigned @ported_from marker.

[LouisTsai-Csie]

Note: I have not yet tested against execute command. I will provide some result later

[fselmo]

Hey @LouisTsai-Csie. I don't have much to add on the implementation. This looks great. I'm going to test this early next week but just had some minor observations at first pass and left as comments.

I wanted to just get a bit of clarity on the scope on how the metadata of each phase will be used. I think we're capturing the data in a good way and I will continue my review next week, but I'd like to understand all the ways we currently plan to use this kind of metadata to get a better understanding for the design.

I think we mentioned being able to call execute on a network with a setup phase first, and later call it with the execution phase. Is there any other case I'm missing so I can understand the end goal on capturing this split? Thanks!

cc: @marioevz

[LouisTsai-Csie]

Yes, we mentioned in bloatnet / stateful testing scenario, we want to only run the setup phase only, which will configure all the state to prepare for the later benchmarking purpose, as discussed in issue ethereum/execution-specs#1586.

I update the metadata also in the PR based on the following reason, you could see a shorter version in PR description, please let me know if this make sense or not:

The current architecture split the benchmark cases into several distinct phases:

• Setup Phase: This phase includes operations such as deploying contracts via deploy_contract, funding accounts through fund_eoa, and initial state related setup tasks. These actions are part of the initial env preparation and should not be considered part of the actual benchmarking target.
• Testing Phase: This is the main benchmarking phase, designed to measure the worst-case performance for specific opcodes/precompiles/scenarios. It is what we include in the benchmark tests and typically corresponds to the last block in a benchmark test.
• Cleanup Phase: After the execute mode completes, tokens are refunded from the EOA to the sender account.

However, there is an issue with the current design. The problem lies within the Testing Phase, currently, all transactions and blocks passed from the BlockchainTest or StateTest are treated as part of the benchmark target:

for block in self.blocks:  # Benchmark target currently includes every block passed from BlockchainTest/StateTest
    signed_txs = []
    for tx_index, tx in enumerate(block):
        # Add metadata
        tx = tx.with_signature_and_sender()
        to_address = tx.to
        label = to_address.label if isinstance(to_address, Address) else None
        tx.metadata = TransactionTestMetadata(
            test_id=request.node.nodeid,
            phase="testing",  # Marking the benchmark phase
            target=label,
            tx_index=tx_index,
        )
        signed_txs.append(tx)
        ...

This approach becomes problematic because even within a benchmark test, some blocks are still part of the setup process.
Take the test_worst_blockhash example below:

def test_worst_blockhash(
    blockchain_test: BlockchainTestFiller,
    pre: Alloc,
    gas_benchmark_value: int,
):
    """
    Test running a block with as many blockhash accesses to the oldest allowed block as possible.
    """
    blocks = [Block()] * 256

    execution_code = Op.PUSH1(1) + While(
        body=Op.POP(Op.BLOCKHASH(Op.DUP1)),
    )
    execution_code_address = pre.deploy_contract(code=execution_code)
    op_tx = Transaction(
        to=execution_code_address,
        gas_limit=gas_benchmark_value,
        sender=pre.fund_eoa(),
    )
    blocks.append(Block(txs=[op_tx]))

    blockchain_test(
        pre=pre,
        post={},
        blocks=blocks,
    )

In this test, 257 blocks are passed to execute, but only the last block is the actual benchmarking target.

Therefore, we need to further subdivide the Testing Phase into smaller units:

• The earlier part of the Testing Phase is still performing setup operations (which should be labeled as Setup).
• The actual benchmark measurement should be explicitly labeled as the Execution Phase.

Currently, we don’t separate different phases. Instead, we simply inform the Nethermind team and others to “measure only the last block for benchmarking.” This requires the test implementer and these teams to be aware of that restriction.

In test_worst_bytecode_single_opcode, there are two transactions: contracts_deployment_tx(setup phase) and opcode_tx(execution phase). These should not be included in the same block, as doing so would pollute the benchmark results.

With the introduction of the phase manager, engineers will be able to explicitly define distinct phases within the benchmark process, improving clarity and control over what is actually measured. In the current design, even if two tx with different phase is inside a single block, we could still classify each one into their corresponding phase.

[LouisTsai-Csie]

I am not sure if my understanding and implementation is correct. Please let me know if there is anything is unclear and needs further revision or discussion!

[fselmo]

With the introduction of the phase manager, engineers will be able to explicitly define distinct phases within the benchmark process, improving clarity and control over what is actually measured. In the current design, even if two tx with different phase is inside a single block, we could still classify each one into their corresponding phase.

Apologies for not being more clear. I completely understand the need for splitting these contexts up. It's clear from this PR that the separation of concerns is working well. I wanted a bit more context on how it will be used. Defining and splitting them up is one thing, but then using this in practice is another.

I think the issue you linked has some good information for me here:

• We could use a two-phase approach (similar to how we fill engine-x tests) where the first phase runs all benchmark tests but in setup mode so it only deploys the contracts, and then the second phase actually runs the tests.
• The setup phase could even be deterministic so we check all whether the contracts are already there before deploying any of them. This could be achieved by having a setup-seed so we can run the workload phase many times over and over with a single setup and just passing a --setup-seed flag or similar.

I was wondering the design here. Would we be able to call something like uv run execute --phase=setup + uv run execute --phase=execution or internally do we always run both or check the testing phase and if it is already set up we go straight to execution, etc? This is definitely out of scope for this PR but I was trying to get some clarity on the execution design so that I can be better informed here.

I think this looks good from my end and is ready to go imo 👍🏼

[fselmo]

lgtm! @marioevz I'll let you merge this in case there is something from the original issue we should cover here but the logic for phase-splitting is sound here imo 👍🏼

[LouisTsai-Csie]

Apologies for not being more clear. I completely understand the need for splitting these contexts up. It's clear from this PR that the separation of concerns is working well. I wanted a bit more context on how it will be used. Defining and splitting them up is one thing, but then using this in practice is another.

No worries! i found i understand the initial question wrong. And yes i think this could be the future goal, but might need a dedicate PR for this feature!

uv run execute --phase=setup + uv run execute --phase=execution

Thanks for your review and suggestion!

[marioevz]

I think this PR needs more iterations because I would lean towards not introduce breaking changes to ethereum_test_specs/blockchain at this point.

We can also defer this to be post weld, it's not critical to have it before we move.

[jochem-brouwer]

Hiya, could you show how this test_worst_blockhash test would be written using this new metadata? I'd assume we have to decorate the blocks with a marker? It would be great to see how this would look when writing the tests 😄 👍

(If this is already somewhere in this PR comments, please link me then I have "read over" that part. Or a link to the changed code is also fine! 😄 👍 )

[fselmo]

Hiya, could you show how this test_worst_blockhash test would be written using this new metadata? I'd assume we have to decorate the blocks with a marker? It would be great to see how this would look when writing the tests 😄 👍

(If this is already somewhere in this PR comments, please link me then I have "read over" that part. Or a link to the changed code is also fine! 😄 👍 )

Does this help clarify how it would work in its current state? It seems that this is still up in the air though, so we may still be iterating on this.

[LouisTsai-Csie]

Thanks everyone for the review and comment, i pushed latest changes now. If you think we need more iterations after review, please feel free to close it for now. I could do it after the Weld! cc @marioevz and @fselmo

[marioevz]

LGTM, thanks for making the changes!

I left just a few comments, and the branch needs a rebase, but we should be ok to merge after that.