feat(plugins): Refiller

[marioevz]

🗒️ Description

Introduces the refiller plugin which is capable of loading static test specs from static files.

The plugin uses static specs which inherit from base in src/ethereum_test_specs/base_static.py.

The inheriting classes must implement the fill_function method which returns the function that will be used by refiller to fill the static tests when the static file is loaded through the class.

This PR contains no implementation of classes that inherit from the base static test class, and these should be implemented by follow-up PRs separately.

An (non-working) example can be seen in the following file: https://github.com/ethereum/execution-spec-tests/blob/filler-reader/src/ethereum_test_specs/state_json.py

Refiller mimics the normal python test filler id convention and therefore the files it loads must always contain sub-tests.

I.e. the loaded static test files must be a JSON/YAML with an object at root level, of which every key is a sub-test.

Example of test IDs from a static test file:

tests/homestead/contractCreationOOGdontLeaveEmptyContractFiller.json::contractCreationOOGdontLeaveEmptyContract[fork_Berlin-state_test] FILLED
tests/homestead/contractCreationOOGdontLeaveEmptyContractFiller.json::contractCreationOOGdontLeaveEmptyContract[fork_Berlin-blockchain_test] FILLED
tests/homestead/contractCreationOOGdontLeaveEmptyContractFiller.json::contractCreationOOGdontLeaveEmptyContract[fork_Berlin-blockchain_test_engine] FILLED

🔗 Related Issues

✅ Checklist

• All: Set appropriate labels for the changes.
• All: Considered squashing commits to improve commit history.
• All: Added an entry to CHANGELOG.md.
• All: Considered updating the online docs in the ./docs/ directory.
• Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
• Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
• Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
• Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

[winsvega]

the function return itself?

[marioevz]

Yes, the returned function is dynamically generated based on the properties of the specific static test that was read from the file.

[winsvega]

so it can call state_test(..) for each m, n ?

[marioevz]

Yes that is correct, you can parametrize it on the fly before returning the function:

@pytest.mark.parametrize("m", [1, 2, 3])
@pytest.mark.parametrize("n", [-1])
def test_fn(...

And the parametrization can be dynamic and depend on the values that you find in the state test filler for example.

[winsvega]

can you show an example how would it work from my test filler parser returning TestVector class. (the one that has env,pre,post,tx,id) ?

[marioevz]

can you show an example how would it work from my test filler parser returning TestVector class. (the one that has env,pre,post,tx,id) ?

Here's the example to implement EOF static file readers:
b21260c

[marioevz]

Currently, all tests are failing due to:

Test doesn't define REFERENCE_SPEC_GIT_PATH and REFERENCE_SPEC_VERSION

I feel like this should be ok for static legacy tests and only be a concern when actually porting the test to python.

Wdyt @spencer-tb @danceratopz @winsvega ?