feature: Fill using multiple `evm` binaries and compare results

[marioevz]

In order to mitigate the lack of state test format, I propose allowing to fill all tests using multiple evm binaries and make a comparison of all results on each call.

This would allow us to indirectly test evmone by filling the test and comparing the results against geth's evm.

We can compare the state root and all accounts, as well as any other values deemed necessary.

I'd like to hear your thoughts on whether this would be useful or not @chfast @winsvega

[winsvega]

This is what I am already doing for years. The discussion we had recently was an argue that t8n is not the tool to actually test clients since its implementation doesn't mimic client behavior as closely as in hive for instance.

However I still see it as an access to evm and find it a good way to cross test different evm implementations

The t8ns I run on tests
geth
Nimbus
Ethereumjs
Evmone
Besu (too slow)
Pyt8n (too slow)

The pyspecs test I also integrate on test runner.

So as a summary: it doesn't matter which client generated the tests as long us all others agree on post state.
The current pyspecs already have support for other evms during filling.

And it makes little sense to test fill on all clients(t8n) as they might pass it during the generation but not on tommorows commit. We need a daily test run stats page to track down commits that break tests.

Here are the daily t8n runs:
http://retesteth.ethdevops.io

[chfast]

This is only partly and temporarily useful. It would be much easier if I could run this process myself so I can use it directly in development.

[danceratopz]

This is only partly and temporarily useful. It would be much easier if I could run this process myself so I can use it directly in development.

Hi @chfast, I think it was Mario's intent to generate fixtures from multiple t8n binaries within one fill execution. The comparison would be made within the execution-spec-tests framework and the test case would fail if any of the generated test fixtures fail. In which case you could absolutely run this process locally or as part of your dev tooling. You would just have the other t8n tool(s) as dependencies.

Would it be useful in that case?

[chfast]

I think this has very limited usability for me. If we are developing any EIP feature I'd need to find a matching other EVM implementation with exactly the same EIP implemented. This is probably not worth the trouble because:

• using fill with evmone only is good enough starting point,
• I will attempt to implement blockchain test format rather soon.

So this feature should be implemented only if you plan to use it.

[winsvega]

now when we have some t8n support in CI
when we run fill command on introduced new tests. we can run it using all supported t8n's (eels, geth, evmone)
to see if there are disagreements in post verification check.