all: Trace types

docs/CHANGELOG.md

@@ -17,6 +17,8 @@ Test fixtures for use by clients are available for each release on the [Github r
 
 ### 🔧 EVM Tools
 
+- ✨ Better trace printing by selecting relevant trace lines depending on error type ([#474](https://github.com/ethereum/execution-spec-tests/pull/474)).
+
 ### 📋 Misc
 
 - 🐞 Fix CI by using Golang 1.21 in Github Actions to build geth ([#484](https://github.com/ethereum/execution-spec-tests/pull/484)).

docs/navigation.md

@@ -13,6 +13,7 @@
     * [Types of Test](writing_tests/types_of_tests.md)
     * [Adding a New Test](writing_tests/adding_a_new_test.md)
     * [Writing a New Test](writing_tests/writing_a_new_test.md)
+    * [Debugging Tests](writing_tests/debugging_tests.md)
     * [Referencing an EIP Spec Version](writing_tests/reference_specification.md)
     * [Verifying Changes Locally](writing_tests/verifying_changes.md)
     * [Exception Tests](writing_tests/exception_tests.md)

docs/writing_tests/debugging_tests.md

@@ -0,0 +1,95 @@
+# Debugging Tests
+
+The easiest way to debug tests while writing them is to run the test with the `--traces` flag. This will output the traces of the test, which can be used to understand the behavior of the test and the client.
+
+```bash
+fill 'tests/frontier/opcodes/test_dup.py::test_dup[fork_Byzantium-state_test-DUP1]' --traces
+```
+
+Depending on the test failure type, a list of relevant traces will be printed.
+
+For example, if the test fails because an expected storage key was not set (`Storage.KeyValueMismatch`), the output will show EVM traces where the storage key was set, or the executing contract returned with error.
+
+```python
+{
+    1: RelevantTraceContext(
+        execution_index=0,
+        transaction_index=0,
+        transaction_hash='0xf8616cc40f214a4729758189c77720a6b672f29dcb62cd206a49ebacaf00e96f',
+        traces=[
+            EVMTraceLine(
+                pc=34,
+                op=128,
+                op_name='DUP1',
+                gas=78949,
+                gas_cost=3,
+                mem_size=0,
+                stack=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
+                depth=1,
+                refund=0,
+                context_address='0x0000000000000000000000000000000000000100'
+            ),
+            EVMTraceLine(
+                pc=35,
+                op=96,
+                op_name='PUSH1',
+                gas=78946,
+                gas_cost=3,
+                mem_size=0,
+                stack=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 16],
+                depth=1,
+                refund=0,
+                context_address='0x0000000000000000000000000000000000000100'
+            ),
+            EVMTraceLine(
+                pc=37,
+                op=85,
+                op_name='SSTORE',
+                gas=78943,
+                gas_cost=20000,
+                mem_size=0,
+                stack=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 16, 0],
+                depth=1,
+                refund=0,
+                context_address='0x0000000000000000000000000000000000000100'
+            )
+        ],
+        description='SSTORE event on address 0x0000000000000000000000000000000000000100 and key 0x0'
+    ),
+    2: RelevantTraceContext(
+        execution_index=0,
+        transaction_index=0,
+        transaction_hash='0xf8616cc40f214a4729758189c77720a6b672f29dcb62cd206a49ebacaf00e96f',
+        traces=[
+            EVMTraceLine(
+                pc=44,
+                op=96,
+                op_name='PUSH1',
+                gas=18937,
+                gas_cost=3,
+                mem_size=0,
+                stack=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
+                depth=1,
+                refund=0,
+                context_address='0x0000000000000000000000000000000000000100'
+            ),
+            EVMTraceLine(
+                pc=46,
+                op=85,
+                op_name='SSTORE',
+                gas=18934,
+                gas_cost=20000,
+                mem_size=0,
+                stack=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 3],
+                depth=1,
+                refund=0,
+                context_address='0x0000000000000000000000000000000000000100'
+            ),
+            EVMCallFrameExit(from_address='0x0000000000000000000000000000000000000100', output=b'', gas_used=79000, error='out of gas')
+        ],
+        description='Exit frame from address 0x0000000000000000000000000000000000000100 with error'
+    )
+}
+```
+
+In this example, the test failed because the storage key `0x0` was not set. The traces show that the storage key was indeed set to `0x10`, but later the contract ran out of gas.

setup.cfg

@@ -33,6 +33,8 @@ install_requires =
     coincurve>=18.0.0,<19
     trie==2.1.1
     semver==3.0.1
+    pydantic>=2.6.3
+    rich>=13.7.0,<14
     click>=8.0.0,<9
 
 [options.package_data]
@@ -61,7 +63,7 @@ test =
 
 lint =
     isort>=5.8,<6
-    mypy==0.982; implementation_name == "cpython"
+    mypy==0.991; implementation_name == "cpython"
     types-requests
     black==22.3.0; implementation_name == "cpython"
     flake8-spellcheck>=0.24,<0.25

src/ethereum_test_tools/common/__init__.py

@@ -1,6 +1,7 @@
 """
 Common definitions and types.
 """
+
 from .base_types import (
     Address,
     Bloom,
@@ -41,6 +42,7 @@
     JSONEncoder,
     Removable,
     Storage,
+    TraceableException,
     Transaction,
     Withdrawal,
     alloc_to_accounts,
@@ -75,6 +77,7 @@
     "TestParameterGroup",
     "TestPrivateKey",
     "TestPrivateKey2",
+    "TraceableException",
     "Transaction",
     "Withdrawal",
     "ZeroPaddedHexNumber",