Commit 830e6de
cmd/evm: parallel workers in evm statetest and blocktest (#21058)
continuation of #20315 and
#21027
## Summary
Improves the `evm blocktest` and `evm statetest` CLI runners — parallel
workers, JSON output, regex filtering, stdin batch mode — plus a few
correctness fixes (EIP-7702 fixture parsing, pre-Prague SetCode
rejection, fresh-DB per subtest, goroutine/datadir leak in `RunCLI`).
End-to-end benchmarks against `fixtures_develop.tar.gz` v5.4.0 on a
16-core host with `tmpfs` (`tools/create-ramdisk`,
`TMPDIR=/mnt/erigon-ramdisk/tmp`), 12 workers / `-parallel 12`:
### State tests
| Run | Set | Tests | Pass | Fail | Wall |
|---|---|---:|---:|---:|---:|
| `evm statetest` | all `state_tests/` | 63,556 | 63,519 | 37 |
**1m59s** |
| `evm statetest` | `static/state_tests/` minus `stTimeConsuming`
(matches `TestState`) | 25,294 | 25,285 | 9 | **47s** |
| `go test -run '^TestState$'` | as configured | 25,294 | 25,294 | 0 |
50s wall (46.7s reported) |
The 9/37 CLI failures are real Erigon validation gaps surfaced by the
CLI's strict `checkError` (EIP-4844 blob `TYPE_3_TX_*` checks, EIP-2930
pre-fork tx-type rejection). `TestState`'s wrapper is permissive — `if
err != nil && len(ExpectException) > 0 { return nil }` — so it ignores
whether the expected error actually fired.
### Blockchain tests
| Run | Tests | Pass | Fail | Wall |
|---|---:|---:|---:|---:|
| `evm blocktest --workers=12` — entire `blockchain_tests/` (no skips) |
**69,256** | 69,256 | 0 | **3m34s** |
| `evm blocktest --workers=12` — Go-test subset only | 17,671 | 17,671 |
0 | 1m04s |
| `go test -parallel 12` — 5 `TestExecutionSpecBlockchain*` packages |
17,671 | 17,671 | 0 | 1m02s |
CLI covers ~4× more blockchain-test subtests than the existing 5 Go test
packages combined. The bulk of the gap is
`blockchain_tests/static/state_tests/` (~40,855 subtests in
blockchain-test format), which `TestExecutionSpecBlockchain` skips with
the comment *"Tested in the state test format by TestState"* — but
`TestState` walks `state_tests/static/state_tests/` (state-test format),
a different directory with different end-to-end coverage. The remaining
~10,730 are 7 "very slow" files (BLS, blob-tx combinations,
intrinsic-gas tx, stack-overflow) that no Go test currently exercises.
On apples-to-apples (same 17,671 subset), CLI and `go test` are within
3% of each other — both MDBX-bound on per-subtest datadir lifecycle.
---
## Changes
### `cmd/evm/staterunner.go`, `cmd/evm/blockrunner.go`,
`cmd/evm/main.go`, `cmd/evm/reporter.go`
CLI runner upgrades shared by both commands:
- New flags: `--workers` (parallel pool), `--jsonout` (machine-readable
array of `{name, pass, stateRoot, fork, error, ...}`), `--run <regex>`
(filter by test key).
- Both commands now accept a directory (recursive walk via
`collectFiles`) or stdin batch mode (newline-separated filenames,
one-by-one).
- Worker pool uses an indexed channel + ordered result slice so JSON
output stays deterministic across runs regardless of completion order.
- `report` writes JSON via streaming `json.Encoder` to stdout (no
intermediate `MarshalIndent` allocation) and uses a buffered writer for
the human-readable path.
- `testResult` carries `Fork` and always includes the `error` field
(empty string when passing) so JSON output is shape-stable.
- `runStateTest` / `runBlockTest` propagate JSON-unmarshal errors
instead of silently skipping non-fixture files.
### `cmd/evm/staterunner.go` — fresh DB per subtest
Previously the runner created one `temporaltest.NewTestDB` for the whole
batch and reused the same write tx across subtests. State from a failing
test (or even a successful one with side effects) leaked into the next
subtest's pre-state. Now each subtest gets its own `os.MkdirTemp` +
datadir + `temporaltest.NewTestDB` + tx, all torn down before moving on.
With `--workers=N` this is also the only way to safely parallelize,
since each goroutine needs its own MDBX env. Infrastructure errors
during setup (`MkdirTemp`, `BeginTemporalRw`) mark that subtest failed
and continue with the next — they don't abort the whole batch.
### `execution/tests/testutil/state_test_util.go` — EIP-7702 fixture
parsing
EEST emits authorization lists with raw fields like `"chainId": "0x00"`
(leading-zero hex), which `hexutil.Big`'s strict parser rejects. New
`stAuthorization` mirror struct uses `math.HexOrDecimal256` and converts
to `types.Authorization` via `ToAuthorization()`.
The empty list `"authorizationList": []` is semantically meaningful — it
marks the tx as type-4 SetCode (changes intrinsic gas) even with zero
entries. A custom `UnmarshalJSON` peeks at the raw JSON to set
`IsSetCodeTx = true` whenever the key is present, so callers can
distinguish "no `authorizationList` key" (legacy/regular tx) from "empty
`authorizationList`" (SetCode tx with no auths).
`Run()` gains a `checkError` helper modeled on geth's: distinguishes
- err==nil + no expected → pass
- err==nil + expected → "expected error X, got no error"
- err!=nil + no expected → "unexpected error: X"
- err!=nil + expected → pass
When an error was expected, post-state root is only re-checked if
`post.Root` is explicitly set (non-zero hash).
`RunNoVerify` now adds a zero-balance touch on the coinbase even for
failing/reverted txs (matches geth's `state_test_util.go`) and
propagates the `ApplyMessage` error through to the caller (was
previously silenced by the trailing `nil` return).
### `execution/protocol/txn_executor.go` — SetCode pre-check
`verifyAuthorities` now distinguishes `auths == nil` (not a SetCode tx)
from `len(auths) == 0` (empty list, still type-4). For non-nil auths it
asserts:
- chain rules are at least Prague (otherwise `"SetCode transaction not
allowed before Prague fork"`),
- not a contract creation (existing check, unchanged),
- list is non-empty (`"SetCode transaction must have at least one
authorization"`).
This pairs with the parsing change above: fixtures using
`"authorizationList": []` to test the empty-list invalid case now drive
a real rejection error, instead of silently being treated as legacy txs.
### `execution/execmodule/execmoduletester/exec_module_tester.go` +
`execution/tests/testutil/block_test_util.go` — RunCLI leak fix
`BlockTest.RunCLI()` previously did `defer m.DB.Close()` only, but
`execmoduletester.New` spawns a background `errgroup` plus an Engine,
BlockSnapshots, and a temp datadir. Across 17k+ blocktest subtests with
12 workers the result was leaked goroutines (CPU at 100% across all
cores), 26k+ leftover `mock-sentry-*` directories under `TMPDIR`, and
the host lagging.
Fix:
- `ExecModuleTester.Close()` now skips the `require.Equal(emt.tb, ...)`
assertion when `tb == nil` (CLI mode panicked otherwise) and removes the
temp datadir at the end (the previous code relied on `tb.Cleanup`, which
doesn't fire in CLI mode).
- `BlockTest.RunCLI()` switches to `defer m.Close()`.
After the fix, the 69,256-test full sweep finishes in 3m34s with 0
leftover datadirs.
---------
Co-authored-by: spencer-tb <spencer.tb@ethereum.org>1 parent 97ff85c commit 830e6de
8 files changed
Lines changed: 389 additions & 125 deletions
File tree
- cmd/evm
- execution
- execmodule/execmoduletester
- protocol
- tests/testutil
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | | - | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
41 | | - | |
| 42 | + | |
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
| 46 | + | |
| 47 | + | |
45 | 48 | | |
| 49 | + | |
46 | 50 | | |
47 | 51 | | |
48 | 52 | | |
49 | 53 | | |
50 | 54 | | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | 55 | | |
55 | 56 | | |
56 | 57 | | |
| |||
59 | 60 | | |
60 | 61 | | |
61 | 62 | | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
69 | 70 | | |
70 | 71 | | |
71 | 72 | | |
72 | | - | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
73 | 88 | | |
74 | | - | |
75 | | - | |
76 | 89 | | |
77 | 90 | | |
78 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
79 | 161 | | |
80 | 162 | | |
81 | 163 | | |
82 | | - | |
83 | 164 | | |
84 | 165 | | |
85 | | - | |
| 166 | + | |
86 | 167 | | |
87 | 168 | | |
88 | 169 | | |
89 | 170 | | |
90 | 171 | | |
91 | 172 | | |
| 173 | + | |
| 174 | + | |
92 | 175 | | |
93 | 176 | | |
94 | 177 | | |
| |||
116 | 199 | | |
117 | 200 | | |
118 | 201 | | |
119 | | - | |
| 202 | + | |
120 | 203 | | |
121 | | - | |
| 204 | + | |
122 | 205 | | |
123 | 206 | | |
124 | 207 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
139 | 139 | | |
140 | 140 | | |
141 | 141 | | |
142 | | - | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
143 | 147 | | |
144 | 148 | | |
145 | 149 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
| 26 | + | |
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
| |||
40 | 42 | | |
41 | 43 | | |
42 | 44 | | |
43 | | - | |
| 45 | + | |
| 46 | + | |
44 | 47 | | |
| 48 | + | |
45 | 49 | | |
46 | 50 | | |
47 | 51 | | |
| |||
55 | 59 | | |
56 | 60 | | |
57 | 61 | | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
58 | 65 | | |
59 | 66 | | |
60 | 67 | | |
| |||
67 | 74 | | |
68 | 75 | | |
69 | 76 | | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
70 | 85 | | |
71 | 86 | | |
72 | 87 | | |
73 | 88 | | |
74 | 89 | | |
75 | 90 | | |
76 | 91 | | |
| 92 | + | |
| 93 | + | |
77 | 94 | | |
78 | | - | |
| 95 | + | |
79 | 96 | | |
80 | | - | |
81 | | - | |
82 | | - | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
83 | 100 | | |
0 commit comments