Skip to content

Commit 830e6de

Browse files
cmd/evm: parallel workers in evm statetest and blocktest (#21058)
continuation of #20315 and #21027 ## Summary Improves the `evm blocktest` and `evm statetest` CLI runners — parallel workers, JSON output, regex filtering, stdin batch mode — plus a few correctness fixes (EIP-7702 fixture parsing, pre-Prague SetCode rejection, fresh-DB per subtest, goroutine/datadir leak in `RunCLI`). End-to-end benchmarks against `fixtures_develop.tar.gz` v5.4.0 on a 16-core host with `tmpfs` (`tools/create-ramdisk`, `TMPDIR=/mnt/erigon-ramdisk/tmp`), 12 workers / `-parallel 12`: ### State tests | Run | Set | Tests | Pass | Fail | Wall | |---|---|---:|---:|---:|---:| | `evm statetest` | all `state_tests/` | 63,556 | 63,519 | 37 | **1m59s** | | `evm statetest` | `static/state_tests/` minus `stTimeConsuming` (matches `TestState`) | 25,294 | 25,285 | 9 | **47s** | | `go test -run '^TestState$'` | as configured | 25,294 | 25,294 | 0 | 50s wall (46.7s reported) | The 9/37 CLI failures are real Erigon validation gaps surfaced by the CLI's strict `checkError` (EIP-4844 blob `TYPE_3_TX_*` checks, EIP-2930 pre-fork tx-type rejection). `TestState`'s wrapper is permissive — `if err != nil && len(ExpectException) > 0 { return nil }` — so it ignores whether the expected error actually fired. ### Blockchain tests | Run | Tests | Pass | Fail | Wall | |---|---:|---:|---:|---:| | `evm blocktest --workers=12` — entire `blockchain_tests/` (no skips) | **69,256** | 69,256 | 0 | **3m34s** | | `evm blocktest --workers=12` — Go-test subset only | 17,671 | 17,671 | 0 | 1m04s | | `go test -parallel 12` — 5 `TestExecutionSpecBlockchain*` packages | 17,671 | 17,671 | 0 | 1m02s | CLI covers ~4× more blockchain-test subtests than the existing 5 Go test packages combined. The bulk of the gap is `blockchain_tests/static/state_tests/` (~40,855 subtests in blockchain-test format), which `TestExecutionSpecBlockchain` skips with the comment *"Tested in the state test format by TestState"* — but `TestState` walks `state_tests/static/state_tests/` (state-test format), a different directory with different end-to-end coverage. The remaining ~10,730 are 7 "very slow" files (BLS, blob-tx combinations, intrinsic-gas tx, stack-overflow) that no Go test currently exercises. On apples-to-apples (same 17,671 subset), CLI and `go test` are within 3% of each other — both MDBX-bound on per-subtest datadir lifecycle. --- ## Changes ### `cmd/evm/staterunner.go`, `cmd/evm/blockrunner.go`, `cmd/evm/main.go`, `cmd/evm/reporter.go` CLI runner upgrades shared by both commands: - New flags: `--workers` (parallel pool), `--jsonout` (machine-readable array of `{name, pass, stateRoot, fork, error, ...}`), `--run <regex>` (filter by test key). - Both commands now accept a directory (recursive walk via `collectFiles`) or stdin batch mode (newline-separated filenames, one-by-one). - Worker pool uses an indexed channel + ordered result slice so JSON output stays deterministic across runs regardless of completion order. - `report` writes JSON via streaming `json.Encoder` to stdout (no intermediate `MarshalIndent` allocation) and uses a buffered writer for the human-readable path. - `testResult` carries `Fork` and always includes the `error` field (empty string when passing) so JSON output is shape-stable. - `runStateTest` / `runBlockTest` propagate JSON-unmarshal errors instead of silently skipping non-fixture files. ### `cmd/evm/staterunner.go` — fresh DB per subtest Previously the runner created one `temporaltest.NewTestDB` for the whole batch and reused the same write tx across subtests. State from a failing test (or even a successful one with side effects) leaked into the next subtest's pre-state. Now each subtest gets its own `os.MkdirTemp` + datadir + `temporaltest.NewTestDB` + tx, all torn down before moving on. With `--workers=N` this is also the only way to safely parallelize, since each goroutine needs its own MDBX env. Infrastructure errors during setup (`MkdirTemp`, `BeginTemporalRw`) mark that subtest failed and continue with the next — they don't abort the whole batch. ### `execution/tests/testutil/state_test_util.go` — EIP-7702 fixture parsing EEST emits authorization lists with raw fields like `"chainId": "0x00"` (leading-zero hex), which `hexutil.Big`'s strict parser rejects. New `stAuthorization` mirror struct uses `math.HexOrDecimal256` and converts to `types.Authorization` via `ToAuthorization()`. The empty list `"authorizationList": []` is semantically meaningful — it marks the tx as type-4 SetCode (changes intrinsic gas) even with zero entries. A custom `UnmarshalJSON` peeks at the raw JSON to set `IsSetCodeTx = true` whenever the key is present, so callers can distinguish "no `authorizationList` key" (legacy/regular tx) from "empty `authorizationList`" (SetCode tx with no auths). `Run()` gains a `checkError` helper modeled on geth's: distinguishes - err==nil + no expected → pass - err==nil + expected → "expected error X, got no error" - err!=nil + no expected → "unexpected error: X" - err!=nil + expected → pass When an error was expected, post-state root is only re-checked if `post.Root` is explicitly set (non-zero hash). `RunNoVerify` now adds a zero-balance touch on the coinbase even for failing/reverted txs (matches geth's `state_test_util.go`) and propagates the `ApplyMessage` error through to the caller (was previously silenced by the trailing `nil` return). ### `execution/protocol/txn_executor.go` — SetCode pre-check `verifyAuthorities` now distinguishes `auths == nil` (not a SetCode tx) from `len(auths) == 0` (empty list, still type-4). For non-nil auths it asserts: - chain rules are at least Prague (otherwise `"SetCode transaction not allowed before Prague fork"`), - not a contract creation (existing check, unchanged), - list is non-empty (`"SetCode transaction must have at least one authorization"`). This pairs with the parsing change above: fixtures using `"authorizationList": []` to test the empty-list invalid case now drive a real rejection error, instead of silently being treated as legacy txs. ### `execution/execmodule/execmoduletester/exec_module_tester.go` + `execution/tests/testutil/block_test_util.go` — RunCLI leak fix `BlockTest.RunCLI()` previously did `defer m.DB.Close()` only, but `execmoduletester.New` spawns a background `errgroup` plus an Engine, BlockSnapshots, and a temp datadir. Across 17k+ blocktest subtests with 12 workers the result was leaked goroutines (CPU at 100% across all cores), 26k+ leftover `mock-sentry-*` directories under `TMPDIR`, and the host lagging. Fix: - `ExecModuleTester.Close()` now skips the `require.Equal(emt.tb, ...)` assertion when `tb == nil` (CLI mode panicked otherwise) and removes the temp datadir at the end (the previous code relied on `tb.Cleanup`, which doesn't fire in CLI mode). - `BlockTest.RunCLI()` switches to `defer m.Close()`. After the fix, the 69,256-test full sweep finishes in 3m34s with 0 leftover datadirs. --------- Co-authored-by: spencer-tb <spencer.tb@ethereum.org>
1 parent 97ff85c commit 830e6de

8 files changed

Lines changed: 389 additions & 125 deletions

File tree

cmd/evm/blockrunner.go

Lines changed: 102 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -20,14 +20,15 @@
2020
package main
2121

2222
import (
23+
"bufio"
2324
"encoding/json"
24-
"errors"
2525
"fmt"
2626
"maps"
2727
"os"
2828
"path/filepath"
2929
"regexp"
3030
"slices"
31+
"sync"
3132

3233
"github.com/urfave/cli/v2"
3334

@@ -38,19 +39,19 @@ import (
3839
var blockTestCommand = cli.Command{
3940
Action: blockTestCmd,
4041
Name: "blocktest",
41-
Usage: "Executes the given blockchain tests",
42+
Usage: "Executes the given blockchain tests. Filenames can be fed via standard input (batch mode) or as an argument (one-off execution).",
4243
ArgsUsage: "<path>",
4344
Flags: []cli.Flag{
4445
&DumpFlag,
46+
&JSONOutputFlag,
47+
&RunFlag,
4548
&VerbosityFlag,
49+
&WorkersFlag,
4650
},
4751
}
4852

4953
func blockTestCmd(ctx *cli.Context) error {
5054
path := ctx.Args().First()
51-
if len(path) == 0 {
52-
return errors.New("path argument required")
53-
}
5455

5556
// Set up logging
5657
if ctx.Int(VerbosityFlag.Name) > 0 {
@@ -59,36 +60,118 @@ func blockTestCmd(ctx *cli.Context) error {
5960
log.Root().SetHandler(log.LvlFilterHandler(log.LvlError, log.StderrHandler))
6061
}
6162

62-
var (
63-
collected = collectFiles(path)
64-
results []testResult
65-
)
66-
67-
for _, fname := range collected {
68-
r, err := runBlockTest(ctx, fname)
63+
if len(path) != 0 {
64+
collected := collectFiles(path)
65+
workers := ctx.Int(WorkersFlag.Name)
66+
if workers <= 0 {
67+
workers = 1
68+
}
69+
results, err := runBlockTestsParallel(ctx, collected, workers)
6970
if err != nil {
7071
return err
7172
}
72-
results = append(results, r...)
73+
report(ctx, results)
74+
return nil
75+
}
76+
// Otherwise, read filenames from stdin and execute back-to-back.
77+
scanner := bufio.NewScanner(os.Stdin)
78+
for scanner.Scan() {
79+
fname := scanner.Text()
80+
if len(fname) == 0 {
81+
return nil
82+
}
83+
results, err := runBlockTest(ctx, fname)
84+
if err != nil {
85+
return err
86+
}
87+
report(ctx, results)
7388
}
74-
75-
report(ctx, results)
7689
return nil
7790
}
7891

92+
// fileResult holds the results from processing a single fixture file.
93+
type fileResult struct {
94+
index int
95+
results []testResult
96+
err error
97+
}
98+
99+
func runBlockTestsParallel(ctx *cli.Context, files []string, workers int) ([]testResult, error) {
100+
if workers == 1 {
101+
results := make([]testResult, 0, len(files)*4) // pre-allocate: most files have a few tests
102+
for _, fname := range files {
103+
r, err := runBlockTest(ctx, fname)
104+
if err != nil {
105+
return nil, err
106+
}
107+
results = append(results, r...)
108+
}
109+
return results, nil
110+
}
111+
var (
112+
wg sync.WaitGroup
113+
fileCh = make(chan struct {
114+
index int
115+
fname string
116+
}, len(files))
117+
resultCh = make(chan fileResult, len(files))
118+
)
119+
for i, fname := range files {
120+
fileCh <- struct {
121+
index int
122+
fname string
123+
}{i, fname}
124+
}
125+
close(fileCh)
126+
127+
for w := 0; w < workers; w++ {
128+
wg.Add(1)
129+
go func() {
130+
defer wg.Done()
131+
for item := range fileCh {
132+
r, err := runBlockTest(ctx, item.fname)
133+
resultCh <- fileResult{index: item.index, results: r, err: err}
134+
}
135+
}()
136+
}
137+
go func() {
138+
wg.Wait()
139+
close(resultCh)
140+
}()
141+
142+
ordered := make([]fileResult, len(files))
143+
for fr := range resultCh {
144+
if fr.err != nil {
145+
return nil, fr.err
146+
}
147+
ordered[fr.index] = fr
148+
}
149+
// Pre-estimate total results
150+
total := 0
151+
for _, fr := range ordered {
152+
total += len(fr.results)
153+
}
154+
results := make([]testResult, 0, total)
155+
for _, fr := range ordered {
156+
results = append(results, fr.results...)
157+
}
158+
return results, nil
159+
}
160+
79161
// collectFiles walks the given path and returns all JSON files.
80162
// If path is a file, it returns that file directly.
81163
func collectFiles(path string) []string {
82-
var out []string
83164
info, err := os.Stat(path)
84165
if err != nil {
85-
return out
166+
return nil
86167
}
87168

88169
if !info.IsDir() {
89170
return []string{path}
90171
}
91172

173+
// Pre-allocate with a reasonable estimate to avoid repeated slice growth
174+
out := make([]string, 0, 256)
92175
err = filepath.WalkDir(path, func(path string, d os.DirEntry, err error) error {
93176
if err != nil {
94177
return err
@@ -116,9 +199,9 @@ func runBlockTest(ctx *cli.Context, fname string) ([]testResult, error) {
116199
return nil, err
117200
}
118201

119-
re, err := regexp.Compile(".*") // Run all tests by default
202+
re, err := regexp.Compile(ctx.String(RunFlag.Name))
120203
if err != nil {
121-
return nil, fmt.Errorf("invalid regex: %v", err)
204+
return nil, fmt.Errorf("invalid regex -%s: %v", RunFlag.Name, err)
122205
}
123206

124207
// Pull out keys to sort and ensure tests are run in order

cmd/evm/main.go

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,11 @@ var (
139139
WorkersFlag = cli.IntFlag{
140140
Name: "workers",
141141
Value: 0,
142-
Usage: "Number of workers to execute tests in parallel (0 means use the command's default)",
142+
Usage: "Number of workers to execute tests in parallel (0 means use the command's defaul)",
143+
}
144+
JSONOutputFlag = cli.BoolFlag{
145+
Name: "jsonout",
146+
Usage: "Output results as JSON array instead of human-readable format",
143147
}
144148
)
145149

cmd/evm/reporter.go

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,10 @@
2020
package main
2121

2222
import (
23+
"bufio"
2324
"encoding/json"
2425
"fmt"
26+
"os"
2527

2628
"github.com/urfave/cli/v2"
2729

@@ -40,8 +42,10 @@ type testResult struct {
4042
Name string `json:"name"`
4143
Pass bool `json:"pass"`
4244
Root *common.Hash `json:"stateRoot,omitempty"`
43-
Error string `json:"error,omitempty"`
45+
Fork string `json:"fork,omitempty"`
46+
Error string `json:"error"`
4447
State *state.Dump `json:"state,omitempty"`
48+
Stats *execStats `json:"benchStats,omitempty"`
4549
}
4650

4751
func (r testResult) String() string {
@@ -55,6 +59,9 @@ func (r testResult) String() string {
5559
var extra string
5660
if !r.Pass {
5761
extra = fmt.Sprintf(", err=%v", r.Error)
62+
if r.Fork != "" {
63+
extra += fmt.Sprintf(", fork=%s", r.Fork)
64+
}
5865
}
5966

6067
out := fmt.Sprintf("%s %s%s", status, r.Name, extra)
@@ -67,17 +74,27 @@ func (r testResult) String() string {
6774

6875
// report prints the after-test summary.
6976
func report(ctx *cli.Context, results []testResult) {
77+
if ctx.Bool(JSONOutputFlag.Name) {
78+
// Write directly to stdout via encoder to avoid the intermediate
79+
// MarshalIndent -> string -> Println allocation chain.
80+
enc := json.NewEncoder(os.Stdout)
81+
enc.SetIndent("", " ")
82+
enc.Encode(results) //nolint:errcheck
83+
return
84+
}
7085
pass := 0
7186
for _, r := range results {
7287
if r.Pass {
7388
pass++
7489
}
7590
}
7691

92+
// Use buffered writer to reduce syscalls when printing many results
93+
w := bufio.NewWriter(os.Stdout)
7794
for _, r := range results {
78-
fmt.Println(r)
95+
fmt.Fprintln(w, r)
7996
}
80-
81-
fmt.Println("--")
82-
fmt.Printf("%d tests passed, %d tests failed.\n", pass, len(results)-pass)
97+
fmt.Fprintln(w, "--")
98+
fmt.Fprintf(w, "%d tests passed, %d tests failed.\n", pass, len(results)-pass)
99+
w.Flush()
83100
}

0 commit comments

Comments
 (0)