feat(l1): ef_tests: add standalone statetest, blocktest, and enginetest CLI runners#6445
Draft
spencer-tb wants to merge 7 commits intolambdaclass:mainfrom
Draft
feat(l1): ef_tests: add standalone statetest, blocktest, and enginetest CLI runners#6445spencer-tb wants to merge 7 commits intolambdaclass:mainfrom
spencer-tb wants to merge 7 commits intolambdaclass:mainfrom
Conversation
Fast state test runner using LEVM directly with in-memory HashMap state. No Store, no trie, no background threads. 36,220 tests in 3.8s (w=8). 89x faster than state_v2 runner. 3 known failures (invalid CREATE blob transactions). Flags: --workers N, --run <regex>, --json, --path <dir>
Thin wrapper around ef_tests-blockchain with CLI flags. 2,776 files in 28.7s (w=8). Same results as cargo test harness. Flags: --workers N, --run <regex>, --json, --path <dir>
Engine test runner for blockchain_test_engine fixtures. Validates engine API version-specific parameters (V1-V5), converts payload to block, executes via Blockchain::add_block_pipeline(), applies fork choice, and verifies post-state. 40,521 tests in 29.5s (w=8). 0 failures on v5.3.0 stable. Flags: --workers N, --run <regex>, --json, --path <dir>
Routes payloads through the real Engine API RPC handler functions: NewPayloadV1-V5Request.parse() + .handle(context) and ForkChoiceUpdatedV1-V4.handle(context). Same code path as consume engine via Hive, minus HTTP transport. Uses shared SyncManager/PeerHandler via OnceCell to avoid thread exhaustion. Per-test RpcApiContext with isolated block_worker_channel. 40,521 tests in 32.7s (w=8). 0 failures on v5.3.0 stable.
Draft
4 tasks
spencer-tb
added a commit
to spencer-tb/execution-specs
that referenced
this pull request
Apr 8, 2026
New `validate` CLI command for running EEST fixtures directly against client EVM binaries, replacing Hive for execution correctness testing. Usage: validate health # health check all clients validate engine --client geth # engine tests validate state --client besu # state tests validate block --client nethermind # block tests Features: - 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus - Per-type Pydantic result models: StateTestResult, BlockTestResult, EngineTestResult with type-specific fields - Exception matching: maps client error strings to EEST exception types via ExceptionMapper, verifies correct exception for every invalid test (--no-exception-check to disable) - Cross-validation: lastBlockHash against fixture, lastPayloadStatus (VALID/INVALID) for engine tests - validate.toml config for client binary paths with per-type overrides (state-bin, block-bin, engine-bin) - Auto bin-workers and xdist tuning per client - Bundled Frontier sanity fixtures for health checks - Shared validate_helpers.py for validation logic Client binary PRs: - geth: ethereum/go-ethereum#34650 - erigon: erigontech/erigon#20315 - besu: besu-eth/besu#10184 - nethermind: NethermindEth/nethermind#11035 - reth: paradigmxyz/reth#23361 - ethrex: lambdaclass/ethrex#6445 - nimbus: status-im/nimbus-eth1#4101 - revm: bluealloy/revm#3544 Tracking issue: ethereum#2319
spencer-tb
added a commit
to spencer-tb/execution-specs
that referenced
this pull request
Apr 8, 2026
New `validate` CLI command for running EEST fixtures directly against client EVM binaries, replacing Hive for execution correctness testing. Usage: validate health # health check all clients validate engine --client geth # engine tests validate state --client besu # state tests validate block --client nethermind # block tests Features: - 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus - Per-type Pydantic result models: StateTestResult, BlockTestResult, EngineTestResult with type-specific fields - Exception matching: maps client error strings to EEST exception types via ExceptionMapper, verifies correct exception for every invalid test (--no-exception-check to disable) - Cross-validation: lastBlockHash against fixture, lastPayloadStatus (VALID/INVALID) for engine tests - validate.toml config for client binary paths with per-type overrides (state-bin, block-bin, engine-bin) - Auto bin-workers and xdist tuning per client - Bundled Frontier sanity fixtures for health checks - Shared validate_helpers.py for validation logic Client binary PRs: - geth: ethereum/go-ethereum#34650 - erigon: erigontech/erigon#20315 - besu: besu-eth/besu#10184 - nethermind: NethermindEth/nethermind#11035 - reth: paradigmxyz/reth#23361 - ethrex: lambdaclass/ethrex#6445 - nimbus: status-im/nimbus-eth1#4101 - revm: bluealloy/revm#3544 Tracking issue: ethereum#2319
spencer-tb
added a commit
to spencer-tb/execution-specs
that referenced
this pull request
Apr 8, 2026
New `validate` CLI command for running EEST fixtures directly against client EVM binaries, replacing Hive for execution correctness testing. Usage: validate health # health check all clients validate engine --client geth # engine tests validate state --client besu # state tests validate block --client nethermind # block tests Features: - 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus - Per-type Pydantic result models: StateTestResult, BlockTestResult, EngineTestResult with type-specific fields - Exception matching: maps client error strings to EEST exception types via ExceptionMapper, verifies correct exception for every invalid test (--no-exception-check to disable) - Cross-validation: lastBlockHash against fixture, lastPayloadStatus (VALID/INVALID) for engine tests - validate.toml config for client binary paths with per-type overrides (state-bin, block-bin, engine-bin) - Auto bin-workers and xdist tuning per client - Bundled Frontier sanity fixtures for health checks - Shared validate_helpers.py for validation logic Client binary PRs: - geth: ethereum/go-ethereum#34650 - erigon: erigontech/erigon#20315 - besu: besu-eth/besu#10184 - nethermind: NethermindEth/nethermind#11035 - reth: paradigmxyz/reth#23361 - ethrex: lambdaclass/ethrex#6445 - nimbus: status-im/nimbus-eth1#4101 - revm: bluealloy/revm#3544 Tracking issue: ethereum#2319
spencer-tb
added a commit
to spencer-tb/execution-specs
that referenced
this pull request
Apr 8, 2026
New `validate` CLI command for running EEST fixtures directly against client EVM binaries, replacing Hive for execution correctness testing. Usage: validate health # health check all clients validate engine --client geth # engine tests validate state --client besu # state tests validate block --client nethermind # block tests Features: - 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus - Per-type Pydantic result models: StateTestResult, BlockTestResult, EngineTestResult with type-specific fields - Exception matching: maps client error strings to EEST exception types via ExceptionMapper, verifies correct exception for every invalid test (--no-exception-check to disable) - Cross-validation: lastBlockHash against fixture, lastPayloadStatus (VALID/INVALID) for engine tests - validate.toml config for client binary paths with per-type overrides (state-bin, block-bin, engine-bin) - Auto bin-workers and xdist tuning per client - Bundled Frontier sanity fixtures for health checks - Shared validate_helpers.py for validation logic Client binary PRs: - geth: ethereum/go-ethereum#34650 - erigon: erigontech/erigon#20315 - besu: besu-eth/besu#10184 - nethermind: NethermindEth/nethermind#11035 - reth: paradigmxyz/reth#23361 - ethrex: lambdaclass/ethrex#6445 - nimbus: status-im/nimbus-eth1#4101 - revm: bluealloy/revm#3544 Tracking issue: ethereum#2319
spencer-tb
added a commit
to spencer-tb/execution-specs
that referenced
this pull request
Apr 8, 2026
New `validate` CLI command for running EEST fixtures directly against client EVM binaries, replacing Hive for execution correctness testing. Usage: validate health # health check all clients validate engine --client geth # engine tests validate state --client besu # state tests validate block --client nethermind # block tests Features: - 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus - Per-type Pydantic result models: StateTestResult, BlockTestResult, EngineTestResult with type-specific fields - Exception matching: maps client error strings to EEST exception types via ExceptionMapper, verifies correct exception for every invalid test (--no-exception-check to disable) - Cross-validation: lastBlockHash against fixture, lastPayloadStatus (VALID/INVALID) for engine tests - validate.toml config for client binary paths with per-type overrides (state-bin, block-bin, engine-bin) - Auto bin-workers and xdist tuning per client - Bundled Frontier sanity fixtures for health checks - Shared validate_helpers.py for validation logic Client binary PRs: - geth: ethereum/go-ethereum#34650 - erigon: erigontech/erigon#20315 - besu: besu-eth/besu#10184 - nethermind: NethermindEth/nethermind#11035 - reth: paradigmxyz/reth#23361 - ethrex: lambdaclass/ethrex#6445 - nimbus: status-im/nimbus-eth1#4101 - revm: bluealloy/revm#3544 Tracking issue: ethereum#2319
spencer-tb
added a commit
to spencer-tb/execution-specs
that referenced
this pull request
Apr 8, 2026
New `validate` CLI command for running EEST fixtures directly against client EVM binaries, replacing Hive for execution correctness testing. Usage: validate health # health check all clients validate engine --client geth # engine tests validate state --client besu # state tests validate block --client nethermind # block tests Features: - 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus - Per-type Pydantic result models: StateTestResult, BlockTestResult, EngineTestResult with type-specific fields - Exception matching: maps client error strings to EEST exception types via ExceptionMapper, verifies correct exception for every invalid test (--no-exception-check to disable) - Cross-validation: lastBlockHash against fixture, lastPayloadStatus (VALID/INVALID) for engine tests - validate.toml config for client binary paths with per-type overrides (state-bin, block-bin, engine-bin) - Auto bin-workers and xdist tuning per client - Bundled Frontier sanity fixtures for health checks - Shared validate_helpers.py for validation logic Client binary PRs: - geth: ethereum/go-ethereum#34650 - erigon: erigontech/erigon#20315 - besu: besu-eth/besu#10184 - nethermind: NethermindEth/nethermind#11035 - reth: paradigmxyz/reth#23361 - ethrex: lambdaclass/ethrex#6445 - nimbus: status-im/nimbus-eth1#4101 - revm: bluealloy/revm#3544 Tracking issue: ethereum#2319
spencer-tb
added a commit
to spencer-tb/execution-specs
that referenced
this pull request
Apr 8, 2026
New `validate` CLI command for running EEST fixtures directly against client EVM binaries, replacing Hive for execution correctness testing. Usage: validate health # health check all clients validate engine --client geth # engine tests validate state --client besu # state tests validate block --client nethermind # block tests Features: - 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus - Per-type Pydantic result models: StateTestResult, BlockTestResult, EngineTestResult with type-specific fields - Exception matching: maps client error strings to EEST exception types via ExceptionMapper, verifies correct exception for every invalid test (--no-exception-check to disable) - Cross-validation: lastBlockHash against fixture, lastPayloadStatus (VALID/INVALID) for engine tests - validate.toml config for client binary paths with per-type overrides (state-bin, block-bin, engine-bin) - Auto bin-workers and xdist tuning per client - Bundled Frontier sanity fixtures for health checks - Shared validate_helpers.py for validation logic Client binary PRs: - geth: ethereum/go-ethereum#34650 - erigon: erigontech/erigon#20315 - besu: besu-eth/besu#10184 - nethermind: NethermindEth/nethermind#11035 - reth: paradigmxyz/reth#23361 - ethrex: lambdaclass/ethrex#6445 - nimbus: status-im/nimbus-eth1#4101 - revm: bluealloy/revm#3544 Tracking issue: ethereum#2319
spencer-tb
added a commit
to spencer-tb/execution-specs
that referenced
this pull request
Apr 8, 2026
New `validate` CLI command for running EEST fixtures directly against client EVM binaries, replacing Hive for execution correctness testing. Usage: validate health # health check all clients validate engine --client geth # engine tests validate state --client besu # state tests validate block --client nethermind # block tests Features: - 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus - Per-type Pydantic result models: StateTestResult, BlockTestResult, EngineTestResult with type-specific fields - Exception matching: maps client error strings to EEST exception types via ExceptionMapper, verifies correct exception for every invalid test (--no-exception-check to disable) - Cross-validation: lastBlockHash against fixture, lastPayloadStatus (VALID/INVALID) for engine tests - validate.toml config for client binary paths with per-type overrides (state-bin, block-bin, engine-bin) - Auto bin-workers and xdist tuning per client - Bundled Frontier sanity fixtures for health checks - Shared validate_helpers.py for validation logic Client binary PRs: - geth: ethereum/go-ethereum#34650 - erigon: erigontech/erigon#20315 - besu: besu-eth/besu#10184 - nethermind: NethermindEth/nethermind#11035 - reth: paradigmxyz/reth#23361 - ethrex: lambdaclass/ethrex#6445 - nimbus: status-im/nimbus-eth1#4101 - revm: bluealloy/revm#3544 Tracking issue: ethereum#2319
spencer-tb
added a commit
to spencer-tb/execution-specs
that referenced
this pull request
Apr 8, 2026
New `validate` CLI command for running EEST fixtures directly against client EVM binaries, replacing Hive for execution correctness testing. Usage: validate health # health check all clients validate engine --client geth # engine tests validate state --client besu # state tests validate block --client nethermind # block tests Features: - 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus - Per-type Pydantic result models: StateTestResult, BlockTestResult, EngineTestResult with type-specific fields - Exception matching: maps client error strings to EEST exception types via ExceptionMapper, verifies correct exception for every invalid test (--no-exception-check to disable) - Cross-validation: lastBlockHash against fixture, lastPayloadStatus (VALID/INVALID) for engine tests - validate.toml config for client binary paths with per-type overrides (state-bin, block-bin, engine-bin) - Auto bin-workers and xdist tuning per client - Bundled Frontier sanity fixtures for health checks - Shared validate_helpers.py for validation logic Client binary PRs: - geth: ethereum/go-ethereum#34650 - erigon: erigontech/erigon#20315 - besu: besu-eth/besu#10184 - nethermind: NethermindEth/nethermind#11035 - reth: paradigmxyz/reth#23361 - ethrex: lambdaclass/ethrex#6445 - nimbus: status-im/nimbus-eth1#4101 - revm: bluealloy/revm#3544 Tracking issue: ethereum#2319
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add three standalone CLI runners for direct execution of EEST fixtures, enabling
consume directintegration without Hive. Each runner has--workers N,--run <regex>,--json, and--path <dir>flags.Why new CLIs?
The existing test infrastructure uses either
cargo testharnesses (datatest-stablefor blockchain tests) or thestate_v2binary. Neither supports the flags needed forconsume direct:datatest-stableharness — no--runregex filter, no--jsonoutput, no--workerscontrol, hardcoded fixture paths. Can't be called as a standalone binary from EELS.state_v2runner — creates a fullStore(with background threads, trie cache, channels) per test case, making it 89x slower than necessary. State tests only need a HashMap → VM → check result.The new runners are standalone binaries that EELS
consume directcan invoke directly.`ef_tests-statetest`
Fast state test runner using LEVM directly with in-memory HashMap state. No Store, no trie, no background threads — just loads pre-state into memory, executes one transaction, and verifies post-state accounts directly.
`ef_tests-blocktest`
Thin CLI wrapper around the existing `ef_tests-blockchain` crate. Calls `parse_and_execute()` per fixture file with rayon parallelism.
`ef_tests-enginetest`
Engine test runner for `blockchain_test_engine` fixtures. Routes payloads through the real Engine API RPC handler functions: `NewPayloadV1-V5Request.parse()` + `.handle(context)` and `ForkChoiceUpdatedV1-V4.handle(context)`. Same code path as `consume engine` via Hive, minus the HTTP transport layer.
Uses a shared `SyncManager`/`PeerHandler` via `OnceCell` to avoid thread exhaustion. Per-test `RpcApiContext` with isolated `block_worker_channel`.
Benchmarks
Tested against EEST v5.3.0 stable fixtures on Apple M-series.
`ef_tests-statetest` (36,220 tests):
`ef_tests-blocktest` (2,776 files, ~40k test cases):
`ef_tests-enginetest` — exercises the same engine code paths as `consume engine` (40,521 tests):
100% pass rate across all three runners (3 known statetest failures from invalid CREATE blob transactions).
Usage
Related: ethereum/go-ethereum#34650, erigontech/erigon#20315, NethermindEth/nethermind#11035, besu-eth/besu#10184, ethereum/execution-spec-tests#2319