feat(validate): new validate command for direct client binary testing

[spencer-tb]

🗒️ Description

New validate (props to @raxhvl for the naming convention) CLI command for running EEST fixtures directly against client EVM binaries. Replaces consume direct with a cleaner UX — type is the subcommand, --client is required, no --bin or --type flags.

validate health                    # health check all clients
validate engine --client geth      # engine tests
validate state --client besu       # state tests
validate block --client nethermind # block tests

Features

• 7 clients: geth, besu, nethermind, erigon, reth, ethrex, nimbus
• Per-type Pydantic models: StateTestResult (stateRoot), BlockTestResult (lastBlockHash), EngineTestResult (lastPayloadStatus)
• Exception matching: maps client error strings to EEST exception types, verifies correct exception for every invalid test (--no-exception-check to disable)
• Cross-validation: lastBlockHash against fixture, lastPayloadStatus (VALID/INVALID) for engine tests
• validate.toml config for client binary paths with per-type overrides (state-bin, block-bin, engine-bin)
• Auto-tuning: bin-workers and xdist settings per client
• Health checks: version detection + sanity fixture per client per type
• Aliases: go-ethereum resolves to geth
• Fully standalone: no dependency on consume plugin
• Removes consume direct: replaced entirely by validate

Pydantic Result Models

All client adapters return structured results via a shared model hierarchy in cli_types.py:

FixtureTestResult          # base — name, pass, fork, error
├── StateTestResult        # + stateRoot
├── BlockTestResult        # + lastBlockHash
│   └── EngineTestResult   # + lastPayloadStatus

Model	Extra Fields	Used By
FixtureTestResult	name, pass, fork, error	base class
StateTestResult	stateRoot	validate state
BlockTestResult	lastBlockHash	validate block
EngineTestResult	lastBlockHash, lastPayloadStatus	validate engine

Each client binary outputs JSON matching these schemas. The shared validate_helpers.py module validates results against fixture expectations:

• stateRoot compared to fixture's postState hash
• lastBlockHash compared to fixture's lastblockhash
• lastPayloadStatus checked as VALID (positive) or INVALID (negative test)
• error matched through ExceptionMapper against fixture's expectException / validationError

Results (v5.3.0 fixtures, Apple M-series)

Client	Engine Time	Engine Pass %	Block Time	Block Pass %	State Time	State Pass %	Exc Check (state/block/engine)	Default Flags
geth	58s	99.98%	64s	99.96%	pending	pending	✓ / ✓ / ✓	--bin-workers 8 (auto)
ethrex	66s	100%	66s	100%	pending	pending	✗ / ✗ / ✗	--bin-workers 8 (auto)
nimbus	71s	100%	67s	100%	pending	pending	✗ / ✗ / ✗	--fast mode, sequential
reth	111s	99.10%	—	broken	pending	pending	✗ / — / ✗	-n 2 (auto)
besu	174s	99.99%	65s	100%	pending	pending	✓ / ✗ / ✓	--bin-workers 8 (auto)
nethermind	176s	99.98%	107s	99.97%	pending	pending	✓ / ✓ / ✓	--bin-workers 4 (auto)
erigon	26m	100%	pending	pending	pending	pending	✗ / ✓ / ✗	--bin-workers 8 (auto)

Exception check = client reports validation error on pass for negative tests, enabling EELS-side exception verification.

Client PRs (adding statetest/blocktest/enginetest runners)

• geth: cmd/evm: add enginetest command for direct engine fixture execution go-ethereum#34650
• erigon: cmd/evm: add enginetest command and parallel workers for test runners erigontech/erigon#20315
• besu: evmtool: add engine-test subcommand and parallel workers for test runners besu-eth/besu#10184
• nethermind: nethtest: add --engineTest, --stateTest, --jsonout, --workers flags NethermindEth/nethermind#11035
• reth: testing: add enginetest, enginextest, and blocktest CLI subcommands paradigmxyz/reth#23361
• ethrex: feat(l1): ef_tests: add standalone statetest, blocktest, and enginetest CLI runners lambdaclass/ethrex#6445
• nimbus: eest: add standalone CLI runners for statetest, blocktest, and enginetest status-im/nimbus-eth1#4101
• revm: revme: add --run regex filter to statetest and blockchaintest bluealloy/revm#3544

🔗 Related Issues or PRs

Fixes #2319

✅ Checklist

• All: Ran fast static checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:

  just static

• All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
• All: Considered updating the online docs in the ./docs/ directory.
• All: Set appropriate labels for the changes (only maintainers can apply labels).