Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
591f2ec
State transition test generator, initial version
ericsson49 Jul 28, 2026
04379cc
update test methodology doc
ericsson49 Jul 28, 2026
d60637a
added builder_deposit_request
ericsson49 Jul 28, 2026
5686912
added process_withdrawal_request
ericsson49 Jul 28, 2026
a3e3d84
added process_consolidation_request
ericsson49 Jul 28, 2026
7dddbdd
parameterized aspects added
ericsson49 Jul 28, 2026
72682c3
add parameterization to builder aspects
ericsson49 Jul 28, 2026
7866744
added process_deposit_request
ericsson49 Jul 28, 2026
b57bec0
added process_proposer_slashing
ericsson49 Jul 28, 2026
2eafc25
added process_payload_attestation
ericsson49 Jul 28, 2026
b1c370a
added process_attestation
ericsson49 Jul 28, 2026
38f76be
improvements
ericsson49 Jul 28, 2026
1e2a18d
added process_withdrawals
ericsson49 Jul 28, 2026
4ad5e73
process_attestation test gen improvements after review
ericsson49 Jul 28, 2026
76f2395
added process_parent_execution_payload
ericsson49 Jul 28, 2026
ae5bde9
added test runner for state transition tests
ericsson49 Jul 28, 2026
53afa76
added process_pending_deposits
ericsson49 Jul 29, 2026
4470442
added support of builder_pending_payments and ptc_window
ericsson49 Jul 29, 2026
2e02b13
add builder_payment_quorum aspect
ericsson49 Jul 29, 2026
c797d98
added ptc_window test gen
ericsson49 Jul 29, 2026
4704551
added builder_pending_payments test gen
ericsson49 Jul 29, 2026
26ef1d4
add bls_setting: 1 to pending_deposits
ericsson49 Jul 29, 2026
85bd622
more tests: 'all' profile, coarser-grained profiles
ericsson49 Aug 19, 2026
1a9b48f
reformat builder_pending_payments/validation
ericsson49 Aug 19, 2026
a663cfb
refactoring of run/validate
ericsson49 Aug 21, 2026
ff68465
move shared code to the main validation.py
ericsson49 Aug 21, 2026
b5e0111
make run.py more uniform
ericsson49 Aug 21, 2026
7ca2666
factor out common materializer code
ericsson49 Aug 25, 2026
f3727be
get rid of main in coverage.py
ericsson49 Aug 25, 2026
dc5eeeb
simplify ptc_window
ericsson49 Aug 25, 2026
6d8d6a4
added more coverage dimensions to ptc_window
ericsson49 Aug 25, 2026
bdee598
add active validator coverage dimension
ericsson49 Aug 25, 2026
4e13247
drop README.md for state transition test generator handlers
ericsson49 Aug 25, 2026
3895965
drop handler's test output directory before test generation
ericsson49 Aug 25, 2026
eddd8e9
improved payload_attestation comptests generator
ericsson49 Aug 25, 2026
cd3da30
simplify handler validation for state transition comptests
ericsson49 Aug 25, 2026
50acdf4
improve deposit_request handler comptests
ericsson49 Aug 25, 2026
a2f58da
introduce a shared helper for actual vs claimed comparison
ericsson49 Aug 25, 2026
94e9422
reorganized test generation, state transition comptests
ericsson49 Aug 26, 2026
23897cb
support validating only specific handler
ericsson49 Aug 26, 2026
36c1f61
add --profile to state transition comptests
ericsson49 Aug 26, 2026
ff235c5
comptests: minor fixes
ericsson49 Aug 26, 2026
5828841
improve withdrawal_credential aspect
ericsson49 Aug 26, 2026
decbff7
imporved deposit_amount aspect
ericsson49 Aug 26, 2026
d91207f
drop a file commited by mistake
ericsson49 Aug 27, 2026
b6768c5
change output directory for state transition comptests
ericsson49 Aug 27, 2026
d6da0c6
make comptests: add support of kind=state_transition
ericsson49 Aug 27, 2026
a945c1d
add state_transition comptests README.md
ericsson49 Aug 27, 2026
be9f836
comptests: pass parent_slot to process_attestation
ericsson49 Aug 27, 2026
24831ec
hash -> sha256
ericsson49 Aug 27, 2026
131fc68
pass payments/withdrawals as named param
ericsson49 Aug 27, 2026
b1b31e9
Use PendingConsolidations(data=[...])
ericsson49 Aug 27, 2026
7bf9800
Construct bid using spec.BlobKZGCommitments(data=commitments)
ericsson49 Aug 27, 2026
fca54a0
fix compatibility issues after specs update
ericsson49 Aug 27, 2026
e38ed84
Use spec.PendingPartialWithdrawals(data=entries)
ericsson49 Aug 27, 2026
b20bc04
fix code fromatting
ericsson49 Aug 27, 2026
64f263c
fix ruff check issues
ericsson49 Aug 27, 2026
ccaa5a8
fix ruff format issues
ericsson49 Aug 27, 2026
6740598
set withdrawals_over_limit=true when there are many sources
ericsson49 Aug 27, 2026
97e36e6
fix wrong package names
ericsson49 Aug 27, 2026
3b88df6
fix wrong package names
ericsson49 Aug 27, 2026
44c3e33
fix model problem for proposer_slashing
ericsson49 Aug 27, 2026
4be8d06
drop stale debugging correction
ericsson49 Aug 28, 2026
4ec7013
drop model_path from Materializer
ericsson49 Aug 31, 2026
18e25db
streamline handler test generator interface
ericsson49 Aug 31, 2026
43974d2
introduce test providers and catalog.py
ericsson49 Aug 31, 2026
6ad2da5
fix formatting
ericsson49 Aug 31, 2026
aae7398
fix formatting
ericsson49 Aug 31, 2026
c5525a9
fix formatting
ericsson49 Aug 31, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 34 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ ALL_EXECUTABLE_SPEC_NAMES = \
_sync \
build_docs \
clean \
comptests \
help \
lint \
serve_docs \
Expand Down Expand Up @@ -106,15 +107,18 @@ help-verbose:
@echo ""
@echo "$(BOLD)make comptests$(NORM)"
@echo ""
@echo " Generates compliance tests for fork choice. These tests verify that"
@echo " implementations correctly handle fork choice scenarios."
@echo " Generates compliance tests. These tests verify that implementations"
@echo " correctly handle fork choice and state transition scenarios."
@echo " Uses pytest collection and xdist parallelism."
@echo ""
@echo " Parameters:"
@echo " kind=<kind> Test kind: fork_choice (default), state_transition"
@echo " fc_gen_config=<config> Configuration size (tiny, small, standard; default: tiny)"
@echo " fork=<fork> Generate for specific fork (comma-separated)"
@echo " preset=<preset> Generate for specific preset (comma-separated)"
@echo " comptests_dir=<dir> Output directory for generated compliance tests"
@echo " handler=<handler> State-transition handler (default: all)"
@echo " profile=<profile> State-transition profile (smoke, standard, all; default: standard)"
@echo " threads=N Number of threads to use"
@echo " seed=N Override test seeds (fuzzing mode)"
@echo " group_slice_index=N 0-based shard index for deterministic test-group slicing"
Expand All @@ -127,6 +131,8 @@ help-verbose:
@echo " make comptests comptests_dir=./compliance-spec-tests/tests"
@echo " make comptests fc_gen_config=standard fork=deneb preset=mainnet threads=8"
@echo " make comptests fc_gen_config=tiny fork=gloas group_slice_index=0 group_slice_count=4"
@echo " make comptests kind=state_transition"
@echo " make comptests kind=state_transition handler=withdrawals profile=smoke"
@echo ""
@echo "$(BOLD)DOCUMENTATION$(NORM)"
@echo "$(BOLD)--------------------------------------------------------------------------------$(NORM)"
Expand Down Expand Up @@ -289,6 +295,9 @@ lint: _pyspec
COMMA:= ,
DEFAULT_COMPTESTS_DIR = $(CURDIR)/../compliance-spec-tests/tests
COMPTESTS_DIR = $(if $(comptests_dir),$(comptests_dir),$(DEFAULT_COMPTESTS_DIR))
COMPTESTS_KIND = $(if $(kind),$(kind),fork_choice)

ifeq ($(COMPTESTS_KIND),fork_choice)

# Generate compliance tests (fork choice).
comptests: FC_GEN_CONFIG := $(if $(fc_gen_config),$(fc_gen_config),tiny)
Expand All @@ -313,6 +322,29 @@ comptests: _pyspec
$(MAYBE_GROUP_SLICE_COUNT) \
$(CURDIR)/tests/generators/compliance_runners/fork_choice/generate_comptests.py

else ifeq ($(COMPTESTS_KIND),state_transition)

# Generate compliance tests (state transition).
comptests: MAYBE_HANDLER := $(if $(handler),--handler $(handler))
comptests: MAYBE_PROFILE := $(if $(profile),--profile $(profile))
comptests: MAYBE_PARALLEL := $(if $(filter 1,$(threads)),,$(if $(threads),-n $(threads) --dist=worksteal,-n logical --dist=worksteal))
comptests: _pyspec
@$(UV_RUN) pytest \
$(MAYBE_PARALLEL) \
--capture=no \
--comptests-output=$(COMPTESTS_DIR) \
$(MAYBE_HANDLER) \
$(MAYBE_PROFILE) \
$(CURDIR)/tests/generators/compliance_runners/state_transition/generate_comptests.py

else

comptests:
@echo "Unsupported compliance test kind: $(COMPTESTS_KIND)" >&2
@exit 1

endif

###############################################################################
# Cleaning
###############################################################################
Expand Down
2 changes: 2 additions & 0 deletions tests/formats/epoch_processing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,8 @@ Sub-transitions:
- `pending_consolidations` (>=Electra)
- `pending_deposits` (>=Electra)
- `pending_randao_commitments` (>=EIP-8321)
- `builder_pending_payments` (>=Gloas)
- `ptc_window` (>=Gloas)

The resulting state should match the expected `post` state.

Expand Down
113 changes: 113 additions & 0 deletions tests/generators/compliance_runners/state_transition/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# State-transition compliance test generator

State-transition compliance test generator intended to produce reference tests
for individual state-transition handlers. Each handler models semantic coverage
conditions, materializes concrete SSZ test vectors, and validates the generated
vectors against the executable specification.

The generator currently targets the `gloas` fork with the `minimal` preset. Its
test cases follow the standard
[operations test format](../../../formats/operations/README.md) or
[epoch processing test format](../../../formats/epoch_processing/README.md),
depending on the handler.

## Handlers

Operation handlers:

- `attestation`
- `builder_deposit_request`
- `builder_exit_request`
- `consolidation_request`
- `deposit_request`
- `execution_payload_bid`
- `parent_execution_payload`
- `payload_attestation`
- `proposer_slashing`
- `withdrawal_request`
- `withdrawals`

Epoch-processing handlers:

- `builder_pending_payments`
- `pending_deposits`
- `ptc_window`

## Generating tests

From the repository root:

```bash
make comptests kind=state_transition
```

The default profile is `standard`, and all handlers are generated. Select a
handler, profile, or output directory with Make variables:

```bash
make comptests kind=state_transition handler=withdrawals profile=smoke
make comptests kind=state_transition profile=all
make comptests kind=state_transition comptests_dir=../compliance-spec-tests/tests
```

The supported profiles are:

- `smoke` — one representative per terminal outcome
- `normal` — cases with no independently failed conditions
- `exceptional` — single-fault cases
- `standard` — `normal` plus `exceptional`
- `all` — every distinct model coverage signature

`make comptests` is the supported generation path. The underlying module can
also be run directly:

```bash
uv run python -m tests.generators.compliance_runners.state_transition.run
uv run python -m tests.generators.compliance_runners.state_transition.run \
--handler withdrawals --profile smoke
uv run python -m tests.generators.compliance_runners.state_transition.run \
--comptests-output /path/to/output
```

The direct command writes to each handler's local `reftests/` directory unless
`--comptests-output` is provided. It validates each handler immediately after
materialization. Handler-specific MiniZinc models, coverage definitions,
materializers, and validators are located in the corresponding provider
directory.

A handler may have multiple provider directories. Their cases are appended to
the same handler output with distinct case numbers and validated independently;
the provider directory is an implementation detail, while the generated manifest
continues to use the protocol handler name.

## Running generated tests

From the repository root, run the compliance runner against a directory
containing generated `reftests`:

```bash
uv run pytest \
tests/generators/compliance_runners/state_transition/runner/test_run.py \
--test-dir ${test_dir}
```

The `--test-dir` option can be repeated to run multiple test roots. Optional
`--start` and `--limit` arguments select a slice of the discovered cases.

## Output

Generated cases use this layout:

```text
<output>/minimal/gloas/
operations/<handler>/main/case_XXXX/
epoch_processing/<handler>/main/case_XXXX/
```

Each case contains `pre.ssz_snappy`, the operation input when applicable,
`post.ssz_snappy` when the handler accepts the input, `meta.yaml`,
`manifest.yaml`, and `dimensions.yaml` with the claimed coverage dimensions.

The modelling approach and the distinction between coverage dimensions,
materialization, and validation are documented in
[`TEST_METHODOLOGY.md`](TEST_METHODOLOGY.md).
Loading