Open
Description
Tldr: Add the ability for clients to mark a test case to conditionally fail or simply be skipped.
End goal:
- Single source of truth for xfails and skips for all clients (by applying pytest marks directly to
./tests
? this seems most robust?) - Ability to add a reason for the xfail/skip.
- Propagation of xfails/skips with reason to hiveview (and other) results for a transparent overview (add registered skips/xfails to the
info
section of the test fixture JSON?).
Two example use cases (Besu):
- Besu's client no longer supports reorgs post-Merge (as its handled by the CL), so can't be expected to pass
UncleFromSideChain_Cancun
. The test is valid for clients that do maintain this functionality, so shouldn't necessarily be removed. - Besu via
consume-rlp
fails, but passesconsume-engine
withtests/cancun/eip4844_blobs/test_excess_blob_gas_fork_transition.py::test_invalid_pre_fork_block_with_blob_fields[fork_ShanghaiToCancunAtTime15k-blockchain_test-excess_blob_gas_present_False-blob_gas_used_present_True]
(and variations). This is due to the RLPBlockImporter not entirely respecting the Engine API validation rules and would require some, other unnecessary, work to fix.