Skip to content

feat(fill,consume): enable xfail/skip marking of tests on a per client basis #1363

Open
@danceratopz

Description

@danceratopz

Tldr: Add the ability for clients to mark a test case to conditionally fail or simply be skipped.

End goal:

  • Single source of truth for xfails and skips for all clients (by applying pytest marks directly to ./tests? this seems most robust?)
  • Ability to add a reason for the xfail/skip.
  • Propagation of xfails/skips with reason to hiveview (and other) results for a transparent overview (add registered skips/xfails to the info section of the test fixture JSON?).

Two example use cases (Besu):

  1. Besu's client no longer supports reorgs post-Merge (as its handled by the CL), so can't be expected to pass UncleFromSideChain_Cancun. The test is valid for clients that do maintain this functionality, so shouldn't necessarily be removed.
  2. Besu via consume-rlp fails, but passes consume-engine with tests/cancun/eip4844_blobs/test_excess_blob_gas_fork_transition.py::test_invalid_pre_fork_block_with_blob_fields[fork_ShanghaiToCancunAtTime15k-blockchain_test-excess_blob_gas_present_False-blob_gas_used_present_True] (and variations). This is due to the RLPBlockImporter not entirely respecting the Engine API validation rules and would require some, other unnecessary, work to fix.

Requires #1430.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions