Skip to content

Is there a way to mark xfail tests, also with matching output? #129

Open
@stdedos

Description

@stdedos

I would like to be able:

  1. Test is marked as xfail if-f it does not match out: |, but matches xfail: | (imaginary attribute), but continues
  2. Test is marked as xpass if-f it matches out: | (preferably strict, since it will lead to actually noticing in CI/CD
  3. Otherwise it is marked as fail

This is clearly wrong:

https://github.com/stdedos/matplotlib-stubs-hoel/actions/runs/6221692521/job/16884150094

And I'd like to:

  1. Track its progress - if it somehow "changes". I cannot do that right now, since expected_fail means "negative test" (instead of pytest's xfail)
  2. Revert "the useless todo". If it happens to match "exactly", then this will be triggered. But otherwise, the slightest deviation will continue masking this

This is (hopefully) illustrated by this example:

$ cat test_test.py
import pytest

EXPECTED = 2
XFAIL = 3


@pytest.mark.parametrize(
    "result",
    [
        1,
        pytest.param(EXPECTED, marks=pytest.mark.xfail(strict=True)),
        pytest.param(XFAIL, marks=pytest.mark.xfail(strict=True)),
    ],
)
def test_increment(result):
    assert result == EXPECTED
$  pytest test_test.py 
========================== test session starts ==========================
platform linux -- Python 3.8.10, pytest-7.4.1, pluggy-1.3.0
configfile: pyproject.toml
plugins: mypy-plugins-3.0.0, dash-2.13.0, pudb-0.7.0, mypy-testing-0.1.1
collected 3 items

test_test.py FFx                                                   [100%]

=============================== FAILURES ================================
___________________________ test_increment[1] ___________________________

result = 1

    @pytest.mark.parametrize(
        "result",
        [
            1,
            pytest.param(EXPECTED, marks=pytest.mark.xfail(strict=True)),
            pytest.param(XFAIL, marks=pytest.mark.xfail(strict=True)),
        ],
    )
    def test_increment(result):
>       assert result == EXPECTED
E       assert 1 == 2

test_test.py:16: AssertionError
___________________________ test_increment[2] ___________________________
[XPASS(strict)] 
======================== short test summary info ========================
FAILED test_test.py::test_increment[1] - assert 1 == 2
FAILED test_test.py::test_increment[2]
===================== 2 failed, 1 xfailed in 0.10s ======================

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions