Skip to content

Make xfail strict - #3232

Merged
Pierre-Sassoulas merged 1 commit into
mainfrom
xfail-strict
Aug 18, 2026
Merged

Make xfail strict#3232
Pierre-Sassoulas merged 1 commit into
mainfrom
xfail-strict

Conversation

@Pierre-Sassoulas

Copy link
Copy Markdown
Member

Type of Changes

Type
🔨 Refactor

Description

Sets xfail_strict = true for pytest.

Right now an xfail that starts passing is silently absorbed: pytest reports
XPASS, the suite stays green, and the mark stays in the file. So the test goes
on being advertised as expected-to-fail long after it works, and the next person
to read it has no reason to doubt that. With xfail_strict an unexpected pass
fails the suite, which is the only moment anyone would think to delete the mark.

This came up on #3211, where two xfails were added for known holes. Non-strict,
they would have outlived the fixes.

No test needed changing. All 15 xfails still fail, on 3.13 and on 3.14:

2080 passed, 61 skipped, 15 xfailed     # 3.13
2050 passed, 90 skipped, 15 xfailed     # 3.14

I can only check Linux locally, so CI is the real test for the two marks whose
outcome could plausibly differ elsewhere --- c_buffer in
tests/brain/test_ctypes.py and test_factory_methods_inside_binary_operation.
If either turns up as XPASS on Windows or PyPy the fix is a condition on the
mark, not a revert.

Noticed while here, not touched: the reason on
test_factory_methods_inside_binary_operation still says "cannot be inferred on
Python 3.8", which astroid has not supported for a while. The test does still
fail, so the mark is right and only the reason is stale.

An xfail that starts passing is a fix nobody hears about: the mark stays,
the test keeps being reported as expected-to-fail, and the next person to
touch that code has no idea it already works. With xfail_strict an
unexpected pass fails the suite, so the mark gets removed when the
underlying issue is.
@codspeed-hq

codspeed-hq Bot commented Aug 18, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 3 untouched benchmarks
⏩ 1 skipped benchmark1


Comparing xfail-strict (f3e3014) with main (62ef6b3)

Open in CodSpeed

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports.

@codecov

codecov Bot commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.65%. Comparing base (62ef6b3) to head (f3e3014).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #3232   +/-   ##
=======================================
  Coverage   93.65%   93.65%           
=======================================
  Files          93       93           
  Lines       11583    11583           
=======================================
  Hits        10848    10848           
  Misses        735      735           
Flag Coverage Δ
linux 93.51% <ø> (ø)
pypy 93.65% <ø> (ø)
windows 93.62% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Pierre-Sassoulas Pierre-Sassoulas added the Skip news 🔇 This change does not require a changelog entry label Aug 18, 2026
@Pierre-Sassoulas
Pierre-Sassoulas merged commit 60ed8a4 into main Aug 18, 2026
44 of 71 checks passed
@Pierre-Sassoulas
Pierre-Sassoulas deleted the xfail-strict branch August 18, 2026 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Skip news 🔇 This change does not require a changelog entry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant