Make xfail strict - #3232
Merged
Merged
Conversation
An xfail that starts passing is a fix nobody hears about: the mark stays, the test keeps being reported as expected-to-fail, and the next person to touch that code has no idea it already works. With xfail_strict an unexpected pass fails the suite, so the mark gets removed when the underlying issue is.
Merging this PR will not alter performance
Comparing Footnotes
|
Pierre-Sassoulas
force-pushed
the
xfail-strict
branch
from
August 18, 2026 10:22
6b87297 to
f3e3014
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3232 +/- ##
=======================================
Coverage 93.65% 93.65%
=======================================
Files 93 93
Lines 11583 11583
=======================================
Hits 10848 10848
Misses 735 735
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type of Changes
Description
Sets
xfail_strict = truefor pytest.Right now an
xfailthat starts passing is silently absorbed: pytest reportsXPASS, the suite stays green, and the mark stays in the file. So the test goeson being advertised as expected-to-fail long after it works, and the next person
to read it has no reason to doubt that. With
xfail_strictan unexpected passfails the suite, which is the only moment anyone would think to delete the mark.
This came up on #3211, where two
xfails were added for known holes. Non-strict,they would have outlived the fixes.
No test needed changing. All 15
xfails still fail, on 3.13 and on 3.14:I can only check Linux locally, so CI is the real test for the two marks whose
outcome could plausibly differ elsewhere ---
c_bufferintests/brain/test_ctypes.pyandtest_factory_methods_inside_binary_operation.If either turns up as
XPASSon Windows or PyPy the fix is a condition on themark, not a revert.
Noticed while here, not touched: the reason on
test_factory_methods_inside_binary_operationstill says "cannot be inferred onPython 3.8", which astroid has not supported for a while. The test does still
fail, so the mark is right and only the reason is stale.