Disable common order-randomising pytest plugins #424
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Order-randomising pytest plugins, while brilliant for their intended purpose of rumbling hidden test dependencies, are near-pessimal for mutmut's use case. For a given mutation run, a subset of tests, ordered by increasing runtime, are fed to pytest. The idea is to run fast tests first, increasing the chances of bailing out quickly with a test failure on the mutant being tested. Randomising the order of such tests destroys that ordering and thus blows out mutation test times, sometimes to the point of mutants that would normally be killed ending up timing out instead. As such, pass parms to the pytest runner when testing a mutant to disable two common order-randomisation plugins, pytest-randomly and pytest-random-order.
Collaborator
Author
Owner
|
If you don't have these plugins that command will just be ignored? |
Collaborator
Author
|
@boxed - yes, as you can see in the CI checks - if neither |
Owner
|
Great! Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Order-randomising pytest plugins, while brilliant for their intended purpose of rumbling hidden test dependencies, are near-pessimal for mutmut's use case.
For a given mutation run, a subset of tests, ordered by increasing runtime, are fed to pytest. The idea is to run fast tests first, increasing the chances of bailing out quickly with a test failure on the mutant being tested.
Randomising the order of such tests destroys that ordering and thus blows out mutation test times, sometimes to the point of mutants that would normally be killed ending up timing out instead.
As such, pass parms to the pytest runner when testing a mutant to disable two common order-randomisation plugins, pytest-randomly and pytest-random-order.
For projects that use neither of those plugins, this PR is a no-op. As the checks show, pytest will gladly run without the to-be-disabled plugins even installed.
For example, running over 11
ObjectStatistics.__init__mutants in PyRoute - which has ~800-odd distinct tests in its test suite and uses thepytest-randomlyplugin:Status quo:
With this change:
That's something, assuming I haven't mucked up the numbers, like a 150x speedup simply by avoiding the test order randomisation. With the exact same tests and mutants in each case.
Addresses #423 .