Added option --order-after-ff

- allows to run pytest-order after built-in hooks
  like the --failed-first option

Author: mrbean-bremen

CHANGELOG.md

@@ -5,15 +5,18 @@
 ### Changes
 * removed official support for Python 3.7-3.9 (EOL), added Python 3.13 and 3.14
 
+### New features
+* added option `--order-after-ff`, that allows to run `pytest-order` after built-in hooks
+  like the `--failed-first` option  (see [#234](https://github.com/pytest-dev/pytest-order/issues/234))
+
 ### Infrastructure
 * use trusted publisher for release (see https://docs.pypi.org/trusted-publishers/)
 * use `pyproject.toml` for project setup
 
 ### Documentation
 * use a theme for documentation supporting dark mode
-
-### Documentation
 * added use case for ordering test modules (see [#51](https://github.com/pytest-dev/pytest-order/issues/51))
+* fixed documentation for `--indulgent-ordering` option
 
 ## [Version 1.3.0](https://pypi.org/project/pytest-order/1.3.0/) (2024-08-22)
 Allows to fail tests that cannot be ordered.

README.md

@@ -84,7 +84,7 @@ yields the output:
 
     $ pytest test_foo.py -vv
     ============================= test session starts ==============================
-    platform darwin -- Python 3.7.1, pytest-5.4.3, py-1.8.1, pluggy-0.13.1 -- env/bin/python
+    platform linux -- Python 3.11.11, pytest-9.0.3, pluggy-1.6.0 -- env/bin/python
     plugins: order
     collected 2 items
 

docs/source/configuration.rst

@@ -503,26 +503,35 @@ gives you now::
 ``--indulgent-ordering``
 ------------------------
 You may sometimes find that you want to suggest an ordering of tests, while
-allowing it to be overridden for good reason. For example, if you run your test
-suite in parallel and have a number of tests which are particularly slow, it
-might be desirable to start those tests running first, in order to optimize
-your completion time. You can use the ``pytest-order`` plugin to inform pytest
-of this.
-
-Now suppose you also want to prioritize tests which failed during the
-previous run, by using the ``--failed-first`` option. By default,
-pytest-order will override the ``--failed-first`` order, but by adding the
-``--indulgent-ordering`` option, you can ask pytest to run the sort from
-pytest-order *before* the sort from ``--failed-first``, allowing the failed
-tests to be sorted to the front (note that in pytest versions from 6.0 on,
-this seems not to be needed anymore, at least in this specific case).
+allowing it to be overridden for good reason. If you have another plugin installed
+that also affects the test order, and you want these changes made after ``pytest-order``
+has ordered the tests, this may not work: ``pytest-order`` always tries to run last.
+Using this option, ``pytest-order`` will try to run first instead, allowing other plugins
+to change the test order afterwards.
+
+.. note::
+  This option was originally added to allow to run ``pytest-order`` before the
+  ``--failed-first`` option hook. This is no longer needed with current ``pytest``
+  version (see also the next option)
+
+.. _order_after_ff:
+
+``--order-after-ff``
+--------------------
+Using the ``pytest`` option ``--failed-first`` runs failing tests first in the next test run.
+Normally, this is done regardless of the ``order`` markers, because the hook implementing
+this option is executed after ``pytest-order``. Usually, this is what you want. However,
+if the failed test depends on the ordering, this may break the test, so it will continue
+to fail for that reason. For such cases, you can instruct ``pytest-order`` to run after
+that hook using the ``--order-after-ff`` option. This will also affect similar hooks
+(like ``failed-last``) or other ordering plugins that try to run last.
 
 .. _sparse-ordering:
 
 ``--sparse-ordering``
 ---------------------
 Ordering tests by ordinals where some numbers are missing by default behaves
-the same as if the the ordinals are consecutive. For example, these tests:
+the same as if the ordinals were consecutive. For example, these tests:
 
 .. code:: python
 
@@ -605,11 +614,11 @@ as it is without this option.
 
 ``--error-on-failed-ordering``
 ------------------------------
-Relative ordering of tests my fail under some circumstances. Mostly this happens if the related marker
+Relative ordering of tests may fail under some circumstances. Mostly this happens if the related marker
 is not found, or if the tests have a cyclic dependency.
 The default behavior in this case is not to order the test in question, issue a warning during test
 collection and execute the test as usual. If you want to make sure that your relative markers work
-without checking all warning messages, you can also make the tests that cannot be ordered fail, so that
+without having to check all warning messages, you can make the tests that cannot be ordered fail, so that
 they show up as errored in the report:
 
 .. code:: python

src/pytest_order/plugin.py

@@ -1,3 +1,5 @@
+from collections.abc import Generator, Callable
+
 import pytest
 from _pytest.config import Config
 from _pytest.config.argparsing import Parser
@@ -27,12 +29,21 @@ def pytest_configure(config: Config) -> None:
     # only when the CLI option is present should the decorator be added.
     # Thus, we manually run the decorator on the class function and
     # manually replace it.
+    hook_function: Callable[
+        [Session, Config, list[Function]], None | Generator[None]
+    ] = modify_items
     if config.getoption("indulgent_ordering"):
+        # try to run before other plugins
         wrapper = pytest.hookimpl(tryfirst=True)
+    elif config.getoption("order_after_ff"):
+        # run after the LFPlugin plugin, handling --failed-first and --failed-last
+        wrapper = pytest.hookimpl(hookwrapper=True, tryfirst=True)
+        hook_function = modify_items_gen
     else:
+        # try to run after other plugins
         wrapper = pytest.hookimpl(trylast=True)
     OrderingPlugin.pytest_collection_modifyitems = wrapper(  # type:ignore[attr-defined]
-        modify_items
+        hook_function
     )
     config.pluginmanager.register(OrderingPlugin(), "orderingplugin")
 
@@ -115,6 +126,12 @@ def pytest_addoption(parser: Parser) -> None:
             "will error instead of generating only a warning."
         ),
     )
+    group.addoption(
+        "--order-after-ff",
+        action="store_true",
+        dest="order_after_ff",
+        help="If set, the plugin will run after the --failed-first and similar option hooks.",
+    )
 
 
 def _get_mark_description(mark: Mark):
@@ -168,3 +185,11 @@ class OrderingPlugin:
 def modify_items(session: Session, config: Config, items: list[Function]) -> None:
     sorter = Sorter(config, items)
     items[:] = sorter.sort_items()
+
+
+def modify_items_gen(
+    session: Session, config: Config, items: list[Function]
+) -> Generator[None]:
+    yield
+    sorter = Sorter(config, items)
+    items[:] = sorter.sort_items()

tests/plugin_reverse.py

@@ -0,0 +1,3 @@
+# simple plugin that reverts the test order for testing indulgent ordering
+def pytest_collection_modifyitems(config, items):
+    items[:] = items[::-1]

tests/test_indulgent_ordering.py

@@ -0,0 +1,115 @@
+def test_order_after_ff(test_path):
+    test_path.makepyfile(
+        test_failing=(
+            """
+            import pytest
+
+            @pytest.mark.order("second")
+            def test_me_second():
+                assert True
+
+            def test_that_fails():
+                assert False
+
+            def test_after_failed():
+                assert True
+
+            @pytest.mark.order("first")
+            def test_me_first():
+                assert True
+            """
+        )
+    )
+    result = test_path.runpytest("-v")
+    result.assert_outcomes(passed=3, failed=1)
+    result.stdout.fnmatch_lines(
+        [
+            "test_failing.py::test_me_first PASSED",
+            "test_failing.py::test_me_second PASSED",
+            "test_failing.py::test_that_fails FAILED",
+            "test_failing.py::test_after_failed PASSED",
+        ]
+    )
+
+    result = test_path.runpytest("-v", "--ff")
+    result.assert_outcomes(passed=3, failed=1)
+    result.stdout.fnmatch_lines(
+        [
+            "test_failing.py::test_that_fails FAILED",
+            "test_failing.py::test_me_first PASSED",
+            "test_failing.py::test_me_second PASSED",
+            "test_failing.py::test_after_failed PASSED",
+        ]
+    )
+
+    result = test_path.runpytest("-v", "--ff", "--order-after-ff")
+    result.assert_outcomes(passed=3, failed=1)
+    result.stdout.fnmatch_lines(
+        [
+            "test_failing.py::test_me_first PASSED",
+            "test_failing.py::test_me_second PASSED",
+            "test_failing.py::test_that_fails FAILED",
+            "test_failing.py::test_after_failed PASSED",
+        ]
+    )
+
+
+def test_indulgent_ordering(test_path):
+    test_path.makepyfile(
+        test_failing=(
+            """
+            import pytest
+
+            @pytest.mark.order("second")
+            def test_me_second():
+                assert True
+
+            def test_me_third():
+                assert True
+
+            @pytest.mark.order("first")
+            def test_me_first():
+                assert True
+
+            def test_me_fourth():
+                assert True
+            """
+        )
+    )
+    result = test_path.runpytest("-v")
+    result.assert_outcomes(passed=4)
+    result.stdout.fnmatch_lines(
+        [
+            "test_failing.py::test_me_first PASSED",
+            "test_failing.py::test_me_second PASSED",
+            "test_failing.py::test_me_third PASSED",
+            "test_failing.py::test_me_fourth PASSED",
+        ]
+    )
+
+    result = test_path.runpytest("-v", "-p", "tests.plugin_reverse")
+    result.assert_outcomes(passed=4)
+    result.stdout.fnmatch_lines(
+        [
+            "test_failing.py::test_me_first PASSED",
+            "test_failing.py::test_me_second PASSED",
+            "test_failing.py::test_me_fourth PASSED",
+            "test_failing.py::test_me_third PASSED",
+        ]
+    )
+
+    result = test_path.runpytest(
+        "-v",
+        "-p",
+        "tests.plugin_reverse",
+        "--indulgent-ordering",
+    )
+    result.assert_outcomes(passed=4)
+    result.stdout.fnmatch_lines(
+        [
+            "test_failing.py::test_me_fourth PASSED",
+            "test_failing.py::test_me_third PASSED",
+            "test_failing.py::test_me_second PASSED",
+            "test_failing.py::test_me_first PASSED",
+        ]
+    )