0.5.0: pytest is failing in `tests/test_render.py::test_sphinx_build_directives` unit

[kloczek]

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix> using 'installer` module
• run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-sphinx-autodoc2-0.5.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-sphinx-autodoc2-0.5.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
============================= test session starts ==============================
platform linux -- Python 3.8.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/tkloczko/rpmbuild/BUILD/sphinx-autodoc2-0.5.0
configfile: pyproject.toml
testpaths: tests
plugins: datadir-1.5.0, regressions-2.5.0
collected 45 items

tests/test_analyse_module.py .....................................       [ 82%]
tests/test_database.py .                                                 [ 84%]
tests/test_render.py ......F                                             [100%]

=================================== FAILURES ===================================
_________________________ test_sphinx_build_directives _________________________

tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-10/test_sphinx_build_directives0')
file_regression = <pytest_regressions.file_regression.FileRegressionFixture object at 0x7fab683055e0>

    def test_sphinx_build_directives(tmp_path: Path, file_regression):
        """Test building the Sphinx docs, using directives."""
        build_package(tmp_path)
        source = tmp_path / "source"
        source.mkdir()
        source.joinpath("conf.py").write_text(
            dedent(
                """\
            project = "tester"
            extensions = ["autodoc2"]
            autodoc2_packages = [
                {
                    "path": "../package",
                    "auto_mode": False,
                }
            ]
            """
            ),
            "utf-8",
        )
        source.joinpath("index.rst").write_text(
            dedent(
                """\
            Test
            ====

            .. autodoc2-docstring:: package.func
               :literal:
               :literal-linenos:
               :literal-lexer: restructuredtext

            .. autodoc2-docstring:: package.func

            .. autodoc2-object:: package.func
               :literal:
               :literal-lexer: restructuredtext

            .. autodoc2-object:: package.func
               :literal:

               render_plugin = "myst"

            .. autodoc2-object:: package.func

            .. autodoc2-summary::

                package.func
                package.a1
            """
            ),
            "utf-8",
        )
        warnings = io.StringIO()
        build = tmp_path / "build"
        app = SphinxTestApp(
            buildername="html",
            srcdir=sphinx_path(source),
            builddir=sphinx_path(build),
            warning=warnings,
        )
        try:
            app.build()
        finally:
            app.cleanup()

        assert not warnings.getvalue()

        doctree = app.env.get_doctree("index")
        doctree["source"] = "index.rst"
        content = "\n".join([line.rstrip() for line in doctree.pformat().splitlines()])
>       file_regression.check(content, extension=".xml")
E       AssertionError: FILES DIFFER:
E       /tmp/pytest-of-tkloczko/pytest-10/test_sphinx_build_directives0/test_render/test_sphinx_build_directives.xml
E       /tmp/pytest-of-tkloczko/pytest-10/test_sphinx_build_directives0/test_render/test_sphinx_build_directives.obtained.xml
E       HTML DIFF: /tmp/pytest-of-tkloczko/pytest-10/test_sphinx_build_directives0/test_render/test_sphinx_build_directives.obtained.diff.html
E       ---
E       +++
E       @@ -1,4 +1,4 @@
E       -<document source="index.rst">
E       +<document source="index.rst" translation_progress="{'total': 0, 'translated': 0}">
E            <section ids="test" names="test">
E                <title>
E                    Test
E       @@ -25,7 +25,7 @@
E                            package.
E                        <desc_name classes="sig-name descname" xml:space="preserve">
E                            func
E       -                <desc_parameterlist xml:space="preserve">
E       +                <desc_parameterlist multi_line_parameter_list="False" xml:space="preserve">
E                            <desc_parameter xml:space="preserve">
E                                <desc_sig_name classes="n">
E                                    a

tests/test_render.py:226: AssertionError
=========================== short test summary info ============================
FAILED tests/test_render.py::test_sphinx_build_directives - AssertionError: F...
========================= 1 failed, 44 passed in 2.27s =========================

List of installed modules in build env:

Package                       Version
----------------------------- -------
alabaster                     0.7.13
astroid                       3.0.1
Babel                         2.13.1
build                         1.0.3
charset-normalizer            3.3.2
cppclean                      0.13
distro                        1.8.0
dnf                           4.18.1
docutils                      0.20.1
exceptiongroup                1.1.3
gpg                           1.23.0
idna                          3.4
imagesize                     1.4.1
importlib-metadata            6.8.0
iniconfig                     2.0.0
installer                     0.7.0
Jinja2                        3.1.2
libdnf                        0.72.0
markdown-it-py                3.0.0
MarkupSafe                    2.1.3
mdit-py-plugins               0.4.0
mdurl                         0.1.2
myst-parser                   2.0.0
packaging                     23.2
pluggy                        1.3.0
Pygments                      2.17.2
pyproject_hooks               1.0.0
pytest                        7.4.3
pytest-datadir                1.5.0
pytest-regressions            2.5.0
python-dateutil               2.8.2
pytz                          2023.3
PyYAML                        6.0.1
requests                      2.31.0
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        7.1.2
sphinxcontrib-applehelp       1.0.4
sphinxcontrib-devhelp         1.0.5
sphinxcontrib-htmlhelp        2.0.4
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.9
tomli                         2.0.1
typing_extensions             4.8.0
urllib3                       1.26.18
wheel                         0.42.0
zipp                          3.17.0

[kloczek]

Just retested wit latest pytest 8.1.1 and sphinx 7.2.1 and now pytest fails like below

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-sphinx-autodoc2-0.5.0-2.fc36.x86_64/usr/lib64/python3.9/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-sphinx-autodoc2-0.5.0-2.fc36.x86_64/usr/lib/python3.9/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/tkloczko/rpmbuild/BUILD/sphinx-autodoc2-0.5.0
configfile: pyproject.toml
testpaths: tests
plugins: mock-3.12.0, hypothesis-6.99.5, datadir-1.5.0, regressions-2.5.0
collected 38 items / 1 error

========================================================================================== ERRORS ===========================================================================================
___________________________________________________________________________ ERROR collecting tests/test_render.py ___________________________________________________________________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/sphinx-autodoc2-0.5.0/tests/test_render.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib64/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_render.py:15: in <module>
    from sphinx.testing.util import path as sphinx_path
E   ImportError: cannot import name 'path' from 'sphinx.testing.util' (/usr/lib/python3.9/site-packages/sphinx/testing/util.py)
================================================================================== short test summary info ==================================================================================
ERROR tests/test_render.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================================================== 1 error in 0.99s ======================================================================================

After add tests/test_render.py to --ignore list rest of the test suite seems is OK.

[ppfeister]

sphinx = ["sphinx>=4.0.0"]
testing = [
    "pytest",
    "pytest-regressions",
    "pytest-cov",
    "sphinx>=4.0.0,<7",
]

autodoc proper depends on Sphinx >= 4.0.0, but the test dependency has a tighter bound, also being < 7. sphinx.testing.util doesn't export path as of that version, and the module where it itself imported it from no longer supports the way it's being used here.

Ignoring doesn't seem to work in my case, as other tests begin to fail when doing so.

[kloczek]

At the moment use anything than >=7 does not make any sense because any older version of the sphinx is no longer maintained.

[ppfeister]

I agree. This is a dependency for another rpm I need to package, so I'd like to see a real resolution of some sort rather than using an old version. Hoping to have some time toon to work on a patch, but it sure doesn't look fun after reading the test src