Skip to content

Stabilize the numfig warning test - #14572

Open
DavidAGInnovation wants to merge 2 commits into
sphinx-doc:masterfrom
DavidAGInnovation:agent/numfig-test-stability
Open

Stabilize the numfig warning test#14572
DavidAGInnovation wants to merge 2 commits into
sphinx-doc:masterfrom
DavidAGInnovation:agent/numfig-test-stability

Conversation

@DavidAGInnovation

@DavidAGInnovation DavidAGInnovation commented Jul 31, 2026

Copy link
Copy Markdown

Summary

  • stop sharing the numfig warning test's build result with the other HTML builder tests
  • keep the test isolated so warning assertions do not depend on shared test output

Checks

  • pytest -q tests/test_builders/test_build_html_numfig.py
  • Ruff check and format validation for the changed file

Fixes #14104

@jdillard

jdillard commented Aug 3, 2026

Copy link
Copy Markdown
Member

This follows up on #14100, which intended to isolate test_numfig_disabled_warn but changed the wrong tests and was subsequently reverted in #14106.

It also fixes #14104 and addresses the numfig flakiness mentioned in #13996, but doesn't close that broader issue.

Fix analysis

test_numfig_disabled_warn only checks that a build emits the “numfig is disabled” warning. It was marked with shared_result='test_build_html_numfig', same as the nearby HTML output tests, so they reuse one build for speed.

Unfortunately, the sharing is wrong for a warning assertion. With shared_result, if another test already built (or artifacts exist while the in-memory restore is empty), the app can skip rebuilding. Then app.warning stays empty and the assert fails intermittently under parallel CI, depending on which shared-result test runs the build first.

FAILED tests/test_builders/test_build_html_numfig.py::test_numfig_disabled_warn - AssertionError: assert 'index.rst:47: WARNING: numfig is disabled. :numref: is ignored.' in ''

This PR drops shared_result from that one test so it always does a fresh build and owns its warning stream. The other numfig tests can keep sharing; they only check HTML, not warnings.

@jdillard

jdillard commented Aug 7, 2026

Copy link
Copy Markdown
Member

@picnixz what do you think of this fix?

The test_numfig_disabled_warn flaked on the Docutils HEAD job in #14574 after I merged in the two recent CI fixes. Seems like this PR would help to make CI even greener.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flaky unit test: test_numfig_disabled_warn

2 participants