Sphinx 8 vs sphinx 9 compatibility

[sydduckworth]

In implementing type hints (#2031) I learned that Sphinx 9+ is only available for Python versions >= 3.12 and older Python versions use Sphinx 8.
The problem is that Sphinx completely reworked how autodoc works, which significantly changes the behavior with respect to type hints.
Since asdf supports Python >=3.10, depending on which Python version you use you will encounter completely different Sphinx behavior.

Right now running the doc generation code with older Python versions fails because sphinx 8 doesn't correctly handle imports gated behind TYPE_CHECKING blocks (among other reasons).
I was originally using the sphinx-autodoc-typehints plugin to resolve these issues but having that plugin enabled in Sphinx 9 causes a whole set of new problems. Additionally, regardless of version it seems to break cross-references for type aliases.

Potential Solutions

• Move the doc generation code into its own unpublished package (in the same repo), set its required Python version to >=3.12, and sphinx >=9.0. To me this seems like the best solution because it allows us to use the latest Sphinx version and it doesn't impact the main package.
• Update the sphinx config to enable autodoc_use_legacy_class_based, which forces sphinx 9+ to use the old autodoc implementation. This would ensure parity across Python versions, but it would also break type hints in the ways described above.
• Change nothing, and just add a note for developers that you need to be using Python >=3.12 in order to build the documentation.

[braingram]

The autodoc_use_legacy_class_based mentioned in option 2 didn't work as intended for other sphinx issues so I'd hesitate trying to use that at all.

I'm curious what option 1 would look like. It already seems odd that we have a "docs" extra which ends up in the package on pypi but the package on pypi doesn't install with the docs.

In my mind the docs only need to be built:

• on rtd (where the python version comes from the environment file)
• by developers (who would be best off using the same version as rtd)

Are there other cases where we'd expect building the docs would be helpful?

@zacharyburnett was recently doing some cleanup to rtd configuration (for example: spacetelescope/jwst#10442) so his feedback is probably more useful here.

[zacharyburnett]

previously we had issues with availability of Sphinx on conda-forge, so I changed the RTD build to install graphviz from apt instead. I made a similar PR here that may or may not help; #2037

it looks like Sphinx 9 requires Python >=3.11: https://github.com/sphinx-doc/sphinx/blob/aea90c92aadb186b82e96c2b7ad6f2b865f70df1/pyproject.toml#L20, so maybe this is only a problem for Python 3.10?

[sydduckworth]

Sphinx 9.0 requires Python >=3.11 but Sphinx 9.1 and above requires Python >=3.12

[sydduckworth]

@braingram if we split the docs into their own package how do we feel about using uv?

The package would have a local dependency on asdf, which is a little finicky in Python. With uv it's relatively straightforward:

# pyproject.toml

dependencies = ["asdf"]

[tool.uv.sources]
asdf = { path = "..", editable = true }

# .readthedocs.yaml

python:
  install:
    - method: uv
      path: docs

Can also use uv's workspaces feature which accomplishes the same thing.

[zacharyburnett]

how does the workspaces feature work exactly? I've been meaning to try it out; does it require the dev to have all of the packages cloned locally?

[sydduckworth]

Its supposed to work like Cargo workspaces, so its really only for packages that are all within a single repo.

For this scenario the first option seems better than using workspaces since workspaces are really a uv-specific thing while overriding package sources is supported by multiple package managers (although not in a standardized way).