Host multiple versions of documentation simultaneously

[krivard]

Overview

Working on #4822

What problem does this address?

PR #4823 created a gh-pages site that can host one single version of the documentation. our RTD deployment hosts multiple versions simultaneously. We can do that in gh-pages, but we have to do it ourselves. This PR offers one way to do it.

People seem to take two primary approaches to this:

• A workflow runs on each branch to build the docs for its own version, and contributes them piecemeal to gh-pages. Example: pyinfra
  • ➕ good for slow docs builds
  • ➕ good for still making docs available for old unmaintained versions that don't build anymore
  • ➖ lots of moving parts; requires faith that they'll all play nice without interfering with each other
  • ➖ getting the thing populated from scratch could be tricky/annoying
  • ➖ difficult to statically integrate information from other versions (like a dropdown selector to send you to another version)
• A central workflow builds the docs for all versions at once, and wipes out and refills gh-pages each time it is run. Example: rtd-github-pages
  • ➕ monolithic; no delicate interactions between multiple parts
  • ➕ no special initial setup
  • ➕ all available versions are known at build time; easy to statically set up a version dropdown
  • ➖ running time is (number of versions) * (time to build docs for each version) = 18 * 6 = 108 minutes, which is probably too much to run on every merge to main
  • ➖ requires all versions we host be and remain buildable

This PR takes the first approach. It borrows most of the general structure from pyinfra (MIT) and lifts the redirect machinery from rtd-github-pages (GPL3).

What did you change?

• Add a task docs-install to pixi which conditionally places built docs into a separate _install file tree which sorts docs by version. Only installs docs to this tree for the gitrefs we want: main (as "latest"), nightly, stable, and tagged releases.
  • Also drops a manual redirect file in the root of the _install tree which will shuffle folks to latest
• Modify the build-deploy-docs GHA to:
  • use the new docs-install pixi task
  • use the _install tree
  • leave existing files in gh-pages alone
  • no longer force an orphan commit history in gh-pages, aka, allow commits to accumulate in this branch

Intended function:

• each execution of build-deploy-docs will only build the documentation once, for the branch the gha ran on
• if the branch matches a version for which we want to publish docs, the _install tree in the gh-pages branch will get updated files in the directory for that version
• all versions get deployed simultaneously by the pages-build-deployment GHA which is provided by GitHub Pages
• you can browse each version directly by going to docs.catalyst.coop/pudl/en/[version]
• you can visit docs.catalyst.coop/pudl/ and get dropped into docs.catalyst.coop/pudl/en/latest/

Setup cost:

• need to manually seed the _install directory, since our existing tags don't have the pixi machinery on them for the GHA to operate

TODO:

• Figure out if we can send people to other versions with a dropdown or something. The nefertiti theme lets you do this statically; can we do it with a dynamic lookup?
  • Yes! The PyData theme lets us use a JSON file for version lookups which can be hosted anywhere
• Implement version dropdown
  • Decide whether to switch to the PyData theme or try to inject a version dropdown into Furo
• Figure out what happens if two versions are building simultaneously and try to update gh-pages at the same time; does the workflow fail gracefully or does the branch end up mangled?

Documentation

Make sure to update relevant aspects of the documentation:

• Update the release notes: reference the PR and related issues.
• Update relevant Data Source jinja templates (see docs/data_sources/templates).
• Update relevant table or source description metadata (see src/metadata).
• Review and update any other aspects of the documentation that might be affected by this PR.

Testing

How did you make sure this worked? How can a reviewer verify this?

Fork experiments!

• Code: https://github.com/krivard/pudl
• Deployed docs:
  • https://krivard.github.io/pudl
  • https://krivard.github.io/pudl/en/latest/
  • https://krivard.github.io/pudl/en/v2099.9.9/

To-do list

• If updating analyses or data processing functions: make sure to update row count expectations in dbt tests.
• Run make pytest-coverage locally to ensure that the merge queue will accept your PR.
• Review the PR yourself and call out any questions or issues you have.
• For PRs that change the PUDL outputs significantly, run the full ETL locally and then run the data validations using dbt. If you can't run the ETL locally then run the build-deploy-pudl GitHub Action manually and ensure that it succeeds.

[zaneselvans]

• With the switch to using pixi tasks, the Makefile has gone away. The tasks are defined in pyproject.toml. Hopefully this setup is easy to adapt!
• I updated the workflow to deploy any time we merge into main so that the deployment keeps getting tested and so we have fresh docs in GitHub Pages.

[zaneselvans]

• @krivard The pydata theme is a bog standard used across many projects we rely on and Furo is a stripped down version of it, so that seems like a great option if it does the multi-version thing well.
• One of the main differences is that the pydata theme adds a top navbar (in addition to the page TOC on the right, and the multi-page TOC on the left). Initially this was overkill for us, but I think would actually be helpful now and let us cleanly divide up our documentation between e.g:
  • Developer/Contributor documentation
  • (Data) Users oriented documentation
  • Release Notes / Changelog
  • External links to e.g. PUDL Data Viewer