Switch to GitHub Pages for hosting the documentation

[agriyakhetarpal]

As stated in the issue title, I'd like to propose moving the hosted documentation away from Read the Docs and using GitHub Pages instead for hosting. This was last discussed in #706 (comment) and, probably, a few times elsewhere.

Some of the advantages and reasons for this suggestion are as follows:

• We receive greater control over the build process and we can modify it as we like, instead of having to configure extra build steps we might end up needing in a .readthedocs.yaml file
• For enabling the rest of the interactive documentation for PyWavelets on latest versions, it is then possible to embed a custom Pyodide distribution, which seems to me should be the best approach for reusing the docs. I've previously tried a similar approach, but this new one that I propose should work better, since it would work without the need for nightly WASM wheels.
• One less GitHub App to configure for this repository/organisation and fewer finicky permissions to handle.
• and so on

Some things to be thought about:

• PyWavelets is a package that has been present for almost nineteen years now, and a lot of links will get subjected to link rot if we don't preserve the style of the Read the Docs links, i.e., to link to the "stable" documentation and "latest" documentation under /en/stable/ and /en/latest/ respectively. Setting redirects for broken links is possible under the Read the Docs admin dashboard, but it's difficult to do the same thing on GitHub Pages.
• How to best navigate "stable" and "latest" documentation updates on GitHub Pages. For the latter, it would be through a GitHub Actions job that gets triggered on pushes to the master branch and uses a fine-grained PAT to update the rebuilt docs in the gh-pages branch for a PyWavelets/docs repository
• The features enabled via the Read the Docs add-ons will be lost. Not a big bummer, though, considering we host only the "latest" and "stable" versions and the version switcher has been the most useful one, IO. The version warning on an "unstable" version can be natively re-implemented using admonitions by customising the conf.py file.
• and more things I might have missed at the time of writing this issue.

[rgommers]

In principle I'm perfectly fine with migrating to GitHub Pages. We don't need the multi-version support that RTD provides, and we also don't need a latest/stable split but can always just deploy the latest dev version of the docs. New features are rare and will anyway be marked with a .. versionadded:: directive.

Don't worry about the redirects either, we can quite easily set those up even with URL rewriting that would (for example) drop the stable/ part of each link (I've done it for meson-python recently).

For enabling the rest of the interactive documentation for PyWavelets on latest versions, it is then possible to embed a custom Pyodide distribution, which seems to me should be the best approach for reusing the docs. I've previously tried a similar approach, but this new one that I propose should work better, since it would work without the need for nightly WASM wheels.

This is the key part, let's focus on that. I think it's a separate issue, one of the existing ones that deals with nightly wheels probably. I doubt we want to be in the business of hosting a whole distribution, since that doesn't scale very well. I'm still not 100% sure how the recent work by @Carreau et al. gets deployed to get us an up-to-date and working set of packages when loading JupyterLite inside the PyWavelets docs.

[Carreau]

I'm also in favor of trying to find a solution that would favor nightly wheels instead of building a whole distribution; It in general better for the ecosystem as it's more flexible and reusable across projects. I'll try to investigate how this ties into JupyterLite.

I think we should also be able to build the wasm32 wheel on RTD as well if we don't want to go upload the wheel to the scientific Python nightly.

[rgommers]

I think we should also be able to build the wasm32 wheel on RTD as well if we don't want to go upload the wheel to the scientific Python nightly.

We're already uploading the nightly wheels to https://anaconda.org/scientific-python-nightly-wheels/pywavelets and don't plan to stop, so I think we're fine in that respect.

[Carreau]

We're already uploading the nightly wheels to anaconda.org/scientific-python-nightly-wheels/pywavelets and don't plan to stop, so I think we're fine in that respect.

Oh, yeah, I missed that; then we shoudl just pull from there indeed.

[Carreau]

Meh, I there is a bug in micorpip that actually mark the wasm32 wheels as incompatible...

[Carreau]

Ok, so forcing micropip to accept *_wasm32 works

I think that's because emscripten_3_1_58_wasm32 and pyodide_2024_0_wasm32 are considered different platforms, but are compatible ? Am I wrong ?

[agriyakhetarpal]

Both of them are supposed to be the same platform actually – the rename from the Emscripten version to Pyodide + Year + build number in the platform tag was just to indicate that the built wheels are meant to work with the Pyodide distribution rather than purely wheels that are compiled with Emscripten/for WASM.

[agriyakhetarpal]

It might be a bug, micropip should accept both of them

[agriyakhetarpal]

I'll try to investigate how this ties into JupyterLite.

I think we should also be able to build the wasm32 wheel on RTD as well if we don't want to go upload the wheel to the scientific Python nightly.

With JupyterLite, the way forward would be that we use https://jupyterlite.readthedocs.io/en/stable/howto/pyodide/packages.html#installing-packages-at-runtime to install the nightly wheels with a new piplite command (and maybe modify jupyterlite-sphinx to insert that command into a cell automatically). But I think the neater way would be that we do build the wasm32 wheel with RTD (or switch to GitHub Pages like with this proposal), so that we have it installed beforehand (https://jupyterlite.readthedocs.io/en/stable/howto/pyodide/packages.html#bundling-additional-packages-by-default) since the issue to hide a particular cell in a notebook from execution—in this case, the cell at the top of the notebook which would install the nightly wheels—hasn't received much traction: jupyterlite/jupyterlite#508.

A custom Pyodide distribution with GitHub Pages would not scale very well, but should be fine for a package like PyWavelets.

[Carreau]

Ok, digging into it more, I think the right place to investigate is pyodide-kernel, per jupyterlite-docs this should be the kernel that matters.

worker.ts seem the be the entry point that gets called for both a same-process or in-worker kernels; It itself has options to load packages and url - but hardcoded for now; I think the reasonable things to do is to thread here an extra-option that install packages / url.

The Pyodide kernel itself is created here; which is where data is read from jupyter-lite.json,
so it should be enough to thread the options from there and update the right interfaces.

I'll try to implement that – but so far I'm struggling a bit with having a dev install that consistently reflect my changes. I'm likley doing something wrong with the dev installes of pyodide kernel.