Rethink how notebooks work in the repo

[ADBond]

Right now we use notebooks for a few slightly-different things:

• to provide documentation
• for interactive Colab notebooks
• to run tests against

Obviously these are not independent, but they do sometimes conflict. Additionally, it is easy for outputs to get out-of-sync with code, and make diffs/merges a bit awkward.

I'm not sure I have a better solution for now, as rendering all the output is fairly heavy-duty, and doesn't often require updating, but something to ponder

Proposal as outlined in this comment

[ADBond]

I have a partial sketch for this using jupytext. I think I have an idea of reasonable workflows for testing + building docs, but remains to be seen if it actually works properly in practice.

[ADBond]

Here's my proposal, using jupytext as a backing. In this repo we keep notebook sources in percent format -- this means we can run them directly as scripts.

We can then use jupytext to convert & execute the notebooks to restore them to their current state, for use as demos & for docs. I haven't fully decided where these would live, see below for a possibility.

In terms of workflows, we have:

• a 'smoke-test' where we run a subset of notebooks on each PR, as we do now
• periodically (weekly?) we run all notebooks to confirm they run correctly
• we re-render notebooks for output when:
  • the notebook source files change explicitly
  • we do a release

The last point means that most ordinary code changes to Splink won't trigger a re-render of the notebooks, which is time-consuming. I also thing this is probably more correct - in terms of documentation we probably ideally want notebooks that reflect the latest release, not necessarily the current dev version. To that end we could even skip re-rendering on source changes, and only do it on releases - we could still allow manual re-rendering in the case where e.g. someone makes explicit changes to the notebooks that they want reflected.

The .ipynb notebooks

To avoid having to re-render the notebooks on each docs build, we need somewhere to store these, so that most of the time we can just read them. There are a few options, but my current preference is in the splink_demos repo.

splink_demos is where they originally lived. We moved them here (#1431) to keep all of the docs source in one place, simplifying keeping things in sync. This new proposal will not change that - the source is still in this repo, with a version of that reflected in splink_demos. I also note that In the original PR we had intended to keep splink_demos up-to-date with the code here to function as an up-to-date set of demo notebooks, which did not ever end up happening.

Another nice part of this approach is that we keep a history of notebook changes (as we do currently). I don't think it is massively useful, but there may be situations where we would want to easily see how things have changed. It is arguably more useful in its function as a repository for interactive demos - if someone is using an older Splink version, they can checkout the relevant snapshot of demos.

A downside to this is the additional complexity for the docs build. Not such an issue in CI, but does make things more fiddly. But we should be able to mitigate that by wrapping things up in scripts & caching these locally, so hopefully not such an issue. Happy for alternative viewpoints!

[RobinL]

This sounds excellent to me. The only possible hiccup i can see is that some of the notebooks may not run in their entirety in CI since the machine sizes are small and some of them are pretty heavy in terms of data processing. They run in the tests because we slim down the input data and some training params, but i'm not sure a 'full run' would complete.

https://github.com/moj-analytical-services/splink/blob/master/scripts/make_test_datasets_smaller.py
https://github.com/moj-analytical-services/splink/blob/master/scripts/reduce_notebook_runtime.py

I haven't tried this - it might be fine, and is more likely to be fine with Splink 5! But just thought I'd flag.

Whilst this is a possible snag, it's not really any different from the current scenario, where we have to manually run the notebooks and commit the results to get the rich outputs for the docs.

[ADBond]

I'm not sure how the full set run in CI, indeed might be too heavy to do without a bit of fiddling. But the set that we run in PRs currently work pretty promptly without needing to adjust anything (those scripts are still running, but the reduce_notebook_runtime.py has no effect, and I notice that make_test_datasets_smaller.py is in fact not in use since #1431) - on my fork tutorials in 33s and examples in 2m29s.

That's not to say we needn't use any kind of analogous reduction, just that it is not a complete blocker for the time being.

[ADBond]

Another thing that would be useful for the rendering step is trying to make the notebooks as deterministic as possible, so that we will usually have consistent output anyway.