Jupyter Book plugin

[SimonBiggs]

This is a more permanent discussion location for the building of the Jupyter Book plugin. Original slack thread:

https://pantsbuild.slack.com/archives/C01CQHVDMMW/p1670151030213789

[SimonBiggs]

I've just hit a case where I am likely going to need to be including C++ within my package. As a result I suspect I am going to need to go the Bazel route. Sorry about getting everyone's hopes up regarding me helping out on this front, but I suspect I'll need to go with Bazel.

[ericfeunekes]

No worries @SimonBiggs, appreciate you coming up with the basis for the plugin!

[benjyw]

I believe @sureshjoshi has a ~working C/C++ Pants plugin, FWIW?

[ericfeunekes]

Regarding approach, there are two decision points so far that I've come up with in reviewing the Jupyter Book (JB) API docs. First is whether to create individual targets for each document and second is whether to support publishing sources, and if so, which ones.

My short answers to those questions are: support individual targets, support publishing initially to github pages.

As background, when you run JB it will cache the results and only re-run notebooks/documents that have changed since the last run, similar to pants. So it would be possible to just use that API to manage the cache, rather than building individual doc targets. However, the downside to that is that you miss out on having hermetic builds for each doc. A use case where this could be a problem is in a large book or blog, where you might be bringing lots of different packages for different topics. With only a single book target, all packages need to be installed for every run, which sort of defeats the purpose of pants. The JB API allows for single target builds, so I would lean towards using that.

Re publishing, given that pants supports it, I think there should also be a publishing target. I suggest GH Pages first mostly because that's what I use, although it should also be the easiest. Just need to create a rule that checks out some other branch and runs the JB build commands there, so that the HTML gets put into the gh-pages branch (or wherever you define).

JB also supports Netlify and ReadTheDocs, which likely wouldn't be too hard to add.

[Eric-Arellano]

support individual targets

Another benefit of individual targets is that you'll have less friction with CLI specs for what to run on. Most of Pants's goals are target based, and we map the specs like ./pants $goal path/to:tgt path/to/dir path/to/dir/f.ipynb to the matching targets, and then pass those targets to each of the goal's implementations. In other words, the implementations don't keep track of what the original specs were, only what targets were matched. So, if foo.ipynb resolves to a target that "owns" 5 notebooks, then we would end up running on all 5 notebooks even when the user says ./pants $goal foo.ipynb. There is a way to work around that, and we used to do that from 1.25 - 2.7, but it would require a lot of work and complexity to restore.

[kaos]

the implementations don't keep track of what the original specs were, only what targets were matched.

Aren't the target generators the answer to this. So you can have a single target in the build file, owning 5 notebooks, but operate on them individually (as generated targets)

Also, I think it could be a good exercise to work from the assumption of no targets, and then add them in as required rather than the other way around; if that makes sense :)

[Eric-Arellano]

Aren't the target generators the answer to this. So you can have a single target in the build file, owning 5 notebooks, but operate on them individually (as generated targets)

Yep, that would work too if we found a) the boilerplate of one explicit target per notebook offensive, and b) that the notebooks tend to share common explicitly set metadata. The key detail with target generators is we still have the "atom" targets as one target per notebook; the target generator is solely for boilerplate reduction.

[ericfeunekes]

The only thing I'd want to confirm is that JB can build an entire book and then update individual pages by atomically building each page. I don't see any blockers but still need to confirm.

Also can targets be language agnostic? For example if I have 5 pages, 2 ipynb and 3 md, can I have one target generator? Or do I need two? My guess is two, and that should be fine since it still reduces boilerplate. But just something to be aware of when building.

[Eric-Arellano]

Also can targets be language agnostic? For example if I have 5 pages, 2 ipynb and 3 md, can I have one target generator? Or do I need two? My guess is two, and that should be fine since it still reduces boilerplate. But just something to be aware of when building.

Yeah that should be fine, jupyter_page would be set to work with either .ipynb or .md. Then you can have your jupyter_pages or jupyter_book (i.e. the target generator) include both the .ipynb and .md files in its sources. You'd want to be careful with the default sources for that target generator—imo it probably shouldn't include .md by default because there are many MD files that aren't Jupyter pages. But users can explicitly set the sources.

[benjyw]

See related issue: #17822

[ericfeunekes]

First is whether to create individual targets for each document

I've been working on the plugin and it looks like jupyter-book doesn't allow individual targets. There are a number of different build outputs but from what I can tell you can't build individual files. Rather, jupyter-book wants to handle all files at once. That said, it does keep a cache of previously built pages. Therefore, I'm working on building the plugin to call jupyter-book on the whole book folder and then save the cache.

A couple other updates:

1. I'm currently using ghp-import in a rule to push the built html docs to their own branch. This is likely functionality that can be put in its own rule to reduce dependencies
2. The PexProcess that runs jupyter-book is having trouble with the jupyter kernel in the pex that is created. I won't go into a detailed description because I'm not too familiar with the issue. However, I got jupyter-book working outside of pants by running pex $(pip freeze) -c jupyter-book -o jb.pex and then running ./jb.pex build book (where book is a directory with jupyter-book files). So I think I just need to get the PexProcess to use the -c flag. My understanding is that I would do that by putting `main=ConsoleScript("jupyter-book") in the "PexProcess"; but that's what I did and still came to the error that I have now, so I'll have to keep digging on this issue.

[ericfeunekes]

I found the resolution to item 2 above. It is because I'm using asdf and needed to set the python search path to search_path = ["<ASDF_LOCAL>"] and not include <ASDF>. Essentially, I think that pants was finding different python distributions to use and trying to combine them. Once I got those settings right then it worked.

[ericfeunekes]

Update on MVP 1.0 implementation

The jupyter-book implementation mvp uses a single target, which points to the directory holding the jupyter-book docs. It passes the directory to a Digest, which jupyter-book runs on. As described above, I haven't been able to make individual sources build well and so decided to let jupyter-book handle most of the work, including caching.

Points to consider for improvement:

• Currently only supports jupyter-book build. Should it support other commands (e.g. create)
• Currently support github pages as a publish target by specifying the branch you want to publish too (subject to some debugging). Should publish targets be supported? Or should they be separate targets?
• I've only done very limited testing with jupyter notebooks and there seems to a kernel related issue that constantly comes up (see debugging for more). This is some of the main functionality, so I want to make sure it works very well.

That's all I can think about for now re improvements

Debugging

• Having trouble getting the plugin installed from pants.toml to be used in the sandbox. I can reference the mermaid plugin as an extra requirement but when running jupyter-book doesn't build the diagram.
• The jupyter kernel issue is a big problem. I thought I'd figured it out by setting a search path to ASDF, but that doesn't work in a new project that I've set up. Still a work in progress. Currently I'm thinking it could have to do with the fact that I can't get pants to generate a lockfile for the environment, so there could be some other underlying issue.

Testing

I setup the plugin in my projects repo. If anyone feels brave, please fork it and test out the plugin. Looking for thoughts/suggestions before moving to the pants repo and adding tests, etc.

[ericfeunekes]

Two updates.

First, with @jsirois help I resolved the jupyter kernel issue that had been plaguing this plugin by switching from a PexProcess to a VenvPexProcess. With that change the plugin meets all requirements that I see in this thread.

Second, I realized that there is a better way to build the targets for the plugin. Rather than have one target that points at a single docs folder, the plugin should use individual source targets, which would bring all the targetted docs into a single digest for jupyter-book to run. So the docs are still being built by a single jupyter book process, but the individual targets allow for more flexibility in spreading documentation around a repo. So for now I'm going to work on refactoring the plugin to use this approach.

A few things I've thought about:

1. There should be part, chapter, and section targets (see jupyter-book docs) to allow organization. These would be called organization targets
2. The source target will likely be called doc. You could then add the organization target as a dependency to the doc.
3. The organization and doc targets would have a prefix field.
4. Items 1-3 allow us to generate a Table of Contents from content files. And the generated toc uses alphabetical sorting, the prefix field allows you to specify the order of your content.
5. One thing is that the distributed docs and organization could get difficult to manage, so it would be good to allow simpler management too. I think the config options should allow you to specify the location of a toc, in which case that one will be used.
6. The digest will be created by first grabbing the folder specified by the directory field (it must exist, since at a minimum you need a _config.yaml file. Then it will add folders for the organizational targets and either build a toc or use the existing one to run jupyter-book on the whole digest.

A future version of this plugin should also parse python_sources to generate api docs according to these instructions

[kaos]

To me doc feels slightly too generic. What if there are competing documentation backends (for different languages maybe..)
I'd suggest considering the slightly longer jupyter_doc etc.. or anything along those lines to help avoid future naming conflicts.

[ericfeunekes]

Good point. jupyter_ is likely also the wrong prefix since it could be confused with something having to do with jupyter notebooks.

What about prefixing everything with jupyter_book_; eg. jupyter_book as the base target, then jupyter_book_chapter, jupyter_book_file, etc.

[kaos]

Ah, yea that sounds good. Shows I'm not in the Jupyter space.. :P