ipypublish and jupytext

[parmentelat]

Is your feature request related to a problem? Please describe.

my use case is pretty simple
I write notebooks for teaching, mostly rise-powered slideshows
I also use ipypublish to produce handouts on readthedocs
recently I have decided to go for jupytext, it is so much simpler for me to store my contents under git as either .py or .md files
however I have not been able to keep on using ipypublish after moving to jupytext

truth be said :

• I have not touched the few repos where that were working fine, I was trying to apply the same recipe that works for me for example from this git repo to this readthedocs result
• so I don't know for sure if that is even supported, nor that I've done everything right, or if I just misconfigured something somewhere

Describe the solution you'd like

first off I'd like to know if that is something that is supposed to be easy to get to work, or not

Describe alternatives you've considered

I have given a shot at nbsphinx, with which I was more successful wrt jupytext (I guess I found the recipe for binding .py extension) but am running into problems of other kinds (images for example seem to be a struggle; also I had created a custom thingy for having the slide notes showup as a custom class); it's be much easier for me to be able to stick to one solution for this need; it's rather crucial, but otoh the whole point would be ideally to not have to worry about this sort of things to much ;-)

Additional context
I have not kept track of my early attempts, that I scratched when going to nbspinx if relevant I can put that back in shape and provide more details

as an aside, and more generally, I have to say that I am quite confused by the wide spectrum of possibilities for doing this sort of things (ipypublish, nbsphinx, jupyter-book ?, this new myst-nb thing I've [come across recently from here]), that are intimidating b/c of number of options, the pace of change, etc.. if anyone has the time and knowledge to write a short overview on all these within a bit of context, it would honestly be tremendously useful IMHO

thanks in any case for the great tool :)

[chrisjsewell]

as an aside, and more generally, I have to say that I am quite confused by the wide spectrum of possibilities for doing this sort of things (ipypublish, nbsphinx, jupyter-book ?, this new myst-nb thing I've [come across recently from here])

Thanks for the feedback @parmentelat. I'm glad you mention this, if you have a look at the contributors to MyST-NB, you may note that the mains ones are myself and @choldgraf (the maintainer of jupyter-book) 😁

The Executable Book Project is essentially a collaboration between ipypublish, jupyter-book and jupinx to take the best bits of our packages and combine them. This project is only a few months old, but we are looking to make a beta release soon,
with the main entry point being jupyter-book 2.0: https://beta.jupyterbook.org/intro.html

Note with the new package(s) we are specifically collaborating with jupytext and considering the use of .md files:

• https://jupytext.readthedocs.io/en/latest/formats.html#myst-markdown
• https://beta.jupyterbook.org/use/myst-notebooks.html

See spatialaudio/nbsphinx#420 for some outline/debate about the differences with nbsphinx, which has some overlapping but somewhat different goals.

Also note at some point, once I'm happy that all the relevant functionality from ipypublish is available in this new package, I imagine I will probably look to "deprecate" ipypublish in some fashion.

So yes it would be great if you have the time to check that out and give any feedback on any improvements you would desire to meet your use case 😄

[choldgraf]

hey @parmentelat - echoing what Chris mentioned - our goal is to (somewhat) reduce the forking-paths problem by unifying ipypublish/jupyter-book/myst-nb (with jupyter-book being the most user-facing of the three, and myst-nb being used "under the hood" or also used in "regular" Sphinx documentation.

Check out beta.jupyterbook.org, and please do let us know if you have questions and such!

[mgeier]

@parmentelat Author of nbsphinx here!

[...] but am running into problems of other kinds (images for example seem to be a struggle; also I had created a custom thingy for having the slide notes showup as a custom class)

It would be great if you could report your problems at https://github.com/spatialaudio/nbsphinx/issues!

[parmentelat]

hiya everyone

thanks for the feedback; I'll be looking into this new option, this looks promising; would anyone care to confirm what I think I gathered after this exchange

• ipypublish is currently no longer actively supported/maintained so I'd be better off picking something else on mid term
• jupyter-book currently has some provision to read jupytext-based notebooks but at first sight it looks limited to one specific subflavour of .md, is that right ?
• if so, but this is just me being curious, at first it felt like supporting jupytext - any jupytext format - was simple enough, but somehow at this point it no longer looks that way; why is that ? I'm asking because I had just transformed all my input material from .ipynb to .py, if only I'd known ;)

thanks again for any additional tip; I'll report here when I have more hands-on feedback with jupyter-book

[choldgraf]

Quick response from me:

ipypublish is currently no longer actively supported/maintained so I'd be better off picking something else on mid term

That's @chrisjsewell's call, but from my perspective, many of the ideas in ipypublish have now made their way into the new MyST-NB/MyST-parser/jupyter-book stack, and I think that stack will get more development attention moving forward.

jupyter-book currently has some provision to read jupytext-based notebooks but at first sight it looks limited to one specific subflavour of .md, is that right ?

Right now that is the case...some functionality was reduced because of the new build system but we could extend it to support some other use-cases, just want to do it intentionally!

if so, but this is just me being curious, at first it felt like supporting jupytext - any jupytext format - was simple enough, but somehow at this point it no longer looks that way; why is that ? I'm asking because I had just transformed all my input material from .ipynb to .py, if only I'd known ;)

Wanna open an issue about this in MyST-NB? To me it seems reasonable to support jupytext .py files as well as .md files, but it may take some discussion from the community.

[parmentelat]

Wanna open an issue about this in MyST-NB? To me it seems reasonable to support jupytext .py files as well as .md files, but it may take some discussion from the community.

right; I'll do that, thank you

[chrisjsewell]

ipypublish is currently no longer actively supported/maintained so I'd be better off picking something else on mid term

I'm still happy to accept/merge PRs and look into "minor" bugfixes. But obviously there is only finite time in the day lol, so I don't intend to do anything more substantial with ipypublish.

at first it felt like supporting jupytext - any jupytext format - was simple enough

Yeh its something we can discuss further on MyST-NB. In lieu of direct support is this not just a matter of calling jupytext --from py:percent --to myst *.py though?

[choldgraf]

Good point @chrisjsewell - as a stopgap, I added a section on converting files with jupytext here: https://beta.jupyterbook.org/content-types/jupytext.html

[parmentelat]

Yeh its something we can discuss further on MyST-NB. In lieu of direct support is this not just a matter of calling jupytext --from py:percent --to myst *.py though?

well, one could make the same answer for executing the notebooks with nbconvert ;)
my point is, when publishing on readthedocs, the deal is to just drop the files in github and let the whole magic happen for you without the need to fuss with any kind of preprocessing, and this truly has real value indeed :)

[choldgraf]

@parmentelat ideally we'd like this to be possible with many different build engines, not just readthedocs. e.g., see https://github.com/ExecutableBookProject/cli/issues/44 and what will hopefully be a growing set of subsections here: https://beta.jupyterbook.org/start/publish.html

[phaustin]

Related to #133 (comment) -- we just converted stack of jupytext py files to myst, and the one gotcha is that the "language_info" metadata field is inferred when you're using py:percent, but if it's missing in a myst:md file sphinx raises a key_error. So I needed to go through and add the language_info to each of the converted myst notebooks.