Speed up loading of REMARKs using JupyterHub or alternative

[shaunagm]

Update: Google Colab is looking very promising. Next steps:

• SGM tries example with full LaTex - how much does it slow down loading?
• check all notebooks work in Colab (will need to at least add import statements) (SGM can do this by forking REMARKs repo)

If we decide on Colab, we will then:

• update how we direct people on the website
• reformat REMARKs to remove now-unnecessary mybinder structure

---

Currently, mybinder takes several minutes to load our remarks and other notebooks. We'd like anyone loading a notebook from the econARK site to be able to access it quickly, definitely in under 30 seconds and ideally faster. Not sure whether we need special hosting, caching, etc to make this happen. Relatedly, our current setup causes issues with dependency management (see issue #12).

Use cases:

• giving a presentation or sharing with a colleague, when you need to launch your own notebook quickly and interactively
• teaching a workshop, with dozens of individuals needing to interact with a notebook
• random visitors to the website

The above use cases have different limitations. A lot of platforms I've investigated have access control such that it would be hard to provide this service to anonymous internet people but easy enough for anyone we approve of and who can spare a few minutes of initial setup. It's possible we could set up a two tiered system where community members (and the students/workshop attendees of community members) can launch notebooks quickly and internet strangers continue to use the mybinder system. (Although this doesn't address the dependency management aspect, just the performance aspect.)

Potential approaches:

1. find a hosting platform that allows us to do this (trying this first, since it's the easier option; see list below)
2. set up our own hosting system, perhaps on AWS (OTS did this for us briefly last year, for a specific event)

People to consult:

• OTS
• QuantEcon (Though they appear to use the same mybinder.org-based solution that we currently do, with corresponding 30 second launch times. What did they customize?)

Hosting Platforms & Notes

• Google Colab
• CodeOcean - major roadblock here is that you have to login to run notebooks interactively

[shaunagm]

@llorracc, @mnwhite - I'm going to check out Google Colab sometime in the next week or so, using one or two of the existing notebooks as tests. Are they all about the same in terms of computational power needed? If not, can you suggest a notebook that's on the computation-heavy end, but not an outlier?

[llorracc]

Shauna, The BufferStockTheory.ipynb REMARK notebook is a good test case. The objective here, for workshops etc, is a bit different from the other case where we wanted computational power. What I'm hoping to find for ordinary workshops and notebooks used therein is a way to speed up the initial launching of the notebook. Especially in cases where the notebook has a lot of setup stuff. The BufferStockTheory.ipynb notebook checks to see whether "latex" has been installed on the VM and if so changes some configuration stuff. Unfortunately, it can take 2-4 minutes for the remote VM to FINALLY "go live" because of all of the prep software installation that has to be done. BufferStockTheory provides a good example of a case where latex is preferred (and used if the tools necessary are available).

…

On Wed, Mar 20, 2019 at 2:46 PM Shauna ***@***.***> wrote: @llorracc <https://github.com/llorracc>, @mnwhite <https://github.com/mnwhite> - I'm going to check out Google Colab sometime in the next week or so, using one or two of the existing notebooks as tests. Are they all about the same in terms of computational power needed? If not, can you suggest a notebook that's on the computation-heavy end, but not an outlier? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQlf7yxnrx-M4U7Q4xwaeVHs2iyxdufks5vYju1gaJpZM4bX937> .

-- - Chris Carroll

[shaunagm]

I'm liking Google CoLab so far. Loading a notebook stored in github was as simple as using the customized url which takes the format:

colab.research.google.com/github/$our_organization_name/$repository_name/blob/master/$relative_path_to_notebook.ipynb

It loaded fairly fast, about 10 seconds by my count, and it looks like the Latex is all there.

The hosted runtime options seem fairly limited - only Python 2.7 and 3.6 are options, and they come with certain libraries pre-installed, which means we don't have as much flexibility in choose the environment we want the notebooks to run in. I think that means we're stuck with a little cell at the top of all our notebooks that looks something like:

!pip install econ-ark
!pip install matplotlib==1.2  # made up example for if we need to use a different version

But the notebooks currently have set-up cells anyway, so. Anyway, here's a list of issues I encountered and their solutions:

1. TkAgg error

ImportError: Cannot load backend 'TkAgg' which requires the 'tk' interactive framework, as 'headless' is currently running

This is due to trying to load get_ipython().run_line_magic('matplotlib', 'auto') instead of get_ipython().run_line_magic('matplotlib', 'inline') (inline is the only matplotlib backend supported on CoLab) (no I don't know what that means). This should be easily fixable by rejiggering that if/else statement to catch the CoLab environment in the if, but for now I just replace 'auto' with 'inline' manually.

2. Import HARK error

As expected I then got told that HARK didn't exist, and added !pip install econ-ark to the top of the first cell.

3. Latex display issue

In cell 25 the latex on lines 22-26 (I think, annoyingly it doesn't give line numbers) is causing an error. I deleted the four lines and it ran fine but without the necessary output. I don't know what's going on well enough to fix it. Maybe the Latex didn't load as well as I thought? Same/similar issue with the last code cell.

Anyway, that was surprisingly straightforward, but I'm also not familiar enough with the notebook to know if there's content errors or missing pieces. Chris & Matt, you should definitely take a look!

[DrDrij]

Great summary @shaunagm! Its exactly where QuantEcon is also at - including a cell with pip install commands to get the requirements. I think it's a good solution, the requirements are transparent and means the notebooks can run standalone.

[llorracc]

If @shauna Gordon-McKeon <shaunagm@gmail.com> and @andrij Stachurski <dr.drij@gmail.com> and the QuantEcon team are all converging on Google colab, I feel that it must be right! What is the next step we need to take to start using CoLab?

…

On Wed, Mar 27, 2019 at 8:59 PM DrDrij ***@***.***> wrote: Great summary @shaunagm <https://github.com/shaunagm>! Its exactly where QuantEcon is also at - including a cell with pip install commands to get the requirements. I think it's a good solution, the requirements are transparent and means the notebooks can run standalone. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQlfx8Sw6_JYr6iDBdt37BLrgr-Tvrcks5vbBQLgaJpZM4bX937> .

-- - Chris Carroll

[shaunagm]

We chatted about this during our meeting, but I'm going to test out how fast Colab is when loading Latex (Chris, do you know what package that is?) and then, if that works, the next steps are a) making sure existing notebooks all work in colab; b) changing how we refer to the notebooks on the website to point to colab instead of mybinder and c) re-organizing the remarks repo to remove the mybinder stuff since it will no longer be necessary.

[shaunagm]

Update: the issue with overline appears to be a minor syntax error. There's a cell in the original notebook that uses the syntax overline c instead of overline{c}. Once I fixed that, and replaced underline with underbar, I ceased to get errors, although I can't verify that the output is what's desired beyond saying "yup sure does include some underlines and overlines".

I'm finding the underline issue deeply confusing, because isn't underline in basic latex? Why would we need to import anything?

I tried adding !pip install jupyterlab_latex and it doesn't change initial load time at all, since it's not executed until the cell is run. When you do run the cell, it adds another 3-4 seconds, which is not great but not terrible. Importing jupyterlab_latex did not solve the underline issue though.

[mnwhite]

Does basic LaTeX have underline in math mode? That might be the issue.

…

On Thu, Mar 28, 2019 at 3:19 PM Shauna ***@***.***> wrote: Update: the issue with overline appears to be a minor syntax error. There's a cell in the original notebook that uses the syntax overline c instead of overline{c}. Once I fixed that, and replaced underline with underbar, I ceased to get errors, although I can't verify that the output is what's desired beyond saying "yup sure does include some underlines and overlines". I'm finding the underline issue deeply confusing, because isn't underline in basic latex? Why would we need to import anything? I tried adding !pip install jupyterlab_latex and it doesn't change initial load time at all, since it's not executed until the cell is run. When you do run the cell, it adds another 3-4 seconds, which is not great but not terrible. Importing jupyterlab_latex did not solve the underline issue though. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANUQFU_bvqcp93WgS1CAgpG5QLIaqLNpks5vbRW_gaJpZM4bX937> .