GESIS Collaboration - Dynamic image building in a JupyterHub

[choldgraf]

Context

We recently started a collaboration with GESIS with the goal of generalizing, improving, and sustaining the "persistent binderhub" deployment at notebooks.gesis.org.

Here are the major things we'd like to do as part of this collaboration:

• Generalize the "Persistent BinderHub" setup so that it is no longer unique to GESIS infrastructure
• Improve aspects of the design and implementation to be more sustainable, scalable, usable, etc
• Upstream as much as possible to parts of the JupyterHub / Binder ecosystem

Here's a link to the deliverables on the collaboration

Project roles

• Point of contact with @arnim and GESIS: @jmunroe
• Engineering project manager: @damianavila
• Engineering developer: @consideRatio (with @yuvipanda's help as of now Feb 1st 2023).

References

• Yuvi's write-up about unifying BinderHub/JupyterHub under one Helm Chart
• Ryan's idea of "Binder for everything"

We'll use this issue to track progress on the collaboration, and to list issues where we have discussion and iteration on more focused parts of the collaboration. Below is a rough list of things to do.

### Updates
- [x] Announce this collaboration publicly in some way
- [ ] https://github.com/2i2c-org/meta/issues/254
- [ ] https://github.com/2i2c-org/infrastructure/issues/2120
- [ ] https://github.com/2i2c-org/infrastructure/issues/2119
- [ ] Have a community meeting for feedback and brainstorms about design and end-product goals

### Cloud cost reminders
- [ ] https://github.com/2i2c-org/meta/issues/547
- [ ] https://github.com/2i2c-org/meta/issues/546

Dedicated board

Ref: BinderHub-JupyterHub.

[choldgraf]

ping to @bitnik @arnim and @MridulS in case any of them are interested in following along, collaborating, or joining in discussions and meetings.

[arnim]

Thank you @choldgraf ;)

[sgibson91]

Just sharing some slides I presented to a Turing-based project around what I perceive the use cases/target audiences are for JupyterHub and BinderHub and how this dynamic image building in JupyterHub work might affect BinderHub as a separate project. This was presented in order for them to understand how work on internal infrastructure would affect their project and help them make decisions, and is only a reflection of my opinion after having conversations with different folks.

• https://docs.google.com/presentation/d/1ueejB2ZcYaZTve3CqVRE1d4fl0TCSMXF_vuD8hm23fU/edit?usp=sharing

[arnim]

@choldgraf should we do some kick-off?

[choldgraf]

Two quick updates:

Meeting / timing

@arnim yes I believe it's a good idea - we have been working to finish up deploying Pangeo's BinderHub before working on this, and I think we figured out a short-term path forward there last week so hopefully can then shift focus to this project. We will ping here once it's time.

Kernels as a service

In a recent conversation with @jlperla we noted that this work might be related to another feature that many have requested in the Jupyter ecosystem, which is something like "Binder / Jupyter kernels as a service for scalable computation". The idea is that you could define an environment via a Binder-like repository, build it into an image with repo2docker, and then scale computation using that repository's environment via some cloud mechanism. This wouldn't be as rapidly scalable as something like Dask Gateway, but could be a useful way to rapidly / interactively parallelize something in the cloud. Just noting that here in case we find a way to connect it with this work.

[arnim]

The "kernels as a service for scalable computation" idea sounds interesting ;) All the best for the deployment of Pangeo's Hub. A while ago @rabernat raised also an extremely interesting question. Maybe we can indeed move one step closer to a generic backplane for all kinds of scientific infrastructure. Computational replicability is key in economics, geoscience, and the social sciences ;)

[yuvipanda]

Here's a document I wrote almost a year ago now that might help https://hackmd.io/OXhKs4xyQra0KgBglegSBQ

[choldgraf]

@sgibson91 and I discussed this one a little bit today as well, and we agree that it might be helpful to spend some of her time acting as community strategic lead w/ JupyterHub to help steward discussions and feedback amongst stakeholders for this work (with technical vision / ideas / etc coming from others in the JupyterHub ecosystem).

This might be a way to help make progress on this issue and also experiment with ways that the Community Strategic Lead can start improving team processes for discussion and feedback-gathering.

[choldgraf]

Update: Damian serving in a PM role

I spoke with @damianavila today and we agreed that this project will make more rapid progress if we can assign somebody to serve in a project manager style role to help us keep track of plans, conversations, timelines, and deliverables.

@damianavila said he would be willing to serve in this role and he will reduce his time on the PyData theme in order to grow the capacity to work here.

So our next steps are to:

• Assign @damianavila in a project manager role on this issue
• Create a place to track and plan this effort (maybe this issue + a project board for deliverables)
• Plan initial meetings to align / plan
• Plan our strategy around Engineering capacity (a mix of growing capacity in the 2i2c team, or bringing in capacity via sub-contracts with others

[arnim]

This is great news. Thank you @damianavila for helping with this 🎉