Meeting Notes 2022

[Zsailer]

Hello everyone,

Welcome to the Jupyter Server Team!

You can find us in our Gitter chat room or visit us at our weekly meetings!

We meet on Thursdays at 8:00am, Pacific Standard Time on Jupyter's Zoom Channel. You can add yourself to the weekly agenda here. Everyone is welcome!

Let's avoid using this thread for discussion. If you'd like to discuss something in the minutes, open a separate issue and reference this thread.

Meeting Notes

• December 15th, 2022
• December 8th, 2022
• December 1st, 2022
• November 17th, 2022
• November 10th, 2022
• November 3rd, 2022
• October 27th, 2022
• October 20th, 2022
• October 13th, 2022
• October 6th, 2022
• September 29th, 2022
• September 22nd, 2022
• September 15th, 2022
• September 8th, 2022
• September 1st, 2022
• August 25th, 2022
• August 18th, 2022
• August 11th, 2022
• August 4th, 2022
• July 28th, 2022
• July 21st, 2022
• July 14th, 2022
• July 7th, 2022
• June 30th, 2022
• June 23rd, 2022
• June 16th, 2022
• June 9th, 2022
• June 2nd, 2022
• May 26th, 2022
• May 19th, 2022
• May 12th, 2022
• May 5th, 2022
• April 28th, 2022
• April 21st, 2022
• April 14th, 2022
• April 7th, 2022
• March 31st, 2022
• March 24th, 2022
• March 17th, 2022
• March 10th, 2022
• March 3rd, 2022
• February 24th, 2022
• February 17th, 2022
• February 10th, 2022
• February 3rd, 2022
• January 27th, 2022
• January 20th, 2022
• January 13th, 2022
• January 6th, 2022

[Zsailer]

January 6th, 2022

Attendees

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Jason Weill	AWS	@jweill-aws
Vidar T Fauske	JP Morgan Chase	@vidartf
David Brochart	QuantStack	@davidbrochart
Eric Charles	Datalayer	@echarles
Kevin Bates	IBM	@kevin-bates
A. T. Darian	Two Sigma	@afshin
Mariko Wakabayashi	OpenZeppelin	@mwakaba2
Sylvain Corlay	QuantStack	@SylvainCorlay
Rahul Goyal	AWS	@rahul26goyal
Frederic Collonval	QuantStack	@fcollonval

Agenda + Minutes

• Terminals as a server extension (initial work):
  • New repo: https://github.com/jupyter-server/jupyter_server_terminals
  • Remove terminals in favor of jupyter_server_terminals extension jupyter_server#651
    • Remove terminals from Jupyter Server
    • Requires jupyter_server_terminals
    • Maintains backwards compatibility, for now.
    • Next major release will not require terminals extension.
• Jupyter Server Team
  • Name initial core team and add documentation about roles #14
  • An important note for how the Server team is being formed:
    • We (accidently, 😅) side-stepped the official approach, defined by Jupyter Governance, to form a decision making body.
    • This process requires a current steering council member to seed (i.e. nominate an initial member). Together, the gradually growing team continues nominating new members.
    • Large, diverse, and inclusive teams is encouraged.
    • We, instead, self nominated ourselves, proposed by @Zsailer.
    • @afshin mentioned in the meeting that this is might be okay
      • we made a good-faith effort at nominating people
      • the current group that's actively participating is relatively small
      • we arrived at the same results anyways.
    • For other teams who might be observing this process, do not follow our process 😆
  • Changes needed in the PR before merging:
    • Absolve the Team Lead position.
    • Define Software Steering Council Representative (we can vote on this person later).
    • Eliminate team colors.
      • General consensus is that we shouldn't distinguish between red and blue.
    • We just have an "active" team member (voting ability) and a "not currently active" team members (no voting ability).
• Filtering request using OpenAPI spec: Filtering request using OpenAPI spec jupyter_server#645
  • All handlers are added to the server, but requests from the user can be blocked/allowed by a defined OpenAPI Spec.
  • There is some overlap with the current work happening around "Authorization" in Jupyter Server
  • Is there a way to merge these two efforts so there aren't two ways to filter requests?
  • Not immediate urgency to merge this work; we'll take some time to compare with authorization work.
• Kernel Parameterization needs a reboot.
  • Form a working group for this work?
    • Meet semi-regularly to discuss this topic.
    • Open a thread on jupyter-server/team-compass repo to begin nucleating around this topic.
  • <= (echarles) Plan to demo https://github.com/deshaw/ksmm at next JupyterLab meeting (can be used a plan B for parametrized kernels)

[Zsailer]

January 13th, 2022

Attendees

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Jason Weill	AWS	@jweill-aws
Vidar T Fauske	JP Morgan Chase	@vidartf
David Brochart	QuantStack	@davidbrochart
Sylvain Corlay	QuantStack	@SylvainCorlay
A. T. Darian	Two Sigma	@afshin
Kevin Bates	IBM	@kevin-bates
Max Klein	JP Morgan Chase	@telamonian
Mariko Wakabayashi	OpenZeppelin	@mwakaba2
Eric Charles	Datalayer	@echarles

Agenda + Minutes

• Protocol Alignment Proposal
  • See the PR for full description. Kudos to David for a clear, well-written proposal. 🎉
  • We need to support both protocols (old and new). We can change the default in a major release sometime in the future.
  • Question from the group: do we use a versioned endpoint or query parameter to select the protocol version?
  • It might be possible to use websocket protocols fields to select the protocol: https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/WebSocket https://html.spec.whatwg.org/multipage/web-sockets.html#the-websocket-interface, there is a question if this could be made backwards-compatible though.
• One 0MQ channel per kernel
• Jupyter Server 1.13.2 released this week.
  • https://github.com/jupyter-server/jupyter_server/releases/tag/v1.13.2
• Finalizing the Jupyter Server Team
  • Outstanding comments/questions.
  • Name initial core team and add documentation about roles #14
• Pending Kernels update
  • Starting, Shutting down, restarting, and interrupting are all pending states.
  • PR in Jupyter Client Further improvements to pending kernels managment jupyter/jupyter_client#732
  • Updated Jupyter Server Tests: Add more awaits for pending kernel in unit tests jupyter_server#654
• Questions about jupyter_server/jupyter_client
  • In the case of ZMQ over TCP, how are the sockets connected?
  • What part of the code consumes the connection file?
    • In local model, these are managed and consumed by local kernel provisioners
• KSMM (Kernelspec Manager) Demo
  • https://github.com/deshaw/ksmm
  • Demo from Eric Charles
  • Parameterize Kernels by generating static kernelspecs using a templating engine
  • Combine with remote_ikernel to generate remote kernels from these specs.

[Zsailer]

January 20th, 2022

Attendees

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Kevin Bates	IBM	@kevin-bates
A. T. Darian	Two Sigma	@afshin
Steven Silvester	MongoDB	@blink1073
David Brochart	QuantStack	@davidbrochart
Sylvain Corlay	QuantStack	@SylvainCorlay
Vidar T Fauske	JP Morgan Chase	@vidartf
Craig Citro	Google	@craigcitro

Agenda + Minutes

• Windows tests are failing from pending kernels work
  • Really flaky on Windows.
  • Might (temporarily) skip these tests on Windows to prevent other PRs from failing.
  • Will open an issue to track this.
• 7.1.0: this version makes JupyterLab very slow
  • Windows-only and unique to user (so far), possible b/c issue with < 1.12 server
  • Conda defaults channel version-lag - what updates defaults? This channel is periodically updated - so version-lags are expected.
  • See also: JupyterLab takes seconds to execute simple code -- It's too SLOW jupyterlab/jupyterlab#11811
  • Actions: Vidar is going to spend some time attempting to reproduce the issue and narrow down the server versions. The current hypothesis is that this may be related to the "nudge" logic that was recently extended in 1.13 to utilize two channels. Once we have more information, we'll move/wrap-up the issues.
• Team Compass is Live!
• Adding SC members who requested to be on the Jupyter Server decision-making body if they aren't already there: @ellisonbg, @ivanov, @damianavila
• Software Steering Council Representative
  • An issue will be opened asking for nominations, after which a vote will be called

[Zsailer]

January 27th, 2022

Name	affiliation	username
Jason Weill	AWS	@jweill-aws
Zach Sailer	Apple	@Zsailer
Damián Avila	2i2c & MADS	@damianavila
Vidar T Fauske	JP Morgan Chase	@vidartf
Steven Silvester	MongoDB	@blink1073
Kevin Bates	IBM	@kevin-bates
Eric Charles	Datalayer	@echarles
Piyush Jain	AWS	@3coins
Akshay Chitneni	Apple	@akshaychitneni
Craig Citro	Google	@craigcitro
Jason Grout	Databricks	@jasongrout

Agenda + Minutes

• SSC Representative
  • Nominations open for 1 more day
  • Vote next week.
• Any feedback?
  • Inject session identifier into environment variable. Inject session identifier into environment variable. jupyter_server#679 - See Inject session identifier into environment variable. jupyter_server#679 (comment) for a summary of the discussion
  • Drop nbclassic/notebook? Drop nbclassic, depend on notebook_shim jupyterlab/jupyterlab#11894 (comment)
    • There's been discussion happening in the notebook meetings around this topic.
    • The current plan is to set nbclassic's requirement on lab to jupyterlab<4. JupyterLab 4 and nbclassic cannot run in the same environment.
  • Terminado 0.13 is out with a new hook that helps get us last_activity. Corresponding PR in jupyter_server_terminals
• Moving Gateway repos (Jupyter Kernel Gateway, Jupyter Enterprise Gateway) into jupyter-server organization?
  • Kevin will open an issue on the team-compass (Move Jupyter Kernel Gateway and Jupyter Enterprise Gateway to the jupyter-server organization #20)
• Discussion: how to onboard new contributors
  • "Good first issue" label might deter some contributions. If you've already contributed once, you might feel guilty picking up another one.
  • Let's not treat "good first issues" as so precious.
  • Often times, someone "claims" a good first issue but never finish. This prevents others from picking up that issue.
  • Host a weekly hack-a-thon/office hour meeting regularly?
    • Interested potential contributors can join and get help with contributing to Jupyter Server
    • Jupyter Server Team members help these contributors get their development environment setup, discuss areas where contributions are needed, teach them about the codebase, etc.
    • If no new contributors show up, Jupyter team members spend the time pair programming on some new software feature in Jupyter Server.

[Zsailer]

February 3rd, 2022

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Vidar T Fauske	JP Morgan Chase	@vidartf
Kevin Bates	IBM	@kevin-bates
Luciano Resende	Apple	@lresende
Jason Weill	AWS	@jweill-aws
Steven Silvester	MongoDB	@blink1073
Eric Charles	Datalayer	@echarles
Rahul Goyal	AWS	@rahul26goyal
Damián Avila	2i2c & MADS	@damianavila
Akshay Chitneni	Apple	@akshaychitneni
Paul Ivanov	Noteable	@ivanov
Craig Citro	Google	@craigcitro
A. T. Darian	Two Sigma	@afshin
Jessica Xu		@jess-x

Agenda + Minutes

• Authorization PR ready for final review
  • Final review between now and next Tuesday
  • If nothing major shows up in the review, we'll plan to merge on Tuesday! (Thank you, Min)!
• Test Flakiness
  • Possible conflict between pytest-tornasync, pytest-timeout, and asyncio.
• Moving Gateways to Server org
  • EG Update:
    • 2.6.0 release is out
    • Major doc re-org coming, along with 3.0 (Spark 3 support)
    • 4.0 will be based on kernel provisioners
• Discussion of starting experimental repos within jupyter-server Github org
  • Ok to use org as umbrella for experiments (instead of personal accounts)
  • communicate explicitly in the README that it is experimental
• Auto pre-commit - stick with the default GitHub Access Token for now?
• Jupyter Server Office/Hacking Hour.
  • Thoughts on this idea?
  • If we like it, how often?
  • Naming this hour session matters.
    • Calling it "office" hours may lead people to come with specific questions about their deployments.
    • Calling it "hacking hour" may lead to people doing a bunch of exploratory work
    • Calling it a "work party" may be more explicit in capturing the intent
• SSC Representative
  • Open an issue to vote.

[Zsailer]

February 10th, 2022

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Vidar T Fauske	JP Morgan Chase	@vidartf
Max Klein	JPM	@telamonian
Kevin Bates	IBM	@kevin-bates
Jason Weill	AWS	@jweill-aws
A. T. Darian	Two Sigma	@afshin
Mariko W.	OpenZeppelin	@mwakaba2
Rahul G	AWS	@rahul26goyal
Eric Charles	Datalayer	@echarles
Piyush Jain	AWS	@3coins
Craig Citro	Google	@craigcitro

Agenda + Minutes

• jupyter kernel init sequence diagram

  • adding the gateway_kernel_init_sequence_diagram.png to the docs enterprise_gateway#1039

• how to use pending kernels

  • and with a gateway?
    • EG can't support pending kernels until it moves to provisioners (it's currently incompatible with juptyer_client 7.x)
    • KG (Kernel Gateway) could support pending kernels if we moved it to jupyter_server (its based on notebook IIRC).

• PR for making contents hook support multiple hooks: https://github.com/jupyter-server/jupyter_server/pull/696/files

  • Triggered by this discussion: Save document from the backend using y-py jupyterlab/jupyterlab#11599 (comment)
  • Should probably have different ways to add/remove hooks as needed
  • Order is probably important
  • Uniqueness should probably be enforced in case the validator triggers multiple times?
  • Maybe expose a new trait exposing the collection of hooks for manipulation.

• Google Group for decision-making body

• Question: Has there been any work in the past or on-going to improve the synchronization (autosave) of notebook to server.

  • maybe Saving and closing a MimeDocument quickly results in "Disposed" error jupyterlab/jupyterlab#5787 ?

• Releasing rules

  • Open an issue in team compass to discuss rules around cutting new releases of software in the Jupyter Server org.

[Zsailer]

February 17th, 2022

Name	affiliation	username
Jason Weill	AWS	@jweill-aws
Zach Sailer	Apple	@Zsailer
Kevin Bates	IBM	@kevin-bates
A. T. Darian	Two Sigma	@afshin
Rahul Goyal	AWS	@rahul26goyal
Mariko W	OpenZeppelin	@mwakaba2
Piyush Jain	AWS	@3coins
Craig Citro	Google	@craigcitro

Agenda + Minutes

• Call for review: Identity API
  • 4 design decision questions to review
• Dependency Handling / upper capping dependencies
  • Dependency Handling jupyter_server#699
  • We have two dependencies with caps
• Call for review: Register pre/post save hooks, call them sequentially
• Discussion around Kernel Specification Manager
  • Provides a way to generate new kernel specs.
  • Schemas from the server inform the client what parts/parameters of the kernel spec are configurable.
  • In the UI, user can set the value of for each parameter.
  • On save, a new kernel spec is created on the server that will be used to launch kernels with the given parameter values.
  • VS Parameterized kernel specs proposal jupyter/enhancement-proposals#87
• Our first "Contributing" hour is happening right after this call!

[Zsailer]

February 24th, 2022

Name	affiliation	username
Jason Weill	AWS	@jweill-aws
Kevin Bates	IBM	@kevin-bates
Zach Sailer	Apple	@Zsailer
Eric Charles	Datalayer	@echarles
Vidar T Fauske	JP Morgan Chase	@vidartf
Rahul Goyal	AWS	@rahul26goyal
Akshay Chitneni	Apple	@akshaychitneni
A. T. Darian	Two Sigma	@afshin
Andrey Velichkevich	Apple	@andreyvelich
Piyush Jain	AWS	@3coins

Agenda + Minutes

• Making sense of the new world: Notebook, RetroLab, JupyterLab, NBClassic, Notebook Shim, and Jupyter Server
  • Notebook v6 —> Tornado web server and legacy Notebook Javascript
  • Notebook v7 —> JupyterLab components with classic Notebook UX (used to be RetroLab) + Jupyter Server
  • JupyterLab —> JupyterLab UX + Jupyter Server
  • NBClassic used to have 2 tasks
    • Adds classic Notebook JS and endpoints to Jupyter Server
    • Handle shimming traits and config (moved to notebook_shim)
  • notebook_shim
    • Handle shimming traits and config
  • What options will users have to run "side-by-side"?
    • Notebook v6, NBClassic, JupyterLab
    • Notebook v7 + JupyterLab
• Current state of custom messages on the control channel?
  • Add a usage request/reply on the Kernel Control channel jupyterlab/jupyterlab#11285
  • Get resource usage from ipykernel: Add usage_request and usage_reply based on psutil ipython/ipykernel#805
  • Can we relax the typing in JupyterLab to allow custom messages to the control channel?
    • This shouldn't require JEP.
    • Discussion on that can happen in Custom message on the Kernel Control Channel jupyterlab/jupyterlab#12117
  • If it's just resource usage, perhaps provisioners are a better option. This was one of the motivations for provisioners.
  • Zach's motivation is to push this one step further than resource usage metrics...
    • Can we get arbitrary "telemetry" events to flow from kernel to frontend client.
    • Open Telemetry could enable us to trace an event from the deepest layer of the stack back to the client.
    • The control channel looks like the best place to flow messages from the kernel to any kernel client.
    • Open Telemetry might be a standard we can add to Jupyter, but would require a JEP.
• "Contributing" Hour right after this meeting!
  • Introduced traitlets
  • Introduced server extensions
  • cover Jupyter Server unit tests next week.

[jess-x]

March 3, 2022

Name	affiliation	username
Jason Weill	AWS	@jweill-aws
Vidar T Fauske	JP Morgan Chase	@vidartf
Zach Sailer	Apple	@Zsailer
Steven Silvester	MongoDB	@blink1073
Rahul Goyal	AWS	@rahul26goyal
Kevin Bates	IBM	@kevin-bates
Kiersten Stokes	IBM	@kiersten-stokes
Akshay Chitneni	Apple	@akshaychitneni
A. T. Darian	Two Sigma	@afshin
Mariko W	OpenZeppelin	@mwakaba2
Andrey Velichkevich	Apple	@andreyvelich

Agenda + Minutes

• Issue #312 - Improving Network Performance opened September 2020

  • In looking at things, @goanpeca found that the server is validating notebooks twice - potential 50% savings per notebook!
    • Nbformat PR #249 adds the ability to capture validation errors]
    • TODO: On next nbformat release, update server to add the additional parameter, remove second validation, and bump dependency floor on nbformat
    • TODO: We should also enable the ability to leverage fastjsonschema that @goanpeca added to nbformat, exposed via env.
      • Should this be configurable, or fastjsonschema is auto-detected and used unconditionally? (leaning toward configurable in case issues arise or other options appear)
        • If configurable, should it expose (and track) available options or a boolean? (leaning toward enum, default = jsonschema)
  • We should probably talk about static asset management posted in the issue as well.

• Maintenence-related cleanups

  • Terminal culling works again on Windows!
  • Testing against min dependencies and pre-releases
  • Removed pinning up upper version deps
  • Working on cleaning up jupyter_client as well so we can handle warnings as errors as well
  • Flit-based backend in jupyter-packaging coming soon

• pre/post_save_hook PR: https://github.com/jupyter-server/jupyter_server/pull/696

  • Urgent, because of issues in JupyterLab
  • Not ready to merge in it’s current state.
  • Current state/question: should there be a way to order these post_save_hooks?
  • Thoughts:
    • The single value trait, post_save_hook, should be deprecated
    • Replace with a List trait, post_save_hooks, where order matters.
    • What happens if the same post_save_hook is given by an extension and by the user configuration? It would add the hook twice, which probably isn’t intended.
    • Need to put more thought into this.

• Identity API /api/me

  • This will likely require a new major version. It may be about time to do this.
  • Adding this API is one of the critical building blocks for RTC

• Websocket subprotocol bug: https://github.com/jupyter-server/jupyter_server/pull/706

  • Causing issues with pending kernels
  • Thank you, David, for quickly fixing!

• “Contributing” hour right after this!

[jess-x]

March 10, 2022

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Kevin Bates	IBM	@kevin-bates
Vidar Tonaas Fauske	JP Morgan Chase	@vidartf
A. T. Darian	Two Sigma	@afshin
Kiersten Stokes	IBM	@kiersten-stokes
Rahul Goyal	AWS	@rahul26goyal
Jason Weill	AWS	@jweill-aws
Steven Silvester	MongoDB	@blink1073
Carlos Herrero	QuantStack	@hbcarlos
Akshay Chitneni	Apple	@akshaychitneni
Andrey Velichkevich	Apple	@andreyvelich
Craig Citro	Google	@craigcitro

Agenda + Minutes

• Darian: Governance questions

  • Add team-compass page including JupyterLab Council members jupyterlab/frontends-team-compass#139
  • "Council"
    • Change the current usage of "team" to "council"
  • Voting
    • Change our voting process around electing new team members from public to a private mailing list.
    • Anytime we're voting on people, we should do this privately.

• Authorization PR introduced a backwards incompatible change

  • Using jupyter_server handlers, but no its webapp causes issues if the custom webapp doesn't provide an authorizer.
  • Make Authorizers singletons?
  • If the tornado web app doesn't have an authorizer in the settings, call Authorizer.instance(...)

• Work in progress: Handle warnings in tests jupyter/jupyter_client#751 to clean up pending kernels and asyncio usage in jupyter_client and jupyter_server.

  • Drop the .ready (a Future) property in the KernelManager.
  • Python's core documentation recommends not using Futures directly.
  • Use public methods to synchronously check if a pending kernel is ready.
  • Drop nested looks, i.e. nested_asyncio.

• absolute vs resolve serverapp: Use .absolute() instead of .resolve() for symlinks jupyter_server#712: some interesting discussion that might be relevant in https://discuss.python.org/t/pathlib-absolute-vs-resolve/2573/2 and https://bugs.python.org/issue39090

• "Contributing" hour right after this!

[Zsailer]

March 17th, 2022

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Kevin Bates	IBM	@kevin-bates
Jason Weill	AWS	@jweill-aws
Kiersten Stokes	IBM	@kiersten-stokes
A. T. Darian	Two Sigma	@afshin
Jessica Xu	Jupyter community	@jess-x
Rahul	AWS	@rahul26goyal
Andrey Velichkevich	Apple	@andreyvelich
Piyush Jain	AWS	@3coins

Agenda + Minutes

• Add Carol to Jupyter Server Council 🎉

• Governance fun times

  • Zach and Darian are working on a team-compass template for other Jupyter subprojects.
  • The plan is to dogfood the processes described in the template here in Jupyter Server.

• Cal Poly's Notifications extension

  • Contains a server and jupyterlab extension
  • Should some of the server-side endpoints land in core server?
  • Needs a websocket for broadcasting events to the frontend
  • Needs a REST API for creating and fetching notifications.
  • Jupyter Eventlogging/telemetry plans to implement plumbing that notifications can use.

• Telemetry/EventLogging

  • Should we track [Telemetry] Add eventlog endpoint for collecting client-side events jupyter_server#501
  • What is the future of telemetry within JupyterLab? Are there any future/current proposals?
  • FYI: Working on upgrading jupyterlab-telemetry to work with JL3. Update for compatibility with JupyterLab 3 jupyterlab/jupyterlab-telemetry#23
  • Relevant links for Eventlogging in Jupyter
    • JEP: Jupyter Telemetry Enhancement Proposal jupyter/enhancement-proposals#41
    • Core Jupyter Telemetry Python Library: https://github.com/jupyter/telemetry
    • Draft PR adding an eventlog to Jupyter Server: [Telemetry] Add basic telemetry to Jupyter server and begin emitting server events jupyter_server#364

• Ready for review? Inject session identifier into environment variable. jupyter_server#679

• "Contributing" hour right after this!

  • Discussed the relationship between JupyterHub and Jupyter Server
  • Wrote a basic Jupyter Server extension from scratch