Persistent Jupyter Kernels - Restore/Re-connect to an existing local/remote kernel (do not shutdown kernel upon closing/reloading vscode/vscode.dev)

[DonJayamanne]

Problem

Local

• User starts a notebook, the kernel is now running on the local machine
• Assume the computer goes to sleep,
• After a while if we go back into the notebook, the Notebook is unable to re-connect to the same kernel (kernel state is lost)
• Similarly if user re-loads VS Code, the notebook is unable to re-connect to the same kernel (kernel state is lost)
• Similarly if a user is using Remote SSH, and connection is reset and user re-connects and opens the same notebook, then the user is unable to re-connect to the same kernel (kernel state is lost)

Remote

• User opens a notebook and runs a cell against a remote kernel
• Similarly if user re-loads VS Code (or vscode.dev), the notebook is unable to re-connect to the same kernel (kernel state is lost)

Investigation Running Server & JupyterLab API for extensibility
Goals:

• Long running kernels
  • User can open a notebook with a cell thats still running & see the output being generated
  • Same as User can open notebooks related to kernels that are still running.
• Extensibility for extension authors
  This is a by-product of the long running kernels (i.e. you get this for free - almost)

Planned (related) Prototypes

• Long running kernels
  Solve problems related to kernel/session being lost due to :
  • VS Code Shutdown
  • VS Code Restart
  • Computer sleeping
  • SSH connection issues
  • AML Compute will benefit
• By product of extensiblity
  • Julia Widgets (i might end up doing this first, might be easier)
  • IPyWidget outside notebooks
  • Variable viewer using the new api
  • Data Frame viewer using the new api

Technical details

• Server
  Background process
  Manages kernels & sessions
  Expose kernel socket connection over this connection (we already have the code/technology for this) - proxy socket (dummy kernel in UI layer, by creating a dummy socket connection)
  Security - how do we secure this web server (will need to be addressed, but i'm leaving that for later)
• Expose Jupyter extension extensibility over Jupyter Lab API
  I wont be exposing a connection, instead will just expose the SessionManager, KernelManager & other class instances from extension API

Also related #300