New kernels REST API

[davidbrochart]

Problem

Currently, the kernel protocol over ZMQ in the back-end is "forwarded" to the front-end over a WebSocket. This brings a lot of complexity to the front-end, which has to speak this protocol.
With JupyterLab moving to RTC, the (notebook) UI becomes merely a shared document editor: the user enters some text in a cell, asks for execution, sees outputs being populated, clears outputs... There is no need for the front-end to e.g. interpret IOPub messages in order to display results, all this could be done in the back-end and directly modify the shared document, which would automatically update in the front-end.

Proposed Solution

We could create a much more simple REST API for kernels, which would consist of e.g. "POST execute request", etc.
All kernel state information (idle, busy, dead, restarting...) could go to the new event system.

[davidbrochart]

For the back-end to be able to make changes to a shared document based on some code execution, the back-end has to know which (part of a) document the code is linked to. For instance in a notebook, a cell input contains code that will produce outputs in the same cell.
This means that this REST API must provide information about the document to modify, as well as the kernel to use. A notebook cell "POST execute request" should consist of the following body:

{
  "document_id": "path/to/notebook.ipynb",
  "cell_id": "b4d4a3fe-bc05-4965-b5eb-94732e4329ae",
  "kernel_id": "c71c23dd-38ba-4967-aa46-415ba7317012"
}

[Zsailer]

I would be in favor of extending the current kernels API in core to include a new REST endpoint for executing code. Something like

POST /api/kernels/<kernel_id>/execute

Body: 
{
  "document_id": "path/to/notebook.ipynb",
  "cell_id": "b4d4a3fe-bc05-4965-b5eb-94732e4329ae",
  "kernel_id": "c71c23dd-38ba-4967-aa46-415ba7317012",
  "code": ...
}

Response:
{
  "document_id": "path/to/notebook.ipynb",
  "cell_id": "b4d4a3fe-bc05-4965-b5eb-94732e4329ae",
  "kernel_id": "c71c23dd-38ba-4967-aa46-415ba7317012"

}

This wouldn't require a websocket to the kernel for two-way comms. It simply sends an execute_request to the kernel. On the server, this request would be handled by the KernelManager and formulated into a proper ZMQ message to be sent across kernels shell channel.

I don't think we should replace any of the the current kernel's REST/Websocket API, though. The kernel websocket APIs are still heavily used by projects other than JupyterLab. We don't want those endpoints to degrade or deprecate anytime soon.

As for checking kernel status, I think it also makes sense to add a GET /api/kernels/<kernel_id>/status endpoint that sends a message to IOPub for kernel status and wait for a response. The event bus websocket is also a good way to shuttle these messages.

[davidbrochart]

This work is happening in jupyter-server/jupyverse#191.

[echarles]

There is no need for the front-end to e.g. interpret IOPub messages in order to display results,

What would the impact for the Comm messages used by IPyWidgets?

[davidbrochart]

That's a good point, currently ipywidgets won't work with this, but there are discussions about using y-py in ipywidgets, which would be very aligned with this REST API in my opinion.

[echarles]

but there are discussions about using y-py in ipywidgets

Any pointers? Any backwards compatibly requirement taken in that discussion?

[davidbrochart]

I think it has been discussed in some meetings, but it's probably not going to happen anytime soon.
Regardless of ipywidgets, this kernels REST API could make it easier for new collaborative front-ends which don't want to bother about low-level kernel protocols. I think it can lower the bar quite a lot in that regard.

[vidartf]

I think it would be important to make it possible for clients to (easily) combine RTC experience with Comms. Having an example of how to do this would be helpful (but I agree that it might not be needed in the first iteration).

[davidbrochart]

I opened an issue in ipywidgets.