Elevator Pitch

Write a wrapper kernel that can act as the proxy between a(ll) language server(s), replacing custom websocket connection wrangling.

Motivation

Extending on #184, (and thinking about #237 which i have been able to reproduce when binder is getting thrashed), and some broader comments in the community... Lab 2 reworked sessions stuff so it is/tries to be even more robust in the face of degraded connections. Trying to get (and maintain) that level of functionality will be hard for us.

Maximizing reuse of the kernel infrastructure would:

• get users all the buffering/reconnection/starting/stopping basically for free
• give us access to better tooling for kernel messages than we'll be able to build for our custom websocket transport
• obviate the need for some of the backend complexity (at least in the manager)
  • though we'd need new complexity to handle dealing with "plain" kernels for the listeners, etc... unless all of that functionality could be moved into the kernel itself
• obviate the need for the clunky (and huge) webpack build of lsp-ws-connection

Design Ideas

The LanguageServerKernel would be an ipykernel.kernelbase.Kernel subclass. It would (awkwardly) show up in the kernel listing (though we could we hide it with css... but it might be cool to interact with it, don't know).

Since we'd still need the serverextension enabled (somehow #257), it could enable the kernelspec pretty directly, rather than needing another file, as something like ilsp.

The frontend would start a KernelSession for ilsp. When it came back with its kernel_info, it might be able to give us all of the server listing stuff, and we'd be able to start talking to it. Presumably the starting/stopping of language servers could be some pithy JSON-schema-constrained language, rather than being "full" python.

Much of the existing stdio machinery would be reused, but would have basically zero chance of hanging the notebook server process, which is A Good Thing. The listener API would be... more complicated to maintain, but it can't all be roses.

As for actually integrating with the kernel messaging protocol: custom comms, used by widgets and vdom, already give us a way to add this kind of feature in a way that could be used by other kernels in the future, provided they had some way to "opt in" to receiving LSP messages.