Support server-side execution

[davidbrochart]

This currently only works with jupyverse, not yet with jupyter-server.
You can try it with:

pip install "jupyterlab>=4.2.0b0"
pip install "jupyverse[jupyterlab, auth]>=0.4.2"
jupyverse --set kernels.require_yjs=true --set jupyterlab.server_side_execution=true

Supersedes jupyterlab/jupyterlab#15448.
Closes jupyterlab/jupyterlab#2833.
Closes jupyterlab/jupyterlab#12867.

Peek.2023-11-28.16-25.mp4

[github-actions]

👈 Launch a Binder on branch davidbrochart/jupyter_collaboration/server-side-execution

[davidbrochart]

I don't understand the error in Test JavaScript:

@jupyter/react-components/lib/index.js:3
export * from './wrap';
^^^^^^
SyntaxError: Unexpected token 'export'

@krassowski or @jtpio do you have an idea?

[davidbrochart]

Not sure what is happening with Test Minimum Versions, any idea @blink1073 ?

[krassowski]

We need similar change as in jupyterlab/jupyterlab#16105

[davidbrochart]

Thanks @krassowski !

[rishabh-dream11]

@davidbrochart Just checking to see if there is consensus on the solution and can be used, I have a product requirement pending on it, hence asking for the same.
We use jupyter-server and have not tried jupyverse.

[imerelli]

Hi, is there any update regarding this feature, which I personally consider very important? When will it be integrated into the official release of Jupyter?

[krassowski]

Good question. I think with #307 and https://github.com/datalayer/jupyter-server-nbmodel it will become at least possible to use it via an extension

[vinaypoduval19]

@krassowski How can I test this feature out? The above extension seems to not be available on pypi

[krassowski]

@vinaypoduval19 I am impatient too, but the jupyter-server-nbmodel extension is in a third-party organization and contributors here do not have a control on when it gets released. I am grateful to @datalayer for developing the extension and I'm trying to be helpful by leaving a review on datalayer/jupyter-server-nbmodel#20 and opening datalayer/jupyter-server-nbmodel#21 but this is as much as I can do.

[b10902118]

This currently only works with jupyverse, not yet with jupyter-server. You can try it with:

pip install "jupyterlab>=4.2.0b0"
pip install "jupyverse[jupyterlab, auth]>=0.4.2"
jupyverse --set kernels.require_yjs=true --set jupyterlab.server_side_execution=true

Supersedes jupyterlab/jupyterlab#15448. Closes jupyterlab/jupyterlab#2833. Closes jupyterlab/jupyterlab#12867.

Peek.2023-11-28.16-25.mp4

This seems not working now. I tried the exact version == or >=, also with jupyter-collaboration installed but still not working.

[fkurushin]

This currently only works with jupyverse, not yet with jupyter-server. You can try it with:

pip install "jupyterlab>=4.2.0b0"
pip install "jupyverse[jupyterlab, auth]>=0.4.2"
jupyverse --set kernels.require_yjs=true --set jupyterlab.server_side_execution=true

Supersedes jupyterlab/jupyterlab#15448. Closes jupyterlab/jupyterlab#2833. Closes jupyterlab/jupyterlab#12867.
Peek.2023-11-28.16-25.mp4

This seems not working now. I tried the exact version == or >=, also with jupyter-collaboration installed but still not working.

I also have a problem but with dependencies versions

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pycrdt-websocket 0.15.1 requires pycrdt<0.11.0,>=0.10.3, but you have pycrdt 0.9.18 which is incompatible.
jupyter-ydoc 3.0.1 requires pycrdt<0.11.0,>=0.10.1, but you have pycrdt 0.9.18 which is incompatible.
Successfully installed pycrdt-0.9.18

[davidbrochart]

You should start from a fresh environment.

[banderlog]

It is not working even from the fresh environment

python3 -m venv venv
./venv/bin/pip install "jupyterlab>=4.2.0b0"
./venv/bin/pip install "jupyverse[jupyterlab, auth]>=0.4.2"
./venv/bin/jupyverse --set kernels.require_yjs=true --set jupyterlab.server_side_execution=true --host=192.168.1.10

[davidbrochart]

@banderlog You need to install jupyter-collaboration.

[banderlog]

OK, so the actual steps are:

python3 -m venv venv
./venv/bin/pip install  jupyter-collaboration
./venv/bin/pip install -U "jupyverse[jupyterlab, auth]"
./venv/bin/jupyverse --set kernels.require_yjs=true --set jupyterlab.server_side_execution=true

Now it's kinda working, but not as on the video at the beginning of this topic.

On page reload it will halt for ~5sec and than produce notebook with updated output, but it will be static
For second page update it will again halt for 5-10 sec and produce output update.
No realtime update on reconnection
And than you will be unable to stop/reload kernel with error 'Context lost'
And even after pkill -9 jupy and server restart I was able to load this notebook only after few minutes if waiting.

When I tried second time, notebook just crashed with detail | "Not Found", no errors in jupyverse.

jupyter-collaboration 4.0.1
jupyverse 0.7.7

[davidbrochart]

Thanks for reporting the issue @banderlog. This should be fixed by the following PRs:

• Follow changes to stream outputs jupyterlab#17469
• Fix stream outputs jupyter-server/jupyverse#480

[davidbrochart]

@banderlog It should be fixed in jupyverse v0.7.8.

[banderlog]

Same set of installation commands as before, now crashes with error

jupyterlab                  4.4.1
jupyter-collaboration       4.0.2
jupyverse                   0.7.8

error log

2025-04-23 18:25:18 [info     ] Server running                 [jupyverse_api.main] url='http://127.0.0.1:8000?token=2809594aedee4fb7a18df8ddfe21ede0'
2025-04-23 18:25:19 [critical ] Module failed while preparing  [fps._module] path=jupyverse.yjs
2025-04-23 18:25:20 [critical ] Application failed             [fps._module]
2025-04-23 18:25:20 [critical ] Exception                      [fps._module]
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/banderlog/TMP/venv/lib/python3.12/site-packages/fps_contents/fileid.py:51 in start         │
│                                                                                                  │
│    48 │   async def start(self) -> None:                                                         │
│    49 │   │   self._db = await connect(self.db_path)                                             │
│    50 │   │   try:                                                                               │
│ ❱  51 │   │   │   await self.watch_files()                                                       │
│    52 │   │   except sqlite3.ProgrammingError:                                                   │
│    53 │   │   │   pass                                                                           │
│    54                                                                                            │
│                                                                                                  │
│ ╭────────────────────────────── locals ───────────────────────────────╮                          │
│ │ self = <fps_contents.fileid.FileIdManager object at 0x7d32a04a6000> │                          │
│ ╰─────────────────────────────────────────────────────────────────────╯                          │
│                                                                                                  │
│ /home/banderlog/TMP/venv/lib/python3.12/site-packages/fps_contents/fileid.py:111 in watch_files  │
│                                                                                                  │
│   108 │   │   │   │   except FileNotFoundError:                                                  │
│   109 │   │   │   │   │   pass                                                                   │
│   110 │   │   │   │   else:                                                                      │
│ ❱ 111 │   │   │   │   │   await cursor.execute(                                                  │
│   112 │   │   │   │   │   │   "INSERT INTO fileids VALUES (?, ?, ?)", (idx, str(path), mtime)    │
│   113 │   │   │   │   │   )                                                                      │
│   114 │   │   │   await self._db.commit()                                                        │
│                                                                                                  │
│ ╭─────────────────────────────── locals ────────────────────────────────╮                        │
│ │ cursor = <sqlite_anyio.sqlite.Cursor object at 0x7d32a04e2630>        │                        │
│ │    idx = 'f371eacac31142aab66cf57417eec506'                           │                        │
│ │  mtime = 1739632546.6498399                                           │                        │
│ │   path = Path('st-flexipatch/ranger/ㅜ1F\udcf8')                      │                        │
│ │   self = <fps_contents.fileid.FileIdManager object at 0x7d32a04a6000> │                        │
│ ╰───────────────────────────────────────────────────────────────────────╯                        │
│                                                                                                  │
│ /home/banderlog/TMP/venv/lib/python3.12/site-packages/sqlite_anyio/sqlite.py:92 in execute       │
│                                                                                                  │
│    89 │   update_wrapper(close, sqlite3.Cursor.close)                                            │
│    90 │                                                                                          │
│    91 │   async def execute(self, sql: str, parameters: Sequence[Any] = (), /) -> Cursor:        │
│ ❱  92 │   │   real_cursor = await to_thread.run_sync(self._real_cursor.execute, sql, parameter   │
│    93 │   │   return Cursor(real_cursor, self._limiter)                                          │
│    94 │                                                                                          │
│    95 │   update_wrapper(execute, sqlite3.Cursor.execute)                                        │
│                                                                                                  │
│ ╭────────────────────────────── locals ──────────────────────────────╮                           │
│ │ parameters = (                                                     │                           │
│ │              │   'f371eacac31142aab66cf57417eec506',               │                           │
│ │              │   'st-flexipatch/ranger/ㅜ1F\udcf8',                │                           │
│ │              │   1739632546.6498399                                │                           │
│ │              )                                                     │                           │
│ │       self = <sqlite_anyio.sqlite.Cursor object at 0x7d32a04e2630> │                           │
│ │        sql = 'INSERT INTO fileids VALUES (?, ?, ?)'                │                           │
│ ╰────────────────────────────────────────────────────────────────────╯                           │
│                                                                                                  │
│ /home/banderlog/TMP/venv/lib/python3.12/site-packages/anyio/to_thread.py:56 in run_sync          │
│                                                                                                  │
│   53 │   │   │   stacklevel=2,                                                                   │
│   54 │   │   )                                                                                   │
│   55 │                                                                                           │
│ ❱ 56 │   return await get_async_backend().run_sync_in_worker_thread(                             │
│   57 │   │   func, args, abandon_on_cancel=abandon_on_cancel, limiter=limiter                    │
│   58 │   )                                                                                       │
│   59                                                                                             │
│                                                                                                  │
│ ╭───────────────────────────────────────── locals ─────────────────────────────────────────╮     │
│ │ abandon_on_cancel = False                                                                │     │
│ │              args = (                                                                    │     │
│ │                     │   'INSERT INTO fileids VALUES (?, ?, ?)',                          │     │
│ │                     │   (                                                                │     │
│ │                     │   │   'f371eacac31142aab66cf57417eec506',                          │     │
│ │                     │   │   'st-flexipatch/ranger/ㅜ1F\udcf8',                           │     │
│ │                     │   │   1739632546.6498399                                           │     │
│ │                     │   )                                                                │     │
│ │                     )                                                                    │     │
│ │       cancellable = None                                                                 │     │
│ │              func = <built-in method execute of sqlite3.Cursor object at 0x7d329f00bb40> │     │
│ │           limiter = <anyio._backends._asyncio.CapacityLimiter object at 0x7d32a06826c0>  │     │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────╯     │
│                                                                                                  │
│ /home/banderlog/TMP/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py:2470 in        │
│ run_sync_in_worker_thread                                                                        │
│                                                                                                  │
│   2467 │   │   │   │   │   worker_scope = scope._parent_scope                                    │
│   2468 │   │   │   │                                                                             │
│   2469 │   │   │   │   worker.queue.put_nowait((context, func, args, future, worker_scope))      │
│ ❱ 2470 │   │   │   │   return await future                                                       │
│   2471 │                                                                                         │
│   2472 │   @classmethod                                                                          │
│   2473 │   def check_cancelled(cls) -> None:                                                     │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ abandon_on_cancel = False                                                                    │ │
│ │              args = (                                                                        │ │
│ │                     │   'INSERT INTO fileids VALUES (?, ?, ?)',                              │ │
│ │                     │   (                                                                    │ │
│ │                     │   │   'f371eacac31142aab66cf57417eec506',                              │ │
│ │                     │   │   'st-flexipatch/ranger/ㅜ1F\udcf8',                               │ │
│ │                     │   │   1739632546.6498399                                               │ │
│ │                     │   )                                                                    │ │
│ │                     )                                                                        │ │
│ │           context = <_contextvars.Context object at 0x7d32a0693000>                          │ │
│ │              func = <built-in method execute of sqlite3.Cursor object at 0x7d329f00bb40>     │ │
│ │            future = <Future finished exception=UnicodeEncodeError('utf-8',                   │ │
│ │                     'st-flexipatch/ranger/ㅜ1F\udcf8', 24, 25, 'surrogates not allowed')>    │ │
│ │      idle_workers = deque([<WorkerThread(AnyIO worker thread, started 137656379901632)>])    │ │
│ │           limiter = <anyio._backends._asyncio.CapacityLimiter object at 0x7d32a06826c0>      │ │
│ │               now = 2367.711488034                                                           │ │
│ │         root_task = <Task pending name='fps._module.Module._main' coro=<Module._main()       │ │
│ │                     running at                                                               │ │
│ │                     /home/banderlog/TMP/venv/lib/python3.12/site-packages/anyio/_backends/_… │ │
│ │                     cb=[_run_until_complete_cb() at                                          │ │
│ │                     /usr/lib/python3.12/asyncio/base_events.py:182, WorkerThread.stop()]>    │ │
│ │             scope = <anyio._backends._asyncio.CancelScope object at 0x7d329f035b20>          │ │
│ │            worker = <WorkerThread(AnyIO worker thread, started 137656379901632)>             │ │
│ │      worker_scope = <anyio._backends._asyncio.CancelScope object at 0x7d32a06824b0>          │ │
│ │           workers = {<WorkerThread(AnyIO worker thread, started 137656379901632)>}           │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /home/banderlog/TMP/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py:967 in run     │
│                                                                                                  │
│    964 │   │   │   │   │   exception: BaseException | None = None                                │
│    965 │   │   │   │   │   threadlocals.current_cancel_scope = cancel_scope                      │
│    966 │   │   │   │   │   try:                                                                  │
│ ❱  967 │   │   │   │   │   │   result = context.run(func, *args)                                 │
│    968 │   │   │   │   │   except BaseException as exc:                                          │
│    969 │   │   │   │   │   │   exception = exc                                                   │
│    970 │   │   │   │   │   finally:                                                              │
│                                                                                                  │
│ ╭────────────────────────────── locals ───────────────────────────────╮                          │
│ │ self = <WorkerThread(AnyIO worker thread, started 137656379901632)> │                          │
│ ╰─────────────────────────────────────────────────────────────────────╯                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf8' in position 24: surrogates not
allowed

2025-04-23 18:25:20 [critical ] Exception                      [fps._module]
TimeoutError: Module timed out while stopping: jupyverse.yjs

2025-04-23 18:25:20 [critical ] Exception                      [fps._module]
TimeoutError: Module timed out while stopping: jupyverse

@davidbrochart

[davidbrochart]

It looks like your db is wrong, could you try removing .jupyter_ystore.db?

[banderlog]

I do not have such file, as far as I know:

sudo updatedb
locate .jupyter_ystore.db
# nothing found

[davidbrochart]

Ah I meant .fileid.db.

[banderlog]

@davidbrochart yep, now it works as intended in the first video 👍