sqlite3.OperationalError: database is locked

[vkatrychenko]

Issue

Jupyter notebook cannot connect to kernel. Try to use different images, but no luck.

Jupyter docker images:
kubeflownotebookswg/jupyter-scipy:v1.9.0
kubeflownotebookswg/jupyter-pytorch-full:v1.9.0

I can see the following errors in the pod of jupyter notebook:

File "/opt/conda/lib/python3.11/site-packages/traitlets/config/configurable.py", line 583, in instance inst = cls(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/ipykernel/ipkernel.py", line 125, in __init__ self.shell = self.shell_class.instance( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/traitlets/config/configurable.py", line 583, in instance inst = cls(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 641, in __init__ self.init_history() File "/opt/conda/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 1908, in init_history self.history_manager = HistoryManager(shell=self, parent=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/IPython/core/history.py", line 544, in __init__ super(HistoryManager, self).__init__(shell=shell, config=config, File "/opt/conda/lib/python3.11/site-packages/IPython/core/history.py", line 223, in __init__ self.init_db() File "/opt/conda/lib/python3.11/site-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/IPython/core/history.py", line 107, in catch_corrupt_db self.init_db() File "/opt/conda/lib/python3.11/site-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/IPython/core/history.py", line 105, in catch_corrupt_db self.hist_file.rename(newpath) File "/opt/conda/lib/python3.11/pathlib.py", line 1175, in rename os.rename(self, target) PermissionError: [Errno 13] Permission denied: '/home/jovyan/.ipython/profile_default/history.sqlite' -> '/home/jovyan/.ipython/profile_default/history-corrupt.sqlite' [I 2024-08-28 12:12:56.433 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (5/5), new random ports [W 2024-08-28 12:12:58.037 ServerApp] Timeout waiting for kernel_info reply from d94713c9-975c-4812-a2f6-bc9ca5d8ece1 [I 2024-08-28 12:12:58.038 ServerApp] Connecting to kernel d94713c9-975c-4812-a2f6-bc9ca5d8ece1. [W 2024-08-28 12:13:01.114 ServerApp] Replacing stale connection: d94713c9-975c-4812-a2f6-bc9ca5d8ece1:b30ac2fd-46c4-4882-8392-e40d58595022

Kubeflow verion 1.9.0
Notebook controller: 1.9.0
AKS cluster 1.28.12

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed if no further activity occurs.
Thank you for your contributions.

Members may comment /lifecycle frozen to prevent this issue from being marked as stale.

[vkatrychenko]

The same issue occurs in v1.10.0-rc.1. MySQL database 8.0.21 (Azure mysql flexible server)