Library libcudart.so.11.0 not found in jupyter-notebook

[hypercube999]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]
Install juju
Install microk8s version 1.24.17
Install kubeflow (juju deploy kubeflow --trust --channel=1.7/stable)
Create a jupyter-notebook with any image designed to work with gpu, e.g. charmedkubeflow/jupyter-tensorflow-cuda-full:v1.7.0_20.04_1
Start the notebook and run a program in it that uses a video card, for example a python program that trains a neural network using tensorflow.
Jupyter show error "Could not load dynamic library 'libcudart.so.11.0' <...> No such file or directory"

What did you expect to happen:
The program works using gpu.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
On another similar deployment done in late August everything works fine

Environment:

• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): 1.7/stable
• Kubernetes platform: (e.g. minikube) microk8s
• Kubernetes version: (use kubectl version): 1.24.17
• OS (e.g. from /etc/os-release): ubuntu server 20.04
  

[thesuperzapper]

@hypercube999 what happens when you use the updated images from Kubeflow 1.8 (which can be used with older Kubeflow versions too).

For example: kubeflownotebookswg/jupyter-tensorflow-cuda-full:v1.8.0

[andreyvelich]

/transfer notebooks

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed if no further activity occurs.
Thank you for your contributions.

Members may comment /lifecycle frozen to prevent this issue from being marked as stale.