Rouge RPM CUDA package in Jupyter Notebook image

[jstourac]

This is a copy of https://issues.redhat.com/browse/RHODS-11680.

Description of problem:

When creating the Jupyter Notebook servers from the images which contain CUDA tooling, I can see that apart from expected CUDA version (11.8 and 11.4 respectively at the moment) RPM packages, there is also one package cuda-toolkit-config-common with unexpected version:

2023.1_image_version

(app-root) (app-root) rpm -qa | grep -i cuda
cuda-toolkit-config-common-12.2.128-1.noarch
cuda-toolkit-11-config-common-11.8.89-1.noarch
cuda-toolkit-11-8-config-common-11.8.89-1.noarch
cuda-cudart-11-8-11.8.89-1.x86_64
cuda-compat-11-8-520.61.05-1.x86_64
...

1.2_image_version

(app-root) (app-root) rpm -qa | grep cuda
cuda-toolkit-config-common-11.8.89-1.noarch
cuda-toolkit-11-config-common-11.8.89-1.noarch
cuda-toolkit-11-4-config-common-11.4.148-1.noarch
cuda-cudart-11-4-11.4.108-1.x86_64
cuda-compat-11-4-470.141.10-1.x86_64
...

Prerequisites (if any, like setup, operators/versions):

Tested with images provided with RHODS 1.32 RC6

Steps to Reproduce

1. Go to RHODS -> applications -> enabled -> launch Jupyter application
2. Choose any of the images with CUDA and click on Start server button
3. Wait till the server is created and then open it and run following command in terminal:

rpm -qa | grep cuda

Fortunately presence of this rouge RPM package doesn't break any functionality at the moment, so our main problem here is just eventual user/customer confusion. Still, there may be a risk that content of the rouge package will change in the future and may actually affect the behavior somehow!

Actual results:

Extra cuda RPM packages are present in the Jupyter Notebook images.

Expected results:

Only those cuda RPM packages present in Jupyter Notebook images, that are relevant for particular CUDA version.

Reproducibility (Always/Intermittent/Only Once):

Always

Additional info:

Presence of this rouge RPM package doesn't break any functionality at the moment since it actually installs only directory that in a result is just a symlink to the actual CUDA installation of a proper version, see:

2023.1_image_version

(app-root) (app-root) rpm -ql cuda-toolkit-config-common-12.2.128-1.noarch
/etc/ld.so.conf.d/000_cuda.conf
(app-root) (app-root) rpm -ql cuda-toolkit-11-config-common-11.8.89-1.noarch
/etc/ld.so.conf.d/989_cuda-11.conf
(app-root) (app-root) rpm -ql cuda-toolkit-11-8-config-common-11.8.89-1.noarch
(contains no files)

(app-root) (app-root) cat /etc/ld.so.conf.d/000_cuda.conf
/usr/local/cuda/targets/x86_64-linux/lib
(app-root) (app-root) cat /etc/ld.so.conf.d/989_cuda-11.conf
/usr/local/cuda-11/targets/x86_64-linux/lib

(app-root) (app-root) ls -dl /usr/local/cuda*
lrwxrwxrwx.  1 root root  22 Aug 17 17:39 /usr/local/cuda -> /etc/alternatives/cuda
lrwxrwxrwx.  1 root root  25 Aug 17 17:39 /usr/local/cuda-11 -> /etc/alternatives/cuda-11
drwxr-xr-x. 13 root root 183 Aug 17 17:43 /usr/local/cuda-11.8

(app-root) (app-root) ls -l /etc/alternatives/cuda 
lrwxrwxrwx. 1 root root 20 Aug 17 17:39 /etc/alternatives/cuda -> /usr/local/cuda-11.8
(app-root) (app-root) ls -l /etc/alternatives/cuda-11
lrwxrwxrwx. 1 root root 20 Aug 17 17:39 /etc/alternatives/cuda-11 -> /usr/local/cuda-11.8

1.2_image_version

(app-root) (app-root) rpm -ql cuda-toolkit-config-common-11.8.89-1.noarch
/etc/ld.so.conf.d/000_cuda.conf
(app-root) (app-root) rpm -ql cuda-toolkit-11-config-common-11.8.89-1.noarch
/etc/ld.so.conf.d/989_cuda-11.conf
(app-root) (app-root) rpm -ql cuda-toolkit-11-4-config-common-11.4.148-1.noarch
(contains no files)

(app-root) (app-root) cat /etc/ld.so.conf.d/000_cuda.conf
/usr/local/cuda/targets/x86_64-linux/lib
(app-root) (app-root) cat /etc/ld.so.conf.d/989_cuda-11.conf
/usr/local/cuda-11/targets/x86_64-linux/lib

(app-root) (app-root) ls -dl /usr/local/cuda*
lrwxrwxrwx.  1 root root  22 Nov 17  2022 /usr/local/cuda -> /etc/alternatives/cuda
lrwxrwxrwx.  1 root root  25 Nov 17  2022 /usr/local/cuda-11 -> /etc/alternatives/cuda-11
drwxr-xr-x. 13 root root 200 Nov 17  2022 /usr/local/cuda-11.4

(app-root) (app-root) ls -l /usr/local/cuda-11
lrwxrwxrwx. 1 root root 25 Nov 17  2022 /usr/local/cuda-11 -> /etc/alternatives/cuda-11
(app-root) (app-root) ls -l /etc/alternatives/cuda-11
lrwxrwxrwx. 1 root root 20 Nov 17  2022 /etc/alternatives/cuda-11 -> /usr/local/cuda-11.4
(app-root) (app-root) ls -l /etc/alternatives/cuda
lrwxrwxrwx. 1 root root 20 Nov 17  2022 /etc/alternatives/cuda -> /usr/local/cuda-11.4

So until the content of the package is only it is now, we are safe. But this may change in the future.

[jstourac]

I did some investigation already and found out that the culprit here is actually right in the CUDA RPM package from Nvidia. Still, I created this issue so we can have it tracked and decide whether we'll just ignore and take the risk, report to Nvidia or come up with some our solution. And also for eventual future reference.

Here are my findings:

1. I believe that these are the dockerfiles from which the Jupyter Notebook images are created: cuda/ubi8-python-3.8 and cuda/ubi9-python-3.9.
2. Place where the rouge RPM package is introduced into the created image is this line in particular.
3. By checking the package requirements, I can see that following packages depends on the rouge package:

(app-root) (app-root) rpm -q --whatrequires cuda-toolkit-config-common
cuda-cudart-11-8-11.8.89-1.x86_64
libnvjpeg-11-8-11.9.0.86-1.x86_64
libnpp-11-8-11.8.0.86-1.x86_64
libcusparse-11-8-11.7.5.86-1.x86_64
libcusolver-11-8-11.4.1.48-1.x86_64
libcurand-11-8-10.3.0.86-1.x86_64
libcufile-11-8-1.4.0.31-1.x86_64
libcufft-11-8-10.9.0.58-1.x86_64
libcublas-11-8-11.11.3.6-1.x86_64

Similarly for UBI8, where there are actually two rouge packages currently: cuda-toolkit-config-common, cuda-toolkit-11-config-common.

At the same time the cuda-toolkit-config-common-12.2.128-1.noarch is the only package that provides cuda-toolkit-config-common:

(app-root) (app-root) rpm -q --whatprovides cuda-toolkit-config-common
cuda-toolkit-config-common-12.2.128-1.noarch

I checked the Nvidia repo to see whether anything changed for the latest release, but it seems it doesn't:

$ rpm -q --requires cuda-cudart-12-2-12.2.140-1.x86_64.rpm 
warning: cuda-cudart-12-2-12.2.140-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY
/sbin/ldconfig
/sbin/ldconfig
cuda-toolkit-12-2-config-common
cuda-toolkit-12-config-common
cuda-toolkit-config-common
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1

---

So if I don't miss anything, we have three options here:

1. Ignore this problem and take the risk that Nvidia may change content of the rouge package in some way that affects an actual CUDA toolkit version installation and that some customers may be confused.
2. Force the RPM installation by removing the rouge package(s) from our image installation.
3. Report to Nvidia so that they try to update/fix their dependency system in their packages somehow.

[jstourac]

For the record - this seem to be still same issue with the images 2024.1:

(app-root) (app-root) rpm -qa | grep cuda
cuda-toolkit-config-common-12.4.127-1.noarch
cuda-toolkit-12-config-common-12.4.127-1.noarch
cuda-toolkit-12-1-config-common-12.1.105-1.noarch
cuda-cudart-12-1-12.1.105-1.x86_64
cuda-compat-12-1-530.30.02-1.x86_64
cuda-opencl-12-1-12.1.105-1.x86_64
cuda-nvrtc-12-1-12.1.105-1.x86_64
cuda-libraries-12-1-12.1.1-1.x86_64
libnccl-2.17.1-1+cuda12.1.x86_64
cuda-nvtx-12-1-12.1.105-1.x86_64
cuda-cccl-12-1-12.1.109-1.x86_64
cuda-cudart-devel-12-1-12.1.105-1.x86_64
cuda-profiler-api-12-1-12.1.105-1.x86_64
cuda-opencl-devel-12-1-12.1.105-1.x86_64
cuda-nvrtc-devel-12-1-12.1.105-1.x86_64
cuda-nvprune-12-1-12.1.105-1.x86_64
cuda-nvprof-12-1-12.1.105-1.x86_64
cuda-nvdisasm-12-1-12.1.105-1.x86_64
cuda-nvcc-12-1-12.1.105-1.x86_64
cuda-gdb-12-1-12.1.105-1.x86_64
cuda-driver-devel-12-1-12.1.105-1.x86_64
cuda-cuxxfilt-12-1-12.1.105-1.x86_64
cuda-cupti-12-1-12.1.105-1.x86_64
cuda-cuobjdump-12-1-12.1.111-1.x86_64
cuda-compiler-12-1-12.1.1-1.x86_64
cuda-minimal-build-12-1-12.1.1-1.x86_64
cuda-libraries-devel-12-1-12.1.1-1.x86_64
cuda-nsight-compute-12-1-12.1.1-1.x86_64
libnccl-devel-2.17.1-1+cuda12.1.x86_64
cuda-nvml-devel-12-1-12.1.105-1.x86_64
cuda-sanitizer-12-1-12.1.105-1.x86_64
cuda-command-line-tools-12-1-12.1.1-1.x86_64
libcudnn8-8.9.0.131-1.cuda12.1.x86_64
libcudnn8-devel-8.9.0.131-1.cuda12.1.x86_64

(app-root) (app-root) rpm -q --whatrequires cuda-toolkit-config-common
cuda-cudart-12-1-12.1.105-1.x86_64
libnvjpeg-12-1-12.2.0.2-1.x86_64
libnvjitlink-12-1-12.1.105-1.x86_64
libnpp-12-1-12.1.0.40-1.x86_64
libcusparse-12-1-12.1.0.106-1.x86_64
libcusolver-12-1-11.4.5.107-1.x86_64
libcurand-12-1-10.3.2.106-1.x86_64
libcufile-12-1-1.6.1.9-1.x86_64
libcufft-12-1-11.0.2.54-1.x86_64
libcublas-12-1-12.1.3.1-1.x86_64
cuda-opencl-12-1-12.1.105-1.x86_64

(app-root) (app-root) nvidia-smi | grep "CUDA Version:"
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |

This makes an impression that CUDA 12.4 is used on the running notebook/workbench, but based on the installed packages, it should still be CUDA 12.1. It is causing confusion.

[jstourac]

Since we're not using GH issues anymore, I'll update this issue with the latest findings https://issues.redhat.com/browse/RHOAIENG-6673.