[Feature Request] Using the cuda dlls installed with pip from official Nvidia python packages in onnxruntime-gpu

[martinResearch]

Describe the feature request

Short Description

Enable the usage of CUDA DLLs installed via pip from the NVIDIA Python index or include the CUDA DLLs in the onnxruntime-gpu wheel when installing with pip install onnxruntime-gpu[cuda_dlls].

Problem

Installing CUDA DLLs for onnxruntime-gpu currently involves limitations such as:

1. Mandatory user account creation on the NVIDIA website.
2. Dependency on admin rights, restricting installation on machines without such privileges.
3. Risk of installing incompatible CUDA versions.
4. Inconvenience of updating CUDA_PATH when switching Python environments with different CUDA versions.

In contrast, PyTorch on Windows includes CUDA DLLs in its wheels, simplifying the installation process and reducing version mismatch risks. On Linux, PyTorch seems to use NVIDIA packages from the feed https://pypi.ngc.nvidia.com (installable with pip install nvidia-pyindex) (I did not double check this).

Possible Solutions

To streamline the installation process for onnxruntime-gpu, the following solutions could be considered:

1. Packaged CUDA DLLs with onnxruntime-gpu Wheels:

   • Create onnxruntime-gpu wheels that include CUDA DLLs, allowing users to install them conveniently with pip install onnxruntime-gpu[cuda_dlls].

2. Dependency Configuration via onnxruntime-gpu Wheel:

   • Create an onnxruntime-gpu wheel installable with pip install onnxruntime-gpu[cuda_dlls].
   • This wheel would list packages from the NVIDIA package index as install dependencies (e.g., nvidia-cudnn-cu12), which can be installed with pip install nvidia-pyindex followed by pip install nvidia-cudnn-cu12.
   • Configure onnxruntime to utilize these DLLs instead of those in CUDA_PATH.

The second solution may facilitate reuse of the same CUDA DLLs by other packages like CuPy or PyTorch, potentially reducing the overall size of the Python environment.

Describe scenario use case

• allow full automation of the the cuda dlls installation
• allow the use different cuda dlls versions in different python environment with editing CUDA_PATH

[snnn]

Due to some internal requirements, we are not allowed to use a second feed.

[snnn]

The CUDA DLLs are so huge that they cannot be hosted at pypi. So they must be hosted at somewhere else. However, due to some security concerns we are not allowed to use a second feed. So this went to a dead end.

[snnn]

I will keep this issue open, while we could continue discussing the details offline.

[martinResearch]

Is the problem that you are you not allowed to consume packages from the external feed https://pypi.ngc.nvidia.com/ in your development environment and/or in your CI tests pipelines (github actions) for security reasons?

[snnn]

In CI.

[martinResearch]

could you possibly add https://pypi.ngc.nvidia.com/ as an upstream to the feed you are currently using?

[snnn]

Azure DevOps Artifact's upstream feature doesn't support that. It only supports pypi. Correct me if I was wrong.

[martinResearch]

Indeed it seems I cannot use https://pypi.ngc.nvidia.com/ as an upstream in Azure Devops.
However It seems that all the required nvidia packages to get the required dlls for cuda 12 are actually on pypi.org, so not need to add https://pypi.ngc.nvidia.com/ as an upstream.

Here is a list of package on pypi.org that allowed me to get all the dlls I needed to use onnxruntime-gpu (some might actually not be required)

nvidia-nvjitlink-cu12
nvidia-nvtx-cu12
nvidia-cuda-runtime-cu12
nvidia-cublas-cu12
nvidia-cuda-cupti-cu12
nvidia-cuda-nvrtc-cu12
nvidia-cudnn-cu12
nvidia-cufft-cu12
nvidia-curand-cu12
nvidia-cusolver-cu12
nvidia-cusparse-cu12

so for cuda 12 it seems we could potentially get onnxruntime-gpu to use the dlls from these packages.
Unfortunately onnxruntime-gpu for cuda 12 is not on pypi.org see #19438, which I hope will be solved soon.