BUG: libcustatevec depends on `libnvidia-ml.so`, doesn't ensure it's present

[h-vetinari]

This shows up in the logs here

WARNING (custatevec,lib/libcustatevec.so.1.11.0.11): $RPATH/libnvidia-ml.so.1 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?

as a warning, but turns into an actual failure in conda-forge/qsimcirq-feedstock#29, where a python package depends on

   INFO (qsimcirq,lib/python3.12/site-packages/qsimcirq/qsim_custatevec.cpython-312-x86_64-linux-gnu.so): Needed DSO lib/libcustatevec.so.1 found in conda-forge/linux-64::custatevec==1.11.0=h58dd1b1_0

but then can't find libnvidia-ml.so.

    qsim_custatevec = importlib.import_module("qsimcirq.qsim_custatevec")
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PREFIX/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: libnvidia-ml.so.1: cannot open shared object file: No such file or directory

That's despite cuda-nvml-dev being in the test environment, but that output has the libraries in other places than $PREFIX/lib.

I'm not sure what the right way to slice this is, but to me the clearest fix seems to be that, if libcustatevec depends on libnvidia-ml.so, it needs to ensure (even just for itself) that this library is present and findable in the environment.

CC @conda-forge/cuquantum

[leofang]

NVML is part of the driver package. It is not installable from conda (same as libcuda) or any other user-space package manager.

[h-vetinari]

NVML is part of the driver package. It is not installable from conda (same as libcuda)

Ah, that explains it, thanks! I wasn't aware that there are other libraries like libcuda.so that we cannot ship. In that case I guess we can close this issue. Though it'd be nice to document all the "unshippable" CUDA libraries in https://github.com/conda-forge/conda-forge.github.io somewhere

[h-vetinari]

Obviously, this is not a great situation though; if we can't even do import tests of packages depending on cuquantum anymore... 😑

[leofang]

Though it'd be nice to document all the "unshippable" CUDA libraries in https://github.com/conda-forge/conda-forge.github.io somewhere

Yes. If you could create an issue to track this either in the CF-GH-IO feedstock or CUDA feedstock, and ping the CUDA team for awareness, it'd be nice 🙏

if we can't even do import tests of packages depending on cuquantum anymore...

Yeah it is not fun. In the past for every single library that directly links to libcuda, I taught people to set up a workaround like this for import tests:
https://github.com/conda-forge/qiskit-aer-feedstock/blob/c01ae1490a1c237251ac084551fcd7273f679a8a/recipe/meta.yaml#L105-L106
the basic idea is to preload the stub (from cuda-driver-dev) so that we make the linker happy.

I think we could do the same thing with libnvidia-ml, whose stub is provided by cuda-nvml-dev:
https://github.com/conda-forge/cuda-nvml-dev-feedstock/blob/bd476bdbbe82632016c1f4e74b67f774a6b9f395/recipe/meta.yaml#L51

However, this has two issues I've encountered in the past:

1. This stub-based workaround has to be propagated manually to all downstream feedstocks (ex: cuStateVec affects qsimcirq, qiskit, etc).
2. Some projects would still fail the import test further down the road because the (nonfunctional) stub is loaded but they assume that loading passes == driver/GPU exists, and start using the stub and explode.

The ultimate solution, from the packaging perspective, is to dlopen any driver libraries. But this is not something we can solve at the feedstock level. The upstream (aka cuQuantum in this case) will need to recognize it's an issue and fix it.

[h-vetinari]

Thanks for the tip @leofang, the LD_PRELOAD trick worked to at least pass the import test in conda-forge/qsimcirq-feedstock#29!

[leofang]

It seems NVML is no longer needed! #101 (comment)