[BUG] cuml-cpu 25.04 fails to import without a GPU present

[jameslamb]

Describe the bug

Working on #6400, I encountered multiple errors when trying to import cuml installed via the cuml-cpu conda package in an environment without a GPU.

Documenting those cases here.

Steps/Code to reproduce bug

Ran the following on an M2 (arm64) macbook pro:

docker run \
    --rm \
    -it rapidsai/ci-conda:latest \
    bash

conda create \
    --name cuml-test \
    'cuml-cpu==25.4.*' \
    'python=3.11'

source activate cuml-test

Environment creation succeeded.

details (click me)

Output of conda info

     active environment : cuml-test                                                                                                                                                                     
    active env location : /opt/conda/envs/cuml-test                                                                                                                                                     
            shell level : 2                                                                                                                                                                             
       user config file : /root/.condarc                                                                                                                                                                
 populated config files : /opt/conda/.condarc                                                                                                                                                           
          conda version : 24.7.1                                                                                                                                                                        
    conda-build version : 24.5.1                                                                                                                                                                        
         python version : 3.12.5.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=aarch64
                          __conda=24.7.1=0
                          __glibc=2.35=0
                          __linux=6.5.0=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/rapidsai/linux-aarch64
                          https://conda.anaconda.org/rapidsai/noarch
                          https://conda.anaconda.org/rapidsai-nightly/linux-aarch64
                          https://conda.anaconda.org/rapidsai-nightly/noarch
                          https://conda.anaconda.org/dask/label/dev/linux-aarch64
                          https://conda.anaconda.org/dask/label/dev/noarch
                          https://conda.anaconda.org/pytorch/linux-aarch64
                          https://conda.anaconda.org/pytorch/noarch
                          https://conda.anaconda.org/conda-forge/linux-aarch64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/nvidia/linux-aarch64
                          https://conda.anaconda.org/nvidia/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-aarch64
             user-agent : conda/24.7.1 requests/2.32.3 CPython/3.12.5 Linux/6.5.0-44-generic ubuntu/22.04.5 glibc/2.35 solver/libmamba conda-libmamba-solver/24.7.0 libmambapy/1.5.9
                UID:GID : 0:0
             netrc file : None
           offline mode : False

Output of conda env export --name cuml-test

name: cuml-test
channels:
  - rapidsai
  - rapidsai-nightly
  - dask/label/dev
  - pytorch
  - conda-forge
  - nvidia
dependencies:
  - _openmp_mutex=4.5=2_gnu
  - bzip2=1.0.8=h68df207_7
  - ca-certificates=2025.1.31=hcefe29a_0
  - colorama=0.4.6=pyhd8ed1ab_1
  - cuml-cpu=25.04.00a95=py311_250306_gee276d5f2_95
  - hdbscan=0.8.38.post2=py311hec9beba_0
  - icu=75.1=hf9b3779_0
  - joblib=1.4.2=pyhd8ed1ab_1
  - ld_impl_linux-aarch64=2.43=h80caac9_4
  - libblas=3.9.0=31_h1a9f1db_openblas
  - libcblas=3.9.0=31_hab92f65_openblas
  - libexpat=2.6.4=h5ad3122_0
  - libffi=3.4.6=he21f813_0
  - libgcc=14.2.0=he277a41_2
  - libgcc-ng=14.2.0=he9431aa_2
  - libgfortran=14.2.0=he9431aa_2
  - libgfortran5=14.2.0=hb6113d0_2
  - libgomp=14.2.0=he277a41_2
  - libhwloc=2.11.2=default_h2c612a5_1001
  - libiconv=1.18=hc99b53d_1
  - liblapack=3.9.0=31_h411afd4_openblas
  - libllvm15=15.0.7=hcc6320c_5
  - liblzma=5.6.4=h86ecc28_0
  - libnsl=2.0.1=h31becfc_0
  - libopenblas=0.3.29=pthreads_h9d3fd7e_0
  - libsqlite=3.49.1=h5eb1b54_1
  - libstdcxx=14.2.0=h3f4de04_2
  - libstdcxx-ng=14.2.0=hf1166c9_2
  - libuuid=2.38.1=hb4cce97_0
  - libxcrypt=4.4.36=h31becfc_1
  - libxml2=2.13.6=h2e0c361_0
  - libzlib=1.3.1=h86ecc28_2
  - llvmlite=0.44.0=py311hbc3773b_0
  - ncurses=6.5=ha32ae93_3
  - numba=0.61.0=py311h26ae67a_1
  - numpy=1.26.4=py311h69ead2a_0
  - nvtx=0.2.11=py311h5487e9b_0
  - openssl=3.4.1=hd08dc88_0
  - pandas=2.2.3=py311h848c333_2
  - pip=25.0.1=pyh8b19718_0
  - pynndescent=0.5.13=pyhd8ed1ab_1
  - python=3.11.11=h1683364_2_cpython
  - python-dateutil=2.9.0.post0=pyhff2d567_1
  - python-tzdata=2025.1=pyhd8ed1ab_0
  - python_abi=3.11=5_cp311
  - pytz=2024.1=pyhd8ed1ab_0
  - readline=8.2=h8382b9d_2
  - scikit-learn=1.2.2=py311hb93614b_2
  - scipy=1.15.2=py311h2973cce_0
  - setuptools=75.8.2=pyhff2d567_0
  - six=1.17.0=pyhd8ed1ab_0
  - tbb=2022.0.0=h243be18_0
  - threadpoolctl=3.5.0=pyhc1e730c_0
  - tk=8.6.13=h194ca79_0
  - tqdm=4.67.1=pyhd8ed1ab_1
  - tzdata=2025a=h78e105d_0
  - umap-learn=0.5.6=py311hec3470c_1
  - wheel=0.45.1=pyhd8ed1ab_1
prefix: /opt/conda/envs/cuml-test

Tried to import like this:

python -c "import cuml"

That failed like this, because the package uses packaging unconditionally.

ModuleNotFoundError: No module named 'packaging'

full traceback (click me)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/__init__.py", line 27, in <module>
    from cuml.internals.base import Base, UniversalBase
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/__init__.py", line 17, in <module>
    from cuml.internals.available_devices import is_cuda_available
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/available_devices.py", line 16, in <module>
    from cuml.internals.device_support import GPU_ENABLED
  File "device_support.pyx", line 18, in init cuml.internals.device_support
ModuleNotFoundError: No module named 'packaging'

So tried installing that and importing again.

conda install --yes packaging
python -c "import cuml"

That fails again, this time because of an import of cupy.

...
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/safe_imports.py", line 86, in __getattr__
    raise UnavailableError(cls._msg)
cuml.internals.safe_imports.UnavailableError: cupy is not installed in non GPU-enabled installations

full traceback (click me)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/__init__.py", line 116, in <module>
    from cuml.linear_model.linear_regression import LinearRegression
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/linear_model/__init__.py", line 21, in <module>
    from cuml.linear_model.logistic_regression import LogisticRegression
  File "logistic_regression.pyx", line 27, in init cuml.linear_model.logistic_regression
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/preprocessing/__init__.py", line 16, in <module>
    from cuml.model_selection import train_test_split
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/model_selection/__init__.py", line 17, in <module>
    from cuml.model_selection._split import train_test_split
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/model_selection/_split.py", line 41, in <module>
    indices: cp.ndarray,
             ^^^^^^^^^^
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/safe_imports.py", line 86, in __getattr__
    raise UnavailableError(cls._msg)
cuml.internals.safe_imports.UnavailableError: cupy is not installed in non GPU-enabled installations

So tried installing that and importing again.

conda install --yes cupy
python -c "import cuml"

That fails again, this time because of an import of cudf

  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/safe_imports.py", line 86, in __getattr__
    raise UnavailableError(cls._msg)
cuml.internals.safe_imports.UnavailableError: cudf is not installed in non GPU-enabled installations

full traceback (click me)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/__init__.py", line 116, in <module>
    from cuml.linear_model.linear_regression import LinearRegression
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/linear_model/__init__.py", line 21, in <module>
    from cuml.linear_model.logistic_regression import LogisticRegression
  File "logistic_regression.pyx", line 27, in init cuml.linear_model.logistic_regression
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/preprocessing/__init__.py", line 17, in <module>
    from cuml.preprocessing.LabelEncoder import LabelEncoder
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/preprocessing/LabelEncoder.py", line 56, in <module>
    class LabelEncoder(Base):
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/preprocessing/LabelEncoder.py", line 212, in LabelEncoder
    def transform(self, y) -> cudf.Series:
                              ^^^^^^^^^^^
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/safe_imports.py", line 86, in __getattr__
    raise UnavailableError(cls._msg)
cuml.internals.safe_imports.UnavailableError: cudf is not installed in non GPU-enabled installations

Tried installing that and importing again.

conda install --yes 'cudf==25.4.*'
python -c "import cuml"

That failed like this:

numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: 
...
CUDA driver library cannot be found.
...
/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cudf/utils/gpu_utils.py:62: UserWarning: Failed to dlopen libcuda.so.1

full traceback (click me)

/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cudf/utils/_ptxcompiler.py:64: UserWarning: Error getting driver and runtime versions:

stdout:

stderr:

Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 295, in __getattr__
    raise CudaSupportError("Error at driver init: \n%s:" %
numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: 

CUDA driver library cannot be found.
If you are sure that a CUDA driver is installed,
try setting environment variable NUMBA_CUDA_DRIVER
with the file path of the CUDA driver shared library.
:
Not patching Numba
  warnings.warn(msg, UserWarning)
/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cudf/utils/gpu_utils.py:62: UserWarning: Failed to dlopen libcuda.so.1
  warnings.warn(str(e)

Expected behavior

Expected to be able to use cuml-cpu in an environment

Environment details (please complete the following information):

See "Steps/Code to reproduce bug" section.

Additional context

This report might be related to these others:

• [BUG] Unable to run Nvidia morpheus-experimental example using cuml-cpu #6349
• [BUG] umap_utils fails to import in cuml-cpu #6341
• [BUG] cuml-cpu conda install broken #6340

From their contents, I'm not sure.

[jameslamb]

Confirmed today that this issue only affects 25.04. It is not present in other releases from 24.02-25.02.

testing code (click me)

docker run \
    --rm \
    -it rapidsai/ci-conda:cuda12.0.1-ubuntu22.04-py3.10 \
    bash

test-version(){
    local version=${1}
    local extra_packages="${2}"
    # delete conda env if it exists
    source deactivate || true
    conda env remove --name "cuml-test-${version}" || true
    
    # create and activate new env
    conda create \
        --name cuml-test-${version} \
        "cuml-cpu==${version}.*" \
        "python=3.10" \
        ${extra_packages}

    source activate cuml-test-${version}

    # try to import cuml
    python -c "import cuml; print(cuml.__version__)"

    echo $?
}

test-version "25.4"
# ModuleNotFoundError: No module named 'packaging'

test-version "25.4" "packaging"
# cuml.internals.safe_imports.UnavailableError: cupy is not installed in non GPU-enabled installations

test-version "25.4" "cupy packaging"
# cuml.internals.safe_imports.UnavailableError: cudf is not installed in non GPU-enabled installations

test-version "25.4" "cudf cupy packaging"
# 25.04.00a99
# 0

test-version "25.2"
# ModuleNotFoundError: No module named 'packaging'

test-version "25.2" "packaging"
# 25.02.01
# 0

test-version "24.12"
# ModuleNotFoundError: No module named 'packaging'

test-version "24.12" "packaging"
# 24.12.00
# 0

test-version "24.10"
# ModuleNotFoundError: No module named 'packaging'

test-version "24.10" "packaging"
# 24.10.00
# 0

test-version "24.08"
# ModuleNotFoundError: No module named 'packaging'

test-version "24.08" "packaging"
# 24.08.00
# 0

test-version "24.06" "packaging"
# 24.06.01
# 0

test-version "24.04" "packaging"
# 24.04.00
# 0

test-version "24.02" "packaging"
# 24.02.00
# 0

All recent releases suffer from the issue of not declaring their runtime dependency on packaging. That will be fixed by #6400, but not a huge problem... it's likely to be satisfied transitively by other packages people install alongside cuml, and the remedy is otherwise just "you have to manually install packaging".

But only 25.04 has this issue with GPU-only imports in a CPU-only environment. Something must have changed somewhere in the import machinery. Every failure like this I've observed so far has been in type hints referencing cupy and cudf types.

I suspect using deferred evaluation of type hints could help:

• https://peps.python.org/pep-0649/
• https://mypy.readthedocs.io/en/stable/runtime_troubles.html

[jameslamb]

I think there's a little more than just type-hints stuff going on here. Just observed a different import error on #6400, unrelated to type hints:

File "/opt/conda/conda-bld/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.10/site-packages/cuml/thirdparty_adapters/adapters.py", line 17, in
from cupyx.scipy import sparse as gpu_sparse
ModuleNotFoundError: No module named 'cupyx'
WARNING: Tests failed for cuml-cpu-25.04.00a107-py310_250310_g51a40785d_107.conda - moving package to /opt/conda/conda-bld/broken

https://github.com/rapidsai/cuml/actions/runs/13767714380/job/38500670811?pr=6400#step:9:2323

That one looks like it should be a GPU-only import, based on how cupyx is handled in the rest of the package.

https://github.com/rapidsai/cuml/blame/c6a03b7d93d6dce7d78c0972a901cf4c15387bc8/python/cuml/cuml/thirdparty_adapters/adapters.py#L17

But... it's been there without being wrapped in gpu_only_import() for 4+ years, since #4196, so not sure why it's showing up as an import-time issue now.

[jameslamb]

Ok yes, this definitely goes deeper than just type hints.

Here's another example.... just saw this on #6400

File "lib/python3.10/site-packages/cuml/linear_model/__init__.py", line 21, in <module>
  from cuml.linear_model.logistic_regression import LogisticRegression
File "logistic_regression.pyx", line 27, in init cuml.linear_model.logistic_regression
File "lib/python3.10/site-packages/cuml/preprocessing/__init__.py", line 21, in <module>
  from cuml.preprocessing import text
File "lib/python3.10/site-packages/cuml/preprocessing/text/__init__.py", line 17, in <module>
  from cuml.preprocessing.text import stem
File "lib/python3.10/site-packages/cuml/preprocessing/text/stem/__init__.py", line 17, in <module>
  from cuml.preprocessing.text.stem.porter_stemmer import PorterStemmer
File "lib/python3.10/site-packages/cuml/preprocessing/text/stem/porter_stemmer.py", line 37, in <module>
  from .porter_stemmer_utils.suffix_utils import (
File "lib/python3.10/site-packages/cuml/preprocessing/text/stem/porter_stemmer_utils/suffix_utils.py", line 87, in <module>
  @cudf.core.buffer.acquire_spill_lock()
File "lib/python3.10/site-packages/cuml/internals/safe_imports.py", line 86, in __getattr__
  raise UnavailableError(cls._msg)
cuml.internals.safe_imports.UnavailableError: cudf is not installed in non GPU-enabled installations

(build link)

I guess this decorator also triggers this:

cuml/python/cuml/cuml/preprocessing/text/stem/porter_stemmer_utils/suffix_utils.py

Line 87 in 29871eb

@cudf.core.buffer.acquire_spill_lock()

That's been there for 2+ years, since #5154