Skip to content

[BUG] Unable to run Nvidia morpheus-experimental example using cuml-cpu #6349

Open
@lazydog2

Description

@lazydog2

Describe the bug
Attempting to run one of the Nvidia morpheus-experimental examples, specifically the ids-detection inference example (https://github.com/nv-morpheus/morpheus-experimental/tree/branch-25.06/ids-detection), using cuml-cpu there is an error when cuml metrics attempts to import the trustworthiness module:

File miniconda3/envs/python-3.10/lib/python3.10/site-packages/cuml/metrics/__init__.py", line 17, in
from cuml.metrics.trustworthiness import trustworthiness
ModuleNotFoundError: No module named 'cuml.metrics.trustworthiness'

In the latest versions of cuml-cpu, the trustworthiness.pyx file exists but it doesn't exist in earlier cuml-cpu versions.

I've attempted to configure cython to compile the trustworthiness.pyx file by adding the following to miniconda3/envs/python-3.11/lib/python3.11/site-packages/cuml/__init__.py and installing the necessary dependencies but the compilation fails with, "‘cuda::mr’ has not been declared" because, I believe, the rmm mr namespace functions refer to types defined within the mr namespace itself (e.g. https://github.com/rapidsai/rmm/blob/branch-25.04/include/rmm/mr/device/device_memory_resource.hpp#L311).

import pyximport
pyximport.install(setup_args={"script_args" : ["--verbose"]})

If I install the full cuml and run the morpheus-experimental ids-detection inference example, then I get the following error:

File "miniconda3/envs/rapids-25.04/lib/python3.12/site-packages/cudf/io/csv.py", line 277, in read_csv
table_w_meta = plc.io.csv.read_csv(options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "csv.pyx", line 631, in pylibcudf.io.csv.read_csv
File "csv.pyx", line 649, in pylibcudf.io.csv.read_csv
MemoryError: std::bad_alloc: CUDA error at: miniconda3/envs/rapids-25.04/include/rmm/mr/device/cuda_memory_resource.hpp

Steps/Code to reproduce bug
Using cuml-cpu
conda create -n python-3.11 python=3.11
conda activate python-3.11
mamba install -c rapidsai -c nvidia -c conda-forge cuml-cpu click matplotlib cudf cython pylibraft cccl libcuvs libcuml
python inference.py

Using cuml
conda create --solver=libmamba -n rapids-25.04 -c rapidsai-nightly -c conda-forge -c nvidia rapids=25.04 python=3.12 'cuda-version>=11.4,<=11.8'
conda activate rapids-25.04
python inference.py

Expected behavior
The ids-detection morpheus-experimental example should run successfully.

Environment details:

  • Environment location: Amazon Workspace
  • Linux Distro/Architecture: Ubuntu 22.04.4 LTS
  • GPU Model/Driver: N/A
  • CUDA: N/A
  • Method of cuDF & cuML install: conda

Additional information:
The CIC-IDS-2017 dataset used by the morpheus-experimental ids-detection example can be found here:
http://205.174.165.80/CICDataset/CIC-IDS-2017/Dataset/CIC-IDS-2017/CSVs/GeneratedLabelledFlows.zip

Metadata

Metadata

Assignees

No one assigned

    Labels

    ? - Needs TriageNeed team to review and classifybugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions