RuntimeError: CUDA error: no kernel image is available for execution on the device.

[AvonYangXX1]

After i install the conda environment, activate it and try to run the outlier model DL detection on the images ,it failed to run on the CUDA and output the error
RuntimeError: CUDA error: no kernel image is available for execution on the device.
I am quite confusing since my CUDA is correctly configured.

[vineet1992]

Do you mind sharing the outputs of "conda list" and "nvidia-smi"?

…

On Tue, Jul 9, 2024 at 8:48 AM AvonYangXX1 ***@***.***> wrote: After i install the conda environment, activate it and try to run the outlier model DL detection on the images ,it failed to run on the CUDA and output the error RuntimeError: CUDA error: no kernel image is available for execution on the device. I am quite confusing since my CUDA is correctly configured. — Reply to this email directly, view it on GitHub <#1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYGLAHRQG27VC5WWHVJ653ZLPLYFAVCNFSM6AAAAABKSZSFQSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM4TQMJSGIYTCNI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Vineet Raghu, PhD Assistant Professor of Radiology Massachusetts General Hospital and Harvard Medical School

[AvonYangXX1]

(Retina_Seg) [ayang071@holygpu7c26104 Retina-Seg]$ conda list

packages in environment at /n/home_fasse/ayang071/.conda/envs/Retina_Seg:

Name Version Build Channel

_libgcc_mutex 0.1 main anaconda
_openmp_mutex 5.1 1_gnu anaconda
beautifulsoup4 4.11.1 py37h06a4308_0 anaconda
blas 1.0 mkl anaconda
bottleneck 1.3.5 py37h7deecbd_0 anaconda
brotlipy 0.7.0 py37h27cfd23_1003 anaconda
bzip2 1.0.8 h7b6447c_0 anaconda
ca-certificates 2024.3.11 h06a4308_0
cairo 1.14.12 h8948797_3 anaconda
catalogue 2.0.7 py37h06a4308_0 anaconda
certifi 2022.12.7 py37h06a4308_0
cffi 1.15.1 py37h74dc2b5_0 anaconda
charset-normalizer 2.0.4 pyhd3eb1b0_0 anaconda
click 8.0.4 py37h06a4308_0 anaconda
colorama 0.4.6 py37h06a4308_0 anaconda
cryptography 39.0.1 py37h9ce1e76_0 anaconda
cuda-cudart 12.1.105 0 nvidia
cuda-cupti 12.1.105 0 nvidia
cuda-libraries 12.1.0 0 nvidia
cuda-nvrtc 12.1.105 0 nvidia
cuda-nvtx 12.1.105 0 nvidia
cuda-opencl 12.4.127 0 nvidia
cuda-runtime 12.1.0 0 nvidia
cudatoolkit 10.2.89 hfd86e86_1 anaconda
cycler 0.11.0 pyhd3eb1b0_0 anaconda
cymem 2.0.6 py37h295c915_0 anaconda
cython-blis 0.7.7 py37hce1f21e_0 anaconda
dataclasses 0.8 pyh6d0b6a4_7 anaconda
dbus 1.13.18 hb2f20db_0 anaconda
docopt 0.6.2 py37_0 anaconda
expat 2.5.0 h6a678d5_0 anaconda
fastai 1.0.61 1 fastai
fastprogress 0.2.2 py_0 fastai
ffmpeg 4.0 hcdf2ecd_0 anaconda
fontconfig 2.14.1 h52c9d5c_1 anaconda
freeglut 3.0.0 hf484d3e_5 anaconda
freetype 2.12.1 h4a9f257_0 anaconda
future 0.18.2 pypi_0 pypi
giflib 5.2.1 h5eee18b_3 anaconda
glib 2.56.2 hd408876_0 anaconda
graphite2 1.3.14 h295c915_1 anaconda
gst-plugins-base 1.14.0 hbbd80ab_1 anaconda
gstreamer 1.14.0 hb453b48_1 anaconda
harfbuzz 1.8.8 hffaf4a1_0 anaconda
hdf5 1.10.2 hba1933b_1 anaconda
icu 58.2 he6710b0_3 anaconda
idna 3.4 py37h06a4308_0 anaconda
importlib-metadata 4.11.3 py37h06a4308_0 anaconda
intel-openmp 2022.1.0 h9e868ea_3769 anaconda
jasper 2.0.14 hd8c5072_2 anaconda
jinja2 3.1.2 py37h06a4308_0 anaconda
jpeg 9e h5eee18b_1 anaconda
kiwisolver 1.4.4 py37h6a678d5_0 anaconda
langcodes 3.3.0 pyhd3eb1b0_0 anaconda
lcms2 2.12 h3be6417_0 anaconda
ld_impl_linux-64 2.38 h1181459_1 anaconda
lerc 3.0 h295c915_0 anaconda
libcublas 12.1.0.26 0 nvidia
libcufft 11.0.2.4 0 nvidia
libcufile 1.9.1.3 0 nvidia
libcurand 10.3.5.147 0 nvidia
libcusolver 11.4.4.55 0 nvidia
libcusparse 12.0.2.55 0 nvidia
libdeflate 1.17 h5eee18b_1 anaconda
libffi 3.3 he6710b0_2 anaconda
libgcc-ng 11.2.0 h1234567_1 anaconda
libgfortran-ng 7.5.0 ha8ba4b0_17 anaconda
libgfortran4 7.5.0 ha8ba4b0_17 anaconda
libglu 9.0.0 hf484d3e_1 anaconda
libgomp 11.2.0 h1234567_1 anaconda
libnpp 12.0.2.50 0 nvidia
libnvjitlink 12.1.105 0 nvidia
libnvjpeg 12.1.1.14 0 nvidia
libopencv 3.4.2 hb342d67_1 anaconda
libopus 1.3.1 h7b6447c_0 anaconda
libpng 1.6.39 h5eee18b_0 anaconda
libstdcxx-ng 11.2.0 h1234567_1 anaconda
libtiff 4.5.1 h6a678d5_0 anaconda
libuuid 1.41.5 h5eee18b_0 anaconda
libuv 1.44.2 h5eee18b_0
libvpx 1.7.0 h439df22_0 anaconda
libwebp 1.2.4 h11a3e52_1 anaconda
libwebp-base 1.2.4 h5eee18b_1 anaconda
libxcb 1.15 h7f8727e_0 anaconda
libxml2 2.9.14 h74e7548_0 anaconda
lz4-c 1.9.4 h6a678d5_0 anaconda
markupsafe 2.1.1 py37h7f8727e_0 anaconda
matplotlib 3.2.2 0 anaconda
matplotlib-base 3.2.2 py37hef1b27d_0 anaconda
mkl 2019.4 243 anaconda
mkl-service 2.3.0 py37he8ac12f_0 anaconda
mkl_fft 1.3.0 py37h54f3939_0 anaconda
mkl_random 1.1.0 py37hd6b4f25_0 anaconda
munch 2.5.0 pypi_0 pypi
murmurhash 1.0.7 py37h295c915_0 anaconda
ncurses 6.4 h6a678d5_0 anaconda
ninja 1.10.2 h06a4308_5 anaconda
ninja-base 1.10.2 hd09550d_5 anaconda
numexpr 2.7.3 py37hb2eb853_0 anaconda
numpy 1.18.5 py37ha1c710e_0 anaconda
numpy-base 1.18.5 py37hde5b4d6_0 anaconda
nvidia-ml-py3 7.352.0 py_0 fastai
opencv 3.4.2 py37h6fd60c2_1 anaconda
openssl 1.1.1w h7f8727e_0
packaging 22.0 py37h06a4308_0 anaconda
pandas 1.0.5 py37h0573a6f_0 anaconda
pathy 0.10.1 py37h06a4308_0 anaconda
pcre 8.45 h295c915_0 anaconda
pillow 9.4.0 py37h6a678d5_0 anaconda
pip 20.1.1 py37_1 anaconda
pixman 0.40.0 h7f8727e_1 anaconda
ply 3.11 py37_0 anaconda
preshed 3.0.6 py37h295c915_0 anaconda
pretrainedmodels 0.7.4 pypi_0 pypi
py-opencv 3.4.2 py37hb342d67_1 anaconda
pycparser 2.21 pyhd3eb1b0_0 anaconda
pydantic 1.9.0 py37h7f8727e_1 anaconda
pyopenssl 23.0.0 py37h06a4308_0 anaconda
pyparsing 3.0.9 py37h06a4308_0 anaconda
pyqt 5.9.2 py37h22d08a2_1 anaconda
pysocks 1.7.1 py37_1 anaconda
python 3.7.7 hcff3b4d_5 anaconda
python-dateutil 2.8.1 py_0 anaconda
pytorch 1.7.0 py3.7_cuda10.2.89_cudnn7.6.5_0 pytorch
pytorch-cuda 12.1 ha16c6d3_5 pytorch
pytz 2022.7 py37h06a4308_0 anaconda
pyyaml 6.0 py37h5eee18b_1 anaconda
qt 5.9.7 h5867ecd_1 anaconda
readline 8.2 h5eee18b_0 anaconda
requests 2.28.1 py37h06a4308_0 anaconda
scikit-learn 0.20.1 py37hd81dba3_0 anaconda
scipy 1.5.0 py37h0b6359f_0 anaconda
setuptools 65.6.3 py37h06a4308_0 anaconda
shellingham 1.5.0 py37h06a4308_0 anaconda
sip 6.6.2 py37h6a678d5_0 anaconda
six 1.16.0 pyhd3eb1b0_1 anaconda
smart_open 5.2.1 py37h06a4308_0 anaconda
soupsieve 2.3.2.post1 py37h06a4308_0 anaconda
spacy 3.3.1 py37h79cecc1_0 anaconda
spacy-legacy 3.0.9 py37h06a4308_0 anaconda
spacy-loggers 1.0.1 pyhd3eb1b0_0 anaconda
sqlite 3.41.2 h5eee18b_0 anaconda
srsly 2.4.3 py37h295c915_0 anaconda
thinc 8.0.15 py37hae6d005_0 anaconda
tk 8.6.12 h1ccaba5_0 anaconda
toml 0.10.2 pyhd3eb1b0_0 anaconda
torch 1.13.1+cu117 pypi_0 pypi
torchaudio 0.13.1+cu117 pypi_0 pypi
torchvision 0.14.1+cu117 pypi_0 pypi
tornado 6.2 py37h5eee18b_0 anaconda
tqdm 4.64.1 py37h06a4308_0 anaconda
typer 0.4.1 py37h06a4308_0 anaconda
typing-extensions 3.10.0.2 hd3eb1b0_0 anaconda
typing_extensions 3.10.0.2 pyh06a4308_0 anaconda
urllib3 1.26.14 py37h06a4308_0 anaconda
wasabi 0.9.1 py37h06a4308_0 anaconda
wheel 0.38.4 py37h06a4308_0 anaconda
xz 5.4.2 h5eee18b_0 anaconda
yaml 0.2.5 h7b6447c_0 anaconda
zipp 3.11.0 py37h06a4308_0 anaconda
zlib 1.2.13 h5eee18b_0 anaconda
zstd 1.5.5 hc292b87_0 anaconda

(Retina_Seg) [ayang071@holygpu7c26104 Retina-Seg]$ nvidia-smi
Tue Jul 9 10:11:41 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.02 Driver Version: 555.42.02 CUDA Version: 12.5 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100-SXM4-40GB On | 00000000:31:00.0 Off | 0 |
| N/A 24C P0 49W / 400W | 1MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA A100-SXM4-40GB On | 00000000:4B:00.0 Off | 0 |
| N/A 25C P0 50W / 400W | 1MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA A100-SXM4-40GB On | 00000000:CA:00.0 Off | 0 |
| N/A 25C P0 50W / 400W | 1MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA A100-SXM4-40GB On | 00000000:E3:00.0 Off | 0 |
| N/A 24C P0 49W / 400W | 1MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

Thank you so much for checking