CUDA out of memory/crashing on NVDIA GeForce RTX 5090

[icombs2017]

Good morning,

I have been trying to troubleshoot this issue our system has been having with TagLab for a few weeks now, I found some forums online saying that the 5000 series chips from NVDIA are having some compatibility issues with pytorch, so I think that may be part of my issue here. I have this version of TagLab installed and working on all of our other machines besides this one with the 5090 GPU. I still installed it in the conda environment, and even tried to specify installing the CPU version since I think part of the problem is the GPU. This is the error messaging I am getting along with a "CUDA out of memory" message when using some of the semiautomatic tools despite zooming in on the image:

C:\Users\Restoration\Documents\GitHub\TagLab\models\isegm\inference\utils.py:22: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state_dict = torch.load(checkpoint, map_location='cpu')
C:\Users\Restoration.conda\envs\taglab11\Lib\site-packages\torch\cuda_init_.py:235: UserWarning:
NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_61 sm_70 sm_75 sm_80 sm_86 sm_90.
If you want to use the NVIDIA GeForce RTX 5090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

warnings.warn(

Is there a work around for this?

Thanks!

[Jordan-Pierce]

@icombs2017

This is just a warning, ignore:

C:\Users\Restoration\Documents\GitHub\TagLab\models\isegm\inference\utils.py:22: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.

state_dict = torch.load(checkpoint, map_location='cpu')

This is the actual "issue":

C:\Users\Restoration.conda\envs\taglab11\Lib\site-packages\torch\cuda_init_.py:235: UserWarning:
NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_61 sm_70 sm_75 sm_80 sm_86 sm_90.
If you want to use the NVIDIA GeForce RTX 5090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

Basically you're NVIDIA GPU (5090, noice!) is too new for the version of pytorch + CUDA you have installed on your computer / conda environment.

See this this thread. You might just need to upgrade your version of pytorch + CUDA (see here)

Can you confirm which version of CUDA you have on your computer currently? Try nvidia-smi, or searching for the version in C:\Program Files or C:\Program Files x86

[icombs2017]

@Jordan-Pierce

Currently this computer is using CUDA version 12.8, but let me know if you think updating it to the latest version would help as well. I will take a look at that thread now.

Thanks!

[icombs2017]

@Jordan-Pierce downloaded cuda 13,

PS C:\Users\Restoration> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Jul_16_20:06:48_Pacific_Daylight_Time_2025
Cuda compilation tools, release 13.0, V13.0.48
Build cuda_13.0.r13.0/compiler.36260728_0
PS C:\Users\Restoration>

however when I run nvidia-smi it still lists 12.8 and when pulling up taglab it is still using 12.8

[Jordan-Pierce]

@icombs2017 it wouldn't matter, because when you re-install pytorch + CUDA for a valid version of CUDA, 13.0 isn't supported:

The goal is to install pytorch + CUDA from Pytorch (via pip) for a version of CUDA that is currently supported (12.8)

I don't think the issue was ever CUDA, it was that you have Pytorch + CUDA in your python environment that isn't aligned with your GPU.

Try for me the following in your taglab11 conda environment:

pip uninstall torch torchvision -y
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128 -U

[icombs2017]

Fixed, thank you so much, @Jordan-Pierce !!