CUBLAS_STATUS_INVALID_VALUE

[lxy1998]

When I run the code below ,it throws an error CUBLAS_STATUS_INVALID_VALUE and terminates. tsnecuda.test() can run Correctly.

from tsnecuda import TSNE
import pickle

if __name__ == '__main__':
    # tsnecuda.test()
    
    X = pickle.load(open('/home/featurize/data/emb_all_768/emb_all_768.data', 'rb'))
    print(X.shape)
    # (1694135, 768)

    '''t-SNE-cuda'''
    X_tsne = TSNE(n_components=2, perplexity=15, learning_rate=10).fit_transform(X)
    pickle.dump(X_tsne, open("/home/featurize/data/tsne_result.data", 'wb'), -1)

Error below:

CUBLAS error in file '/tsnecuda/src/util/cuda_utils.cu', line 101, error: CUBLAS_STATUS_INVALID_VALUE
terminating!
python: /tsnecuda/src/util/cuda_utils.cu:103: void __CublasSafeCall(cublasStatus_t, const char*, int): Assertion `0' failed.
[2]    23531 abort (core dumped)  python /home/featurize/work/EventTriplesExtraction/t_sne_cuda.py

Please tell me what causes this error and how to fix it, thank you!

[SeanPedersen]

Same issue here. tsnecuda.test() works but using a bigger dataset causes a crash. I monitored VRAM and it was not maxxed out.

Error:

Loaded: (3039767, 128)
Initializing cuda handles... done.
KNN Computation... done.
Computing Pij matrix...
CUBLAS error in file '/tsnecuda/src/util/cuda_utils.cu', line 101, error: CUBLAS_STATUS_INVALID_VALUE
terminating!
python: /tsnecuda/src/util/cuda_utils.cu:103: void __CublasSafeCall(cublasStatus_t, const char*, int): Assertion `0' failed.
/commands/python: line 37: 41706 Aborted                 (core dumped) "$@"

Version:

sudo apt-get install intel-mkl cuda-toolkit-11-3
pip3 install tsnecuda==3.0.1+cu113 -f https://tsnecuda.isx.ai/tsnecuda_stable.html

[SeanPedersen]

Ok simply using conda install tsnecuda -c conda-forge instead of the pip install fixed it for me...