cuBLAS API Call Fails with CUBLAS_STATUS_INTERNAL_ERROR (14) When Using Intercept Library

[shaoxun6033]

During testing, we called cuBLAS APIs but encountered cuBLAS Error: 14 (CUBLAS_STATUS_INTERNAL_ERROR) when using an intercept library. Without the intercept library, the code runs successfully. We suspect the issue may be related to the intercept library's implementation or configuration, but the exact cause is unclear. Please assist in diagnosing the issue or suggest potential solutions. Thank you.

for example：

cublasHandle_t handle;
cublasCreate(&handle);

cuBLAS Error: 14

[Naplesoul]

Hi, could you please provide some concrete hardware configuration and software environment information (e.g., cuda version, cuBLAS version) so that we can reproduce this issue? If convenient, please also provide your code.

[shaoxun6033]

Our system configuration is as follows:

• OS: Ubuntu 20.04.6 LTS
• CUDA Toolkit: 12.8
• cuBLAS: 12.8
• CUDA Driver: 12.8
• GPU: NVIDIA RTX 4090 24G

We are using the following command to compile our code:

nvcc -o matrix_multiplication matrix_multiplication.cu -lcublas

The interception library we are using is from https://github.com/XpuOS/cuda-intercept. When we run a simple example like hello.cu, the interception is successful, and we see the following output:

[INFO @ T857882 @ 17:01:50.191463] cuInit called
[INFO @ T857882 @ 17:01:50.339868] Launch kernel: 0x5586880c79c0, grid (1, 1, 1), block (1, 1, 1), sharedMemBytes 0, stream (nil)
[INFO @ T857882 @ 17:01:50.343167] cuCtxSynchronize called
Hello, World!

However, when we run our matrix_multiplication.cu code with the interception library, we encounter a cuBLAS error. Here is the output:

Performing matrix multiplication C(256x128) = A(256x512) * B(512x128)
Running simple matrix multiplication on CPU...
CPU computation finished.
Running matrix multiplication using CUBLAS on GPU...
[INFO @ T858284 @ 17:07:49.432707] cuInit called
[INFO @ T858284 @ 17:07:49.563955] cuInit called
CUBLAS Error at matrix_multiplication.cu:98 - Status code: 14
The program fails with CUBLAS_STATUS_INTERNAL_ERROR (code 14).

Without the interception library, the program runs successfully:

Comparing a custom CUDA kernel with CUBLAS library.
Matrix dimensions: C(512x512) = A(512x512) * B(512x512)

1. Running custom CUDA kernel...
   Custom kernel execution time: 0.19488 ms
2. Running CUBLAS sgemm...
   CUBLAS sgemm execution time: 54.2454 ms
3. Verifying results...
   SUCCESS: The results from the original kernel and CUBLAS match.
   Cleaning up resources.

We would be very grateful for any help in analyzing why the cuBLAS library cannot be intercepted correctly. Thank you!

kernel_launch.zip

[Naplesoul]

We tested a simple cuBLAS app with XSched interception and it works fine. This issue may only belong to the cuda-intercept repo. We have reproduced it and will fix it soon. BTW, kernel_launch.zip seems to be an empty zip file.

[shaoxun6033]

Thank you very much for your prompt reply. Regarding the kernel_launch.zip file, I'm sorry. We didn't check it carefully. We will use XSched interception and the cuda-intercept repo that you have fixed. Reconduct the experiment.