Batched strided GEMM tests fail

[jasontuna]

Pkg.test("CuArrays") fails, and I was able to locate the source of the error to the test below. (All other tests pass if I remove this test)

@testset "gemm_strided_batched with element type $elty" for elty in [Float32, Float64, ComplexF32, ComplexF64]

The function CuArrays.CUBLAS.gemm_strided_batched! fails with repeated calls.
The function CuArrays.CUBLAS.gemm_batched! works will repeated calls.

I think this is because there is an issue with using A::CuArray{$elty, 3} in the call to

@check ccall(($(string(fname)), libcublas), cublasStatus_t,
(cublasHandle_t, cublasOperation_t,
cublasOperation_t, Cint, Cint, Cint, Ptr{$elty},
Ptr{$elty}, Cint, Cint, Ptr{$elty}, Cint, Cint, Ptr{$elty},
Ptr{$elty}, Cint, Cint, Cint),
handle(), cutransA,
cutransB, m, n, k, [alpha], A, lda, strideA, B, ldb, strideB, [beta],
C, ldc, strideC, batchCount)
C

I can recreate the error above by running the following in jupyter

using  CUDAnative, CuArrays
using CuArrays.CUBLAS

m = 20
n = 35
k = 13
elty= Float32
nbatch = 10

# generate matrices
alpha = one(elty)
beta = one(elty)
A = rand(elty, m, k, nbatch)
B = rand(elty, k, n, nbatch)
C = rand(elty, m, n, nbatch)

# move to device
d_A = CuArray{elty, 3}(A)
d_B = CuArray{elty, 3}(B)
d_C = CuArray{elty, 3}(C)

CuArrays.CUBLAS.gemm_strided_batched!('N', 'N', alpha, d_A, d_B, beta, d_C)

CuArrays.CUBLAS.gemm_strided_batched!('N', 'N', alpha, d_A, d_B, beta, d_C)

Here is the error from running the test

gemm_strided_batched with element type Float64: Error During Test at "juliahome"\.julia\dev\CuArrays\test\blas.jl:659
  Got exception outside of a @test
  CUDA error: an illegal memory access was encountered (code JuliaGPU/CUDA.jl#85, ERROR_ILLEGAL_ADDRESS)
  Stacktrace:
   [1] macro expansion at "juliahome"\.julia\packages\CUDAdrv\LC5XS\src\base.jl:147 [inlined]
   [2] #download!#11(::Bool, ::Function, ::Ptr{Float64}, ::CUDAdrv.Mem.Buffer, ::Int64, ::CUDAdrv.CuStream) at "juliahome"\.julia\packages\CUDAdrv\LC5XS\src\memory.jl:254
   [3] download! at "juliahome"\.julia\packages\CUDAdrv\LC5XS\src\memory.jl:248 [inlined] (repeats 2 times)
   [4] unsafe_copyto! at "juliahome"\.julia\dev\CuArrays\src\array.jl:166 [inlined]
   [5] copyto!(::Array{Float64,3}, ::CuArray{Float64,3}) at "juliahome"\.julia\packages\GPUArrays\HmVTY\src\abstractarray.jl:110
   [6] Array(::CuArray{Float64,3}) at .\array.jl:497
   [7] macro expansion at "juliahome"\.julia\dev\CuArrays\test\blas.jl:690 [inlined]
   [8] macro expansion at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Test\src\Test.jl:1156 [inlined]
   [9] macro expansion at "juliahome"\.julia\dev\CuArrays\test\blas.jl:659 [inlined]
   [10] macro expansion at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Test\src\Test.jl:1083 [inlined]
   [11] macro expansion at "juliahome"\.julia\dev\CuArrays\test\blas.jl:546 [inlined]
   [12] macro expansion at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Test\src\Test.jl:1083 [inlined]
   [13] top-level scope at "juliahome"\.julia\dev\CuArrays\test\blas.jl:3
   [14] include at .\boot.jl:317 [inlined]
   [15] include_relative(::Module, ::String) at .\loading.jl:1044
   [16] include(::Module, ::String) at .\sysimg.jl:29
   [17] include(::String) at .\client.jl:392
   [18] macro expansion at "juliahome"\.julia\dev\CuArrays\test\runtests.jl:21 [inlined]
   [19] macro expansion at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Test\src\Test.jl:1083 [inlined]
   [20] top-level scope at "juliahome"\.julia\dev\CuArrays\test\runtests.jl:19
   [21] include at .\boot.jl:317 [inlined]
   [22] include_relative(::Module, ::String) at .\loading.jl:1044
   [23] include(::Module, ::String) at .\sysimg.jl:29
   [24] include(::String) at .\client.jl:392
   [25] top-level scope at none:0
   [26] eval(::Module, ::Any) at .\boot.jl:319
   [27] exec_options(::Base.JLOptions) at .\client.jl:243
   [28] _start() at .\client.jl:425

[maleadt]

Cannot reproduce. Please provide some details: versions of Julia and packages, OS, GPU model, CUDA toolchain, driver version, etc.

[jasontuna]

I am using
Windows 10
Julia 1.0.3
CUDA 10.0
Nvidia RTX 2070
Nvidia Driver Version 416.94
and I installed the CuArrays from the master branch on github.

I also installed
Julia 0.7 with the CuArrays in the repo, and everything works fine, but CuArrays.CUBLAS.gemm_strided_batched
is not present in that version.

with Julia 0.7 I can not get CuArrays to work with the master branch on github.

[jasontuna]

I installed Julia Version 1.1.0-rc1.0
and then had to install
CUDAnative from git by doing
]add CUDAnative#master

now all the tests go through in jupyter and on the command line,
by running
using Pkg
using CuArrays
Pkg.test("CuArrays")

if I run

m = 20
n = 35
k = 13
elty= Float32
nbatch = 10
alpha = one(elty)
beta = one(elty)
A = rand(elty, m, k, nbatch)
B = rand(elty, k, n, nbatch)
C = rand(elty, m, n, nbatch)
d_A = CuArray{elty, 3}(A)
d_B = CuArray{elty, 3}(B)
d_C = CuArray{elty, 3}(C)
CuArrays.CUBLAS.gemm_strided_batched!('N', 'N', alpha, d_A, d_B, beta, d_C)
CuArrays.CUBLAS.gemm_strided_batched!('N', 'N', alpha, d_A, d_B, beta, d_C)

In jupyter, it crashes with error,
CUDA error: an illegal memory access was encountered (code #85, ERROR_ILLEGAL_ADDRESS)

If i run the same code at the julia command line, it works without error.

Not sure if this is actually a bug with CuArrays.

[kshyatt]

Is this still an issue given the tests seem to be passing now?

[maleadt]

This may have been fixed by the memory management refactor, which aligned the behavior on the REPL and in e.g. Jupyter or Pluto. Please re-open if this is still an issue.