Tensor size limits are too strict for Gemm

[aobolensk]

Starting with v.52.6.0 (commit d4207ce, to be more precise) whisper encoder network started to fail in OpenVINO on ARM CPU.

Limits that we have faced with are in gemm implementation.

func: validate in src/cpu/operators/internal/CpuGemmAssemblyDispatch.cpp:1087
tensor_info->total_size(): 2304000000
total_size: 2304000000
tensor_shape(): [1500, 1500, 1, 512]
tensor_shape().total_size(): 1152000000
element_size: 2
num_channels: 1
max_size_in_bytes: 2147483647

So total size in our tensor (2304000000) exceeds max allowed one (2147483647 == 2^31 -1)

Could you please consider relaxing these limits or provide some alternatives for us to mitigate this limitation?

Thank you in advance!

[gunes-arm]

Hi @aobolensk thanks for reporting this, can you share a reproducible example here?

[aobolensk]

Sure

#include "arm_compute/core/Error.h"
#include "arm_compute/core/TensorShape.h"
#include "arm_compute/core/utils/misc/MMappedFile.h"
#include "arm_compute/runtime/NEON/NEFunctions.h"
#include "arm_compute/runtime/Tensor.h"
#include <iostream>
#include <vector>

using namespace arm_compute;

int main(int argc, char* argv[]) {
    GEMMInfo gemmInfo;
    TensorInfo srcTensorInfo = TensorInfo(TensorShape(1500, 1500, 1, 512), 1, DataType::F16, DataLayout::NCHW);
    TensorInfo weiTensorInfo = TensorInfo(TensorShape(1500, 1500, 1, 512), 1, DataType::F16, DataLayout::NCHW);
    TensorInfo dstTensorInfo = TensorInfo(TensorShape(1500, 1500, 1, 512), 1, DataType::F16, DataLayout::NCHW);

    auto status = NEGEMM::validate(&srcTensorInfo, &weiTensorInfo, nullptr, &dstTensorInfo, 1.0f, 0.0f, gemmInfo);
    if (status.error_code() != ErrorCode::OK) {
        std::cout << "ERROR: " << status.error_description().c_str() << std::endl;
        exit(1);
    }
    std::cout << "PASSED VALIDATION" << std::endl;

    Tensor srcTensor;
    Tensor weiTensor;
    Tensor dstTensor;
    srcTensor.allocator()->init(srcTensorInfo);
    weiTensor.allocator()->init(weiTensorInfo);
    dstTensor.allocator()->init(dstTensorInfo);

    NEGEMM gemm;
    gemm.configure(&srcTensor, &weiTensor, nullptr, &dstTensor, 1.0f, 0.0f, gemmInfo);
    std::cout << "PASSED CONFIGURATION" << std::endl;

    srcTensor.allocator()->allocate();
    weiTensor.allocator()->allocate();
    dstTensor.allocator()->allocate();

    gemm.run();
    std::cout << "PASSED RUN" << std::endl;

    srcTensor.allocator()->free();
    weiTensor.allocator()->free();
    dstTensor.allocator()->free();

    return 0;
}

Observed:

$ g++ -std=c++17 -O2 gemm.cpp -I./openvino/src/plugins/intel_cpu/thirdparty/ComputeLibrary -Lopenvino/build/src/plugins/intel_cpu/thirdparty/acl_build/build/arm64-v8a -larm_compute-static -I./openvino/src/plugins/intel_cpu/thirdparty/ComputeLibrary/include -o gemm
$ ./gemm 
ERROR: in validate src/cpu/operators/CpuGemm.cpp:226: Maximum supported tensor size is 2^31-1 bytes

reproduced on v.52.6.0 release tag, culprit commit: d4207ce