Incorrect Matrix Multiplication Results in OpenVINO for two dimension Randn Tensor ,the result is full of zero

[laoyao0822]

OpenVINO Version

openvino-2025.2.0.dev20250408

Operating System

Other (Please specify in description)

Device used for inference

CPU

Framework

ONNX

Model used

pytorch

Issue description

I found that switching my model to openvino produced completely different results compared to pytorch and onnx,
After a long time of troubleshooting, the problem was located to the following example

i think the bug comes from inference(stage of model compile)
Netron Validation: Confirmed IR model structure is equal to only generate (3,3) shape randn tensor
Observed Behavior: Output contains unexpected zeros only when y is (?,?) or other two dimension shape.
when you change it to (3,) or (3,3,3) it works well
cpu:INTEL(R) XEON(R) PLATINUM 8581C
system: rocky9.5

Step-by-step reproduction

import torch
import torch.nn as nn
import numpy as np
import random
import onnxruntime as ort
import openvino as ov
import openvino.properties as properties


class BugModel(nn.Module):
    def __init__(self):
        super(BugModel, self).__init__()
    def forward(self, x) -> torch.Tensor:
        #you could try to change the size  ,for example ,(3,) (3,3,3) 
        y=torch.randn(size=(2,2),dtype=x.dtype,device=x.device)
        return torch.matmul(x,y)

x=torch.randn(size=(3,4,2),dtype=torch.float32)
bug_model = BugModel().eval()
print(bug_model(x))
seq_len = torch.export.Dim('seq_len', min=2, max=1600)
onnx_path='/root/model.onnx'
torch.onnx.export(bug_model,
                  (x), f=onnx_path,
                  input_names=['x'],
                  output_names=['out'],
                  optimize=True,
                  opset_version=22,
                  dynamo=True,
                  dynamic_shapes={#  when close  it, result will full of zero 
                    'x': {0: seq_len},
                  }
                  )


ov_model=ov.convert_model(onnx_path,verbose=True)

ov.save_model(ov_model, '/root/model.xml')

config = {
    properties.inference_num_threads: 60,
}
compiled_model = ov.Core().compile_model(
            model=ov_model,
            device_name='CPU',
            config=config,
        )
infer_request = compiled_model.create_infer_request()
infer_request.set_input_tensor(0, ov.Tensor(torch.randn(size=(3,4,2),dtype=torch.float32).numpy()))
infer_request.infer()
output_tensor = infer_request.get_output_tensor(0)
out = torch.from_numpy(output_tensor.data)
print('openvino -------------')
print(out.shape)
print(out)

Relevant log output

when use dynamic shape
openvino -------------
torch.Size([3, 4, 3])
tensor([[[     0.,      0.,  -2232.],
         [     0.,      0.,  -3480.],
         [     0.,      0.,  -8352.],
         [     0.,      0.,   9216.]],

        [[     0.,      0., -43776.],
         [     0.,      0.,  17856.],
         [     0.,      0.,  -2472.],
         [     0.,      0.,  21024.]],

        [[     0.,      0.,  18432.],
         [     0.,      0.,  -4800.],
         [     0.,      0.,  55680.],
         [     0.,      0.,  -6384.]]])
when use fixed shape
tensor([[[0., 0., 0.],
         [0., 0., 0.],
         [0., 0., 0.],
         [0., 0., 0.]],

        [[0., 0., 0.],
         [0., 0., 0.],
         [0., 0., 0.],
         [0., 0., 0.]],

        [[0., 0., 0.],
         [0., 0., 0.],
         [0., 0., 0.],
         [0., 0., 0.]]])
when the randn shape is (3,),or (3,3,3)
openvino -------------
torch.Size([3, 4])
tensor([[-0.2812,  1.0469,  1.2109,  0.4141],
        [ 0.4043,  1.2344, -1.5469, -0.0835],
        [-1.2812,  0.9570,  1.1562,  0.0413]])

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[mvafin]

Hi, @laoyao0822,

You model have torch.randn operation which produce random numbers. We cannot guarantee same random numbers to be generated between pytorch and openvino. Usually this is not a problem for models, they produce correct results, but results may be different between pytorch and openvino.

[laoyao0822]

Hi, @laoyao0822,

You model have operation which produce random numbers. We cannot guarantee same random numbers to be generated between pytorch and openvino. Usually this is not a problem for models, they produce correct results, but results may be different between pytorch and openvino.torch.randn

But almost all of them are zero, which is an irrational result，just a matrix multiplication

[laoyao0822]

Hi, @laoyao0822,

You model have operation which produce random numbers. We cannot guarantee same random numbers to be generated between pytorch and openvino. Usually this is not a problem for models, they produce correct results, but results may be different between pytorch and openvino.torch.randn

Only two-dimensional random numbers can cause this situation, which is obviously irrational

[laoyao0822]

Hi, @laoyao0822,

You model have torch.randn operation which produce random numbers. We cannot guarantee same random numbers to be generated between pytorch and openvino. Usually this is not a problem for models, they produce correct results, but results may be different between pytorch and openvino.

You can reproduce it in only 20 lines of code

class BugModel(nn.Module):
    def __init__(self):
        super(BugModel, self).__init__()
    def forward(self, x) -> torch.Tensor:
        return torch.matmul(x,torch.randn(size=(3,3),dtype=x.dtype,device=x.device))
x=torch.randn(size=(3,3),dtype=torch.float32)
bug_model = BugModel().eval()
print("torch result\n",bug_model(x))
torch.onnx.export(bug_model,(x,), f=onnx_path,input_names=['x'],output_names=['out'])
ov_model=ov.convert_model(onnx_path,verbose=True)
compiled_model = ov.Core().compile_model(
            model=ov_model,
            device_name='CPU',
        )
infer_request = compiled_model.create_infer_request()
infer_request.set_input_tensor(0, ov.Tensor(torch.randn(size=(3,3),dtype=torch.float32).numpy()))
infer_request.infer()
out = infer_request.get_output_tensor(0).data
print("vino result\n",out)

and the result is

torch result
 tensor([[-4.2975,  0.5531, -0.8126],
        [-1.5862, -0.5080,  0.7227],
        [ 1.2694,  0.7758,  0.9973]])
Model Conversion arguments:
Common parameters:
        - Input Model:  /root/model.onnx
        - Input layers:         Not specified, inherited from the model
        - Output layers:        Not specified, inherited from the model
[ SUCCESS ] Total execution time: 0.06 seconds. 
[ SUCCESS ] Peak memory consumption (includes only memory allocated in Python): 0.36 MB. 
vino result
 [[0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]

when you change the Generated randn shape into (3,) or (3,3,3) it will get normal result

[mvafin]

Yes, I confirm, this looks like a bug. Need further investigation.

[mvafin]

Looks like a CPU plugin issue. GPU looks fine

[yuxu42]

Hi @laoyao0822 couldyou please try out this PR #30197 to see whether it can solve the problem in your side? Thanks!

[laoyao0822]

您好，您能否尝试一下这个 PR #30197，看看它是否可以解决您身边的问题？谢谢！

hi, thanks for your repair, but the model still can't get normal results. After a long time of investigation, I found the following phenomenon, when running for many times, the result obtained by openvino is significantly smaller than onnx and pytorch.As the number of runs increases, pytorch and onnx tends to be stable, but openvino will gradually approach 0. I don't know whether this is normal, the following code can be reproduced, pay attention to use the PR #30197 just submitted, otherwise the result will be full of 0

import torch
import torch.nn as nn
import onnxruntime as ort

execute_num=50

def random_rotation(device, dtype):

    v0, v1 = torch.randn(size=(2, 3), dtype=dtype, device=device)
    e0 = v0 / torch.clamp(torch.linalg.norm(v0), min=1e-10)
    v1 = v1 - e0 * torch.dot(v1, e0)
    e1 = v1 / torch.clamp(torch.linalg.norm(v1), min=1e-10)
    e2 = torch.cross(e0, e1, dim=-1)
    return torch.stack([e0, e1, e2])

def mean(mask, value, dim=None, keepdim=False, eps=1e-6):
    return torch.sum(mask * value, keepdim=keepdim, dim=dim) / (
        torch.clamp(torch.sum(mask, keepdim=keepdim, dim=dim) , min=eps)
    )
class BugModel(nn.Module):
    def __init__(self):
        super(BugModel, self).__init__()
    def forward(self, x,mask) -> torch.Tensor:
        center = mean(
            mask[..., None], x, dim=(-2, -3), keepdim=True, eps=1e-6
        )
        rot = random_rotation(device=x.device, dtype=x.dtype)
        out = torch.matmul(x - center, rot)
        return out * mask[..., None]

onnx_path='/root/pycharm/model.onnx'

bug_model = BugModel()
bug_model.eval()


x=torch.randn((20,25,3),dtype=torch.float32)*100
mask=torch.zeros(size=(20,25),dtype=torch.bool)
mask[..., 0:4]=True

result = bug_model(x,mask)
for i in range(execute_num):
    result = bug_model(result, mask)

print("tensor result-----------")
print(result[0])

inputs={
    'x': x.numpy(),
    'mask': mask.numpy()
}

ordered_keys = ["x","mask"]
output_names = ["out"]
kwarg_inputs = {'x': x, 'mask': mask}
ordered_inputs = tuple(kwarg_inputs[key] for key in ordered_keys)
input_len = torch.export.Dim('input_len', min=2, max=1600)
torch.onnx.export(bug_model,
                  ordered_inputs, f=onnx_path,
                  input_names=ordered_keys,
                  output_names=output_names,
                  # custom_translation_table=custom_translation_table,
                  optimize=True,
                  opset_version=22,
                  dynamo=True,
                  # export_params=True,
                  dynamic_shapes={
                    'x': {0: input_len},
                    'mask': {0: input_len},
                  }
                  )


sess_options = ort.SessionOptions()
session = ort.InferenceSession(onnx_path, sess_options=sess_options, provider_options=['CPUExecutionProvider'])
output=session.run(['out'], inputs)[0]
for i in range(execute_num):
    inputs = {
        'x': output,
        'mask': mask.numpy()
    }
    output = session.run(['out'], inputs)[0]
print('onnx-----------')
print(output[0])

import openvino as ov
import openvino.properties as properties
ov_model=ov.convert_model(onnx_path,verbose=True)
ov.save_model(ov_model, '/root/pycharm/model.xml')
config = {
    properties.inference_num_threads: 60,
}
compiled_model = ov.Core().compile_model(
            model=ov_model,
            device_name='CPU',
            config=config,
        )
infer_request = compiled_model.create_infer_request()

infer_request.set_input_tensor(0, ov.Tensor(x.numpy()))
infer_request.set_input_tensor(1, ov.Tensor(mask.numpy()))
infer_request.infer()

output_tensor = infer_request.get_output_tensor(0)

for i in range(execute_num):
    infer_request.set_input_tensor(0, output_tensor)
    infer_request.infer()
    output_tensor=infer_request.get_output_tensor(0)

vino_out = torch.from_numpy(output_tensor.data)

print('openvino -------------')
print(vino_out.shape)
print(vino_out[0])

[liubo-intel]

您好，您能否尝试一下这个 PR #30197，看看它是否可以解决您身边的问题？谢谢！

hi, thanks for your repair, but the model still can't get normal results. After a long time of investigation, I found the following phenomenon, when running for many times, the result obtained by openvino is significantly smaller than onnx and pytorch.As the number of runs increases, pytorch and onnx tends to be stable, but openvino will gradually approach 0. I don't know whether this is normal, the following code can be reproduced, pay attention to use the PR #30197 just submitted, otherwise the result will be full of 0

import torch
import torch.nn as nn
import onnxruntime as ort

execute_num=50

def random_rotation(device, dtype):

v0, v1 = torch.randn(size=(2, 3), dtype=dtype, device=device)
e0 = v0 / torch.clamp(torch.linalg.norm(v0), min=1e-10)
v1 = v1 - e0 * torch.dot(v1, e0)
e1 = v1 / torch.clamp(torch.linalg.norm(v1), min=1e-10)
e2 = torch.cross(e0, e1, dim=-1)
return torch.stack([e0, e1, e2])

def mean(mask, value, dim=None, keepdim=False, eps=1e-6):
return torch.sum(mask * value, keepdim=keepdim, dim=dim) / (
torch.clamp(torch.sum(mask, keepdim=keepdim, dim=dim) , min=eps)
)
class BugModel(nn.Module):
def init(self):
super(BugModel, self).init()
def forward(self, x,mask) -> torch.Tensor:
center = mean(
mask[..., None], x, dim=(-2, -3), keepdim=True, eps=1e-6
)
rot = random_rotation(device=x.device, dtype=x.dtype)
out = torch.matmul(x - center, rot)
return out * mask[..., None]

onnx_path='/root/pycharm/model.onnx'

bug_model = BugModel()
bug_model.eval()

x=torch.randn((20,25,3),dtype=torch.float32)*100
mask=torch.zeros(size=(20,25),dtype=torch.bool)
mask[..., 0:4]=True

result = bug_model(x,mask)
for i in range(execute_num):
result = bug_model(result, mask)

print("tensor result-----------")
print(result[0])

inputs={
'x': x.numpy(),
'mask': mask.numpy()
}

ordered_keys = ["x","mask"]
output_names = ["out"]
kwarg_inputs = {'x': x, 'mask': mask}
ordered_inputs = tuple(kwarg_inputs[key] for key in ordered_keys)
input_len = torch.export.Dim('input_len', min=2, max=1600)
torch.onnx.export(bug_model,
ordered_inputs, f=onnx_path,
input_names=ordered_keys,
output_names=output_names,
# custom_translation_table=custom_translation_table,
optimize=True,
opset_version=22,
dynamo=True,
# export_params=True,
dynamic_shapes={
'x': {0: input_len},
'mask': {0: input_len},
}
)

sess_options = ort.SessionOptions()
session = ort.InferenceSession(onnx_path, sess_options=sess_options, provider_options=['CPUExecutionProvider'])
output=session.run(['out'], inputs)[0]
for i in range(execute_num):
inputs = {
'x': output,
'mask': mask.numpy()
}
output = session.run(['out'], inputs)[0]
print('onnx-----------')
print(output[0])

import openvino as ov
import openvino.properties as properties
ov_model=ov.convert_model(onnx_path,verbose=True)
ov.save_model(ov_model, '/root/pycharm/model.xml')
config = {
properties.inference_num_threads: 60,
}
compiled_model = ov.Core().compile_model(
model=ov_model,
device_name='CPU',
config=config,
)
infer_request = compiled_model.create_infer_request()

infer_request.set_input_tensor(0, ov.Tensor(x.numpy()))
infer_request.set_input_tensor(1, ov.Tensor(mask.numpy()))
infer_request.infer()

output_tensor = infer_request.get_output_tensor(0)

for i in range(execute_num):
infer_request.set_input_tensor(0, output_tensor)
infer_request.infer()
output_tensor=infer_request.get_output_tensor(0)

vino_out = torch.from_numpy(output_tensor.data)

print('openvino -------------')
print(vino_out.shape)
print(vino_out[0])

Hi, @laoyao0822 : from my understanding, this kind of difference between openvino and pytorch/onnx comes from the torch.randn operation(which is normal) just as mvafin previously mentioned (Hi, @mvafin please correct me if my understanding is incorrect)

Hi, @laoyao0822,

You model have torch.randn operation which produce random numbers. We cannot guarantee same random numbers to be generated between pytorch and openvino. Usually this is not a problem for models, they produce correct results, but results may be different between pytorch and openvino.