When using the int8 quantization model to convert to onnx, an error occurs during runtime

[jungyin]

I am trying to convert my model to onnxruntime, and my model itself has been int8 quantized. During the runtime, the following error occurred
NOT_IMPLEMENTED : Could not find an implementation for Add(14) node with name '/self_attn/q_proj/Add'

During the debugging process, I found that after quantization, the linear layer became the QuantLinear layer, where qweight was int32 and bias was float16. Perhaps the error was caused by the different types of the two layers

I passed the model loading check when loading the model, as shown in the following code，it not have error report

onnx.checker.check_model(self.model_path)

Does the onnx model not support converting pytorch's quantization model? If supported, what do I need to do?

[skottmckay]

If you view the model in Netron and search for that Add node what data types do the inputs/outputs of it have?

What version of ONNX Runtime are you using?

[jungyin]

If you view the model in Netron and search for that Add node what data types do the inputs/outputs of it have?

What version of ONNX Runtime are you using?

this was my onnx version:

C:\Users\30585>pip list | findstr onnx
keras2onnx 1.7.0
onnx 1.17.0
onnx2tf 1.22.4
onnxconverter-common 1.14.0
onnxruntime 1.20.1
onnxruntime-gpu 1.20.1
onnxsim 0.4.36
tf2onnx 1.16.1
tflite2onnx 0.4.1

and it input was float16,ouput was float16.but it have smoe int32 value

[skottmckay]

Not following where the int32 value comes into it. Are both inputs to the Add float16?

[jungyin]

Not following where the int32 value comes into it. Are both inputs to the Add float16?

Okay, maybe we can set aside this issue for now, because I found a similar problem. I used PyTorch to convert BFLOAT16 to float32, then performed PoW calculation, and then converted it to the calculation of multiplying BFLOAT16 with another BFLOAT16. Similar errors occurred during this process. My analysis suggests that if there are two different data types in my model, such as converting one value type during the calculation process and then continuing the calculation, an error will occur. Do you have any good suggestions for solving this error

and this is my code

class Qwen2RMSNorm(nn.Module):
    def __init__(self, hidden_size, eps=1e-6):
        """
        Qwen2RMSNorm is equivalent to T5LayerNorm
        """
        super().__init__()
        self.weight = nn.Parameter(torch.ones(hidden_size))
        self.variance_epsilon = eps
        self .test1 = torch.full([5,5],1,dtype=torch.bfloat16)
        self .test2 = torch.full([5,5],1,dtype=torch.float32)
    def forward(self, hidden_states,kk = 0):
        # if(kk == 1):
            # return self.test1*self.test2
        input_dtype = hidden_states.dtype
        # if not (input_dtype == torch.float32 or input_dtype == torch.float16 or input_dtype == torch.bfloat16 or input_dtype == torch.float64):
        hidden_states = hidden_states.to(torch.float32)
        variance = hidden_states.pow(2).mean(-1, keepdim=True)
        rsqrt = torch.rsqrt(variance + self.variance_epsilon)
        hidden_states = hidden_states * rsqrt
        return self.weight * hidden_states.to(input_dtype)

In this code, if I try to modify the last line of
return self. height * hidden_dates. to (input_dtype)
to
return (self. height * hidden_dates). to (input_dtype)
, the error will disappear. However, if I try to modify it to
return self. height. to (input_dtype) * hidden_dates. to (input_dtype)
, the error will recur. Do you have any good suggestions

[jungyin]

In this process, since my model benchmark is BF16, but there is a pow calculation in between, I considered exporting my model directly without class conversion, but that would result in an error
'tensor(bfloat16)' of input parameter (/input_layernorm/Constant_output_0) of operator (Pow) in node (/input_layernorm/Pow) is invalid.

[jungyin]

Not following where the int32 value comes into it. Are both inputs to the Add float16?

Now, I am trying to replace the original BF16 as the input value with float16 or float32. This will not result in an error, but it seems to have some impact on the original model? Do you have any good suggestions for this conversion loss

[jungyin]

Okay, after reviewing the model source code, I ultimately believe that there is a type reset in my model, and onnx does not save this type reset operation when exporting the model, which caused the model exception. My solution was to find the original model of BF16 and attempt to export it as an FP32 model (BF16 does not support some operations such as pow in Onnxruntime) and export it again. Finally, I completed my model export work