ONNXRuntime Optimization Causes Output Discrepancy in Specific Model Structure (Output Y)

[Thrsu]

Describe the issue

The optimization of an ONNX model using ONNXRuntime results in discrepancies between the original and optimized outputs, particularly for output Y. The issue occurs when running the optimized model and is not dependent on the optimization level (opt_level), but instead appears to be related to the specific structure of the model.

• Actual Behavior

AssertionError: 
Not equal to tolerance rtol=0.001, atol=0.001

Mismatched elements: 1 / 726 (0.138%)
Max absolute difference: 56.31724
Max relative difference: 2601.0862
 x: array([[[56.29559 ],
        [56.29559 ],
        [56.29559 ],...
 y: array([[[ 5.629559e+01],
        [ 5.629559e+01],
        [ 5.629559e+01],...

To reproduce

1. Download the ONNX model
2. Run the script:

import onnx
import onnxruntime as ort
import numpy as np
from onnxruntime.transformers import optimizer

model_path = "inconsis6.onnx"
optimized_model_path = f"./opt.onnx"
sess_options = ort.SessionOptions()
sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
this_provider_list = ort.get_available_providers()

original_session = ort.InferenceSession(model_path, sess_options, providers=this_provider_list)
input_data = {"v8_0": np.random.rand(33, 1, 1).astype(np.float32)}
output_names = [output.name for output in original_session.get_outputs()]
original_result = original_session.run(output_names, input_data)

optimized_model = optimizer.optimize_model(model_path, opt_level=1, use_gpu=True)
optimized_model.save_model_to_file(optimized_model_path)
optimized_session = ort.InferenceSession(optimized_model_path, providers=this_provider_list)
optimized_result = optimized_session.run(output_names, input_data)
for r1, r2 in zip(original_result, optimized_result):
    np.testing.assert_allclose(r1, r2, atol=1e-3, rtol=1e-3)

Urgency

No response

Platform

Linux

OS Version

Ubuntu 20.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

5c1b7cc

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

No response

[FFchopon]

I reproduced this script. After analyzing the output of each node, I observed the following behaviors:

1. In some cases, some of the three Slice nodes produces inconsistent outputs between the original and optimized models. However, these discrepancies do not propagate, as the sequence constructed by SequenceConstruct appears consistent in both models.
2. The input to the SequenceMap operation, derived from SequenceConstruct, remains consistent between the original and optimized models. Despite this, the final output from SequenceMap (i.e., the tensor Y) exhibits significant numerical inconsistencies.
   ...

There are many similar situations. I think this may be related to this model structure, but it's hard to figure out.
@fdwr

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.