ONNXRuntime produces inconsistent results for specific output v10_0 (flaky test behavior) #23212
Description
Describe the issue
When running inference on an ONNX model with ONNXRuntime, I observe inconsistent results for a specific output (v10_0). The mismatch is observed in the v10_0 output path, which involves the following operations: input → round → Reshape → cos → sin → output. This behavior is inconsistent and appears to be flaky, as it does not always happen in every run.
- Actual Behavior
AssertionError:
Not equal to tolerance rtol=0.001, atol=0.001
Mismatched elements: 362 / 765 (47.3%)
Max absolute difference: 0.32707573
Max relative difference: 0.63584514
x: array([[0.841471, 0.841471, 0.841471, 0.514395, 0.841471, 0.841471,
0.514395, 0.514395, 0.841471, 0.514395, 0.514395, 0.514395,
0.841471, 0.514395, 0.514395, 0.514395, 0.841471],...
y: array([[0.841471, 0.841471, 0.841471, 0.514395, 0.841471, 0.841471,
0.514395, 0.514395, 0.841471, 0.514395, 0.514395, 0.514395,
0.841471, 0.514395, 0.514395, 0.514395, 0.841471],...
To reproduce
- Download the model
- Run the script:
import onnx
import onnxruntime as ort
import numpy as np
from onnxruntime.transformers import optimizer
model_path = "inconsis5.onnx"
optimized_model_path = f"./opt.onnx"
sess_options = ort.SessionOptions()
sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
this_provider_list = ort.get_available_providers()
original_session = ort.InferenceSession(model_path, sess_options, providers=this_provider_list)
input_data = {"v3_0": np.random.rand(1, 765).astype(np.float64)}
output_names = [output.name for output in original_session.get_outputs()]
original_result = original_session.run(output_names, input_data)
optimized_model = optimizer.optimize_model(model_path, opt_level=1, use_gpu=True)
optimized_model.save_model_to_file(optimized_model_path)
optimized_session = ort.InferenceSession(optimized_model_path, providers=this_provider_list)
optimized_model = onnx.load(optimized_model_path)
optimized_result = optimized_session.run(output_names, input_data)
for r1, r2 in zip(original_result, optimized_result):
np.testing.assert_allclose(r1, r2, atol=1e-3, rtol=1e-3)Urgency
No response
Platform
Linux
OS Version
Ubuntu 20.04
ONNX Runtime Installation
Built from Source
ONNX Runtime Version or Commit ID
ONNX Runtime API
Python
Architecture
X64
Execution Provider
CUDA
Execution Provider Library Version
No response