Skip to content

Fix float16 QuantizeLinear rounding on CPU - #32452

Open
kadyrbekovhamit-cyber wants to merge 1 commit into
microsoft:mainfrom
kadyrbekovhamit-cyber:codex/fix-float16-quantizelinear-rounding
Open

Fix float16 QuantizeLinear rounding on CPU#32452
kadyrbekovhamit-cyber wants to merge 1 commit into
microsoft:mainfrom
kadyrbekovhamit-cyber:codex/fix-float16-quantizelinear-rounding

Conversation

@kadyrbekovhamit-cyber

Copy link
Copy Markdown

Description

Fixes the still-reproducible numerical issue reported in #18576.

The CPU MLFloat16 overload of ParQuantizeLinearStd converted x / scale directly with static_cast<int32_t>, which truncates toward zero. ONNX QuantizeLinear requires round-to-nearest with ties to even.

This change applies std::nearbyint before the integer conversion, matching the ONNX contract and the nearby blocked float16 paths in the same file.

Minimal reproduction

For opset 19:

x (float16)     = [ 0.050018310546875, -0.050018310546875]
scale (float16) = 0.0999755859375
zero_point      = 0 (int8)
expected        = [ 1, -1]
ORT CPU 1.29.0  = [ 0,  0]

NumPy nearest-even and ONNX ReferenceEvaluator both return [1, -1]. A 4,096-value float16 differential check found 2,041 mismatches, all toward zero and by at most one quantization unit. The float32 control path matched the oracle.

Changes

  • round the scalar float16 CPU path with std::nearbyint;
  • add a non-integral float16-to-int8 regression test.

Testing

  • reproduced on macOS ARM64 with onnxruntime==1.29.0 and onnx==1.22.0;
  • verified against NumPy and ONNX ReferenceEvaluator;
  • full C++ unit suite not run locally; the new OpTester case is included for CI.

This is a numerical-correctness fix, not a security change.

Copilot AI balanced review requested due to automatic review settings September 5, 2026 17:08
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The test does not yet verify exact ties-to-even behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes CPU float16 QuantizeLinear to round fractional values before integer conversion.

Changes:

  • Applies nearest rounding in the scalar float16 quantization path.
  • Adds a regression test for positive and negative fractional values.
File summaries
File Description
onnxruntime/core/util/qmath.h Corrects float16 quantization rounding.
onnxruntime/test/providers/cpu/tensor/quantize_linear_test.cc Adds regression coverage.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread onnxruntime/test/providers/cpu/tensor/quantize_linear_test.cc Outdated
Signed-off-by: kadyrbekovhamit-cyber <288885044+kadyrbekovhamit-cyber@users.noreply.github.com>
@kadyrbekovhamit-cyber
kadyrbekovhamit-cyber force-pushed the codex/fix-float16-quantizelinear-rounding branch from 144e0b8 to 70dd31b Compare September 5, 2026 17:37
@kadyrbekovhamit-cyber

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused implementation matches existing quantization paths and is covered by an appropriate regression test.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants