Consolidate `ZeroPointDomain.NONE` & `None` zero point domains #1556

sanchitintel · 2025-01-13T21:38:59Z

Summary

Both ZeroPointDomain.NONE & None zero point domains were being used. The latter was being used for float8. This PR consolidates both & retains ZeroPointDomain.NONE
Using ZeroPointDomain.NONE would now produce None for zero-point
int8_dynamic_activation_int8_weight now uses ZeroPointDomain.NONE as weight zero point domain (as weight is quantized symmetrically to int8).

Some of the older changes in this PR (such as supporting torch.compile with optional zero_point) were rendered redundant by more recent changes in the main branch, so I removed them & modified the description accordingly. Thanks!

pytorch-bot · 2025-01-13T21:39:02Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1556

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

torchao/quantization/quant_primitives.py

jerryzh168

LGTM

sanchitintel · 2025-01-14T04:27:27Z

Thanks for reviewing, @jerryzh168! I'll fix the lint error once other CI checks would complete.

sanchitintel · 2025-01-14T08:18:12Z

The UTs that failed are failing because of some Inductor codegen issues that have since been fixed, but the CI jobs used PyTorch v2.4. I'll skip some of those UTs, so that they'd run with PyTorch v2.5 & beyond instead.

Thanks!

jerryzh168 · 2025-01-15T05:23:22Z

for https://github.com/pytorch/ao/actions/runs/12760810579/job/35567540271?pr=1556 you can install pre-commit

pip install pre-commit

and run it:

pre-commit run

then it will run the formatting before every git commit

test/integration/test_integration.py

torchao/quantization/quant_primitives.py

jerryzh168 · 2025-01-15T20:22:28Z

test/quantization/test_observer.py

@@ -199,7 +200,6 @@ def test_linear_observer_tensor(self, observe_weight: bool):
            input_scale.item(),
            max_val / max_fp8,
        )
-        self.assertIsNotNone(input_zero_point)


is there a change of behavior when you change zero_point_domain for None to ZeroPointDomain.NONE?

Yes, input_zero_point would now be None. So, instead of removing that line, I now added self.assertIsNone(input_zero_point). Thanks!

I see, so what is the meaning of zero_point_domain == None before?

Some APIs were creating a None zero_point when zero_point_domain ZeroPointDomain.NONE or None was used, while choose_qparams_affine was not.

jerryzh168 · 2025-01-15T20:23:03Z

test/quantization/test_quant_primitives.py

@@ -838,6 +838,32 @@ def test_fake_quantize_affine_cachemask(self):
        torch.testing.assert_close(dequantized, fake_quantized)
        torch.testing.assert_close(expected_mask, mask)

+    # ZeroPointDomain.NONE should work
+    def test_none_zero_point_domain(self):


we could also have a test for zero_point_domain being None and throw an IllegalArgumentError exception I think

Thanks again for reviewing! I added code for raising ValueError if zero_point_domain would be None, but only for choose_qparams_affine and choose_qparams_affine_with_min_max.

For other public-facing APIs that have zero_point_domain as a parameter, I added asserts instead.

Please advise if this is fine, or if other places in the code should also throw exceptions instead of asserting.

Thanks!

torchao/quantization/quant_api.py

jerryzh168 · 2025-01-17T03:37:13Z

torchao/dtypes/affine_quantized_tensor.py

@@ -302,10 +303,8 @@ def from_hp_to_intx_static(
        zero_point_domain: Optional[ZeroPointDomain] = ZeroPointDomain.INT,
        _layout: Layout = PlainLayout(),
    ):
+        assert zero_point_domain is not None, "zero_point_domain must not be None"


zero_point_domain should not be Optional in L303 I think

also maybe raise ValueError here, that might be more user friendly than assertion

Thank you! Made those modifications

jerryzh168 · 2025-01-17T03:42:34Z

torchao/dtypes/affine_quantized_tensor.py

@@ -85,6 +85,7 @@ def __new__(
        dtype=None,
        strides=None,
    ):
+        assert zero_point_domain is not None, "zero_point_domain must not be None"


for the error message, I think we should include: "please use ZeroPointDomain.NONE" instead

and after some point it should be OK to remove these asserts when we are confident that no misuse in the codebase I think

…eight

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 13, 2025

sanchitintel mentioned this pull request Jan 13, 2025

[WIP] Use int_scaled_matmul with int8_dynamic_activation_int8_weight(act_mapping_type=MappingType.ASYMMETRIC) #1402

Closed

2 tasks

sanchitintel force-pushed the zero_point_domain_none branch from 473c42f to c53a9d5 Compare January 13, 2025 23:26

sanchitintel commented Jan 13, 2025

View reviewed changes

torchao/quantization/quant_primitives.py Outdated Show resolved Hide resolved

jerryzh168 approved these changes Jan 14, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

jerryzh168 added topic: bug fix Use this tag for PRs that fix bugs topic: improvement Use this tag if this PR is an improvement (doesn't fit into any of the other categories) and removed topic: bug fix Use this tag for PRs that fix bugs labels Jan 15, 2025

sanchitintel force-pushed the zero_point_domain_none branch from 837ee22 to c44df1f Compare January 15, 2025 06:09

sanchitintel marked this pull request as ready for review January 15, 2025 06:26

sanchitintel requested a review from jerryzh168 January 15, 2025 06:26

Fix ZeroPointDomain.NONE support & make it default for da8w8 weights

da2e9e0

sanchitintel force-pushed the zero_point_domain_none branch from c83f470 to da2e9e0 Compare January 15, 2025 06:48

sanchitintel changed the title ~~Fix ZeroPointDomain.NONE support & make it default for da8w8 weights~~ Consolidate ZeroPointDomain.NONE & None zero point domains Jan 15, 2025

drisspg reviewed Jan 15, 2025

View reviewed changes

test/integration/test_integration.py Show resolved Hide resolved

test/integration/test_integration.py Outdated Show resolved Hide resolved

torchao/quantization/quant_primitives.py Outdated Show resolved Hide resolved

Fix bug & apply review recommendations

bbc8dcd

drisspg approved these changes Jan 15, 2025

View reviewed changes

jerryzh168 reviewed Jan 15, 2025

View reviewed changes

sanchitintel commented Jan 15, 2025

View reviewed changes

torchao/quantization/quant_api.py Outdated Show resolved Hide resolved

jerryzh168 reviewed Jan 17, 2025

View reviewed changes

sanchitintel added 2 commits January 17, 2025 12:49

Throw exceptions when None zero_point_domain is used

4956a2e

Use ZeroPointDomain.NONE for weight in int8_dynamic_activation_int8_w…

8116c0c

…eight

sanchitintel force-pushed the zero_point_domain_none branch from cd7abef to 8116c0c Compare January 17, 2025 21:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consolidate `ZeroPointDomain.NONE` & `None` zero point domains #1556

Consolidate `ZeroPointDomain.NONE` & `None` zero point domains #1556

sanchitintel commented Jan 13, 2025 •

edited

Loading

pytorch-bot bot commented Jan 13, 2025 •

edited

Loading

jerryzh168 left a comment

sanchitintel commented Jan 14, 2025

This comment was marked as outdated.

sanchitintel commented Jan 14, 2025

jerryzh168 commented Jan 15, 2025

jerryzh168 Jan 15, 2025

sanchitintel Jan 15, 2025

jerryzh168 Jan 17, 2025

sanchitintel Jan 17, 2025

jerryzh168 Jan 15, 2025

sanchitintel Jan 15, 2025

jerryzh168 Jan 17, 2025

jerryzh168 Jan 17, 2025

sanchitintel Jan 17, 2025

jerryzh168 Jan 17, 2025 •

edited

Loading

Consolidate ZeroPointDomain.NONE & None zero point domains #1556

Are you sure you want to change the base?

Consolidate ZeroPointDomain.NONE & None zero point domains #1556

Conversation

sanchitintel commented Jan 13, 2025 • edited Loading

Summary

pytorch-bot bot commented Jan 13, 2025 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1556

jerryzh168 left a comment

Choose a reason for hiding this comment

sanchitintel commented Jan 14, 2025

This comment was marked as outdated.

sanchitintel commented Jan 14, 2025

jerryzh168 commented Jan 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerryzh168 Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Consolidate `ZeroPointDomain.NONE` & `None` zero point domains #1556

Consolidate `ZeroPointDomain.NONE` & `None` zero point domains #1556

sanchitintel commented Jan 13, 2025 •

edited

Loading

pytorch-bot bot commented Jan 13, 2025 •

edited

Loading

jerryzh168 Jan 17, 2025 •

edited

Loading