Update Readme #1526

drisspg · 2025-01-08T03:47:34Z

Stacked PRs:

->Update Readme #1526

Update Readme

pytorch-bot · 2025-01-08T03:47:38Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1526

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 024c52c with merge base 1c0ea5b ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

stack-info: PR: #1526, branch: drisspg/stack/24

README.md

jerryzh168 · 2025-01-15T01:09:53Z

README.md

-2. [2:4 Sparse Marlin GEMM](https://github.com/pytorch/ao/pull/733) 2x speedups for FP16xINT4 kernels even at batch sizes up to 256
-3. [int4 tinygemm unpacker](https://github.com/pytorch/ao/pull/415) which makes it easier to switch quantized backends for inference
+# Different CUDA versions
+pip install torchao --index-url https://download.pytorch.org/whl/cu121  # CUDA 12.1


we are supporting 12.4 by default now I think, we also removed 12.1 in CI at some point

supriyar · 2025-01-15T01:12:03Z

README.md


-## Alpha features
+## Composability


should this also mention DTensor composability?

yeah I think so, @drisspg you can check out the sglang blog post for some desciprtions

supriyar · 2025-01-15T01:14:58Z

README.md

@@ -2,7 +2,7 @@

 [![](https://dcbadge.vercel.app/api/server/gpumode?style=flat)](https://discord.gg/gpumode)

-[Introduction](#introduction) | [Inference](#inference) | [Training](#training)  | [Composability](#composability) | [Custom Kernels](#custom-kernels) | [Alpha Features](#alpha-features) | [Installation](#installation) | [Integrations](#integrations) | [Videos](#videos) | [License](#license) | [Citation](#citation)
+[Introduction](#introduction) | [Inference](#inference) | [Training](#training)  | [Installation](#installation) |[Composability](#composability) | [Custom Kernels](#custom-kernels) | [Prototype Features](#prototype-features)  | [Integrations](#integrations) | [Videos](#videos) | [License](#license) | [Citation](#citation)


should we highlight some benchmark numbers in the main readme like for int4 and float8 inference and composing sparsity + quantization, currently they are all one-click away?

added a blurb

stack-info: PR: #1526, branch: drisspg/stack/24

andrewor14 · 2025-01-15T16:14:07Z

README.md


 ## Training

 ### Quantization Aware Training

-Post-training quantization can result in a fast and compact model, but may also lead to accuracy degradation. We recommend exploring Quantization Aware Training (QAT) to overcome this limitation. In collaboration with Torchtune, we've developed a QAT recipe that demonstrates significant accuracy improvements over traditional PTQ, recovering **96% of the accuracy degradation on hellaswag and 68% of the perplexity degradation on wikitext** for Llama3 compared to post-training quantization (PTQ). And we've provided a full recipe [here](https://pytorch.org/blog/quantization-aware-training/). For more details, please see the [QAT README](./torchao/quantization/qat/README.md).
+Post-training quantization can result in a fast and compact model, but may also lead to accuracy degradation. We recommend exploring Quantization Aware Training (QAT) to overcome this limitation. In collaboration with [Torchtune](https://github.com/pytorch/torchtune/blob/main/recipes/quantization.md#quantization-aware-training-qat), we've developed a QAT recipe that demonstrates significant accuracy improvements over traditional PTQ, recovering **96% of the accuracy degradation on hellaswag and 68% of the perplexity degradation on wikitext** for Llama3 compared to post-training quantization (PTQ). And we've provided a full recipe [here](https://pytorch.org/blog/quantization-aware-training/)


should we keep the link to the QAT README?

Hey sorry this was likely rebase bug

vkuzo · 2025-01-15T17:04:51Z

README.md


-1. [MX](torchao/prototype/mx_formats) training and inference support with tensors using the [OCP MX spec](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf) data types, which can be described as groupwise scaled float8/float6/float4/int8, with the scales being constrained to powers of two. This work is prototype as the hardware support is not available yet.


it would be nice to keep mx and int8 quantized training callouts as we do plan to bring them out of prototype

Lets have a prototype Readme and once they are moved out of prototype lets update the readme?

I think a user should be able to text search the main README.md and know where to go for both MX training/inference and int8 training. Can we keep short references in the main readme please, with links to more info? Moving the more descriptive sections to a prototype readme sounds fine.

Lets have a prototype Readme and once they are moved out of prototype lets update the readme?

Can this be in this PR, instead of deleting?

if we just have someting like

Prototype folder contains upcoming features such as MX training and inference (link) and int8 quantized training (link), and more.

that would sgtm.

drisspg added a commit that referenced this pull request Jan 8, 2025

Update Readme

184768d

stack-info: PR: #1526, branch: drisspg/stack/24

drisspg force-pushed the drisspg/stack/24 branch from 7478471 to 184768d Compare January 8, 2025 03:47

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 8, 2025

drisspg added the topic: documentation Use this tag if this PR adds or improves documentation label Jan 8, 2025

drisspg added a commit that referenced this pull request Jan 15, 2025

Update Readme

e9ed4d3

stack-info: PR: #1526, branch: drisspg/stack/24

drisspg force-pushed the drisspg/stack/24 branch from 184768d to e9ed4d3 Compare January 15, 2025 00:04

drisspg added a commit that referenced this pull request Jan 15, 2025

Update Readme

1f43c3c

stack-info: PR: #1526, branch: drisspg/stack/24

drisspg force-pushed the drisspg/stack/24 branch from e9ed4d3 to 1f43c3c Compare January 15, 2025 00:10

drisspg added a commit that referenced this pull request Jan 15, 2025

Update Readme

9757375

stack-info: PR: #1526, branch: drisspg/stack/24

drisspg force-pushed the drisspg/stack/24 branch from 1f43c3c to 9757375 Compare January 15, 2025 00:53

drisspg added a commit that referenced this pull request Jan 15, 2025

Update Readme

83a79a4

stack-info: PR: #1526, branch: drisspg/stack/24

drisspg force-pushed the drisspg/stack/24 branch from 9757375 to 83a79a4 Compare January 15, 2025 01:03

drisspg added a commit that referenced this pull request Jan 15, 2025

Update Readme

b9f5c83

stack-info: PR: #1526, branch: drisspg/stack/24

drisspg force-pushed the drisspg/stack/24 branch from 83a79a4 to b9f5c83 Compare January 15, 2025 01:04

jerryzh168 reviewed Jan 15, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

jerryzh168 reviewed Jan 15, 2025

View reviewed changes

supriyar reviewed Jan 15, 2025

View reviewed changes

drisspg added a commit that referenced this pull request Jan 15, 2025

Update Readme

e443258

stack-info: PR: #1526, branch: drisspg/stack/24

drisspg force-pushed the drisspg/stack/24 branch from b9f5c83 to e443258 Compare January 15, 2025 01:15

drisspg added a commit that referenced this pull request Jan 15, 2025

Update Readme

5347249

stack-info: PR: #1526, branch: drisspg/stack/24

drisspg force-pushed the drisspg/stack/24 branch from e443258 to 5347249 Compare January 15, 2025 02:11

Update Readme

024c52c

stack-info: PR: #1526, branch: drisspg/stack/24

drisspg force-pushed the drisspg/stack/24 branch from 5347249 to 024c52c Compare January 15, 2025 02:36

andrewor14 reviewed Jan 15, 2025

View reviewed changes

vkuzo reviewed Jan 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Readme #1526

Update Readme #1526

drisspg commented Jan 8, 2025 •

edited

Loading

pytorch-bot bot commented Jan 8, 2025 •

edited

Loading

jerryzh168 Jan 15, 2025 •

edited

Loading

supriyar Jan 15, 2025

jerryzh168 Jan 15, 2025

supriyar Jan 15, 2025

drisspg Jan 15, 2025

andrewor14 Jan 15, 2025

drisspg Jan 15, 2025

vkuzo Jan 15, 2025

drisspg Jan 15, 2025

vkuzo Jan 15, 2025

vkuzo Jan 15, 2025


		1. [MX](torchao/prototype/mx_formats) training and inference support with tensors using the [OCP MX spec](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf) data types, which can be described as groupwise scaled float8/float6/float4/int8, with the scales being constrained to powers of two. This work is prototype as the hardware support is not available yet.

Update Readme #1526

Are you sure you want to change the base?

Update Readme #1526

Conversation

drisspg commented Jan 8, 2025 • edited Loading

pytorch-bot bot commented Jan 8, 2025 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1526

✅ No Failures

jerryzh168 Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drisspg commented Jan 8, 2025 •

edited

Loading

pytorch-bot bot commented Jan 8, 2025 •

edited

Loading

jerryzh168 Jan 15, 2025 •

edited

Loading