Update example to apply compressed-tensors to a model

[dsikka]

Summary:

• Currently, this example exists which applies W4A16 to a Tiny Llama model: https://github.com/vllm-project/compressed-tensors/blob/main/examples/quantize_and_pack_int4.ipynb
• Update to use the latest tools in compressed-tensors. Specifically, the model should use compress_model when compressing the model, https://github.com/vllm-project/compressed-tensors/blob/main/examples/quantize_and_pack_int4.ipynb
• Remove the use of compress_quantized_weights which is out-dated

[zkl-ai]

Hi, I‘m interested in working on this issue. I’ve read the description and would like to give it a try. Could you assign it to me?

[dsikka]

Sure thing, thank you!

[zkl-ai]

Hi @dsikka

I'm ready to get started on this! I took a look and understand that the actual file to update (quantize_and_pack_int4.ipynb) is in the separate compressed-tensors repository. https://github.com/vllm-project/compressed-tensors/blob/main/examples/quantize_and_pack_int4.ipynb

Just to confirm my plan:

• Fork the vllm-project/compressed-tensors repo.
• Update the notebook, switching out the old compress_quantized_weights for the new compress_model tool.
• Submit a Pull Request to the main compressed-tensors repo and link it back here.

Does that sound like the right approach? Or are there any other specifics or tips I should know about before diving in? 😄

Thanks!

[dsikka]

Hi @zkl-ai yes this is the right idea!
In addition to what you've outlined, we want to simplify the steps and remove the use of compress_quantized_weights

[zkl-ai]

Hi @dsikka , I’ve opened a PR in vllm-project/compressed-tensors that updates the W4A16 TinyLlama example as requested here. Please approve the workflows from a fork so CI can run, and help review.
vllm-project/compressed-tensors#526

[dsikka]

Thank you for your contribution! The PR has landed!
Let us know if you'd like to contribute for other features / issues as well!