Always Cache QuantMetadata

[Giuseppe5]

Is your feature request related to a problem? Please describe.
During the internal call to our export functions for torch/onnx, we have multiple forward passes that are executed to perform caching of quant metadata.

Describe the solution you'd like

We have flags that use to enable/disable caching. The idea would be to always enable caching (in eval mode), and remove the need of extra forward passes.
Check if this has any meaningful impact on execution time.

[shivansh980]

Hello @Giuseppe5,

I would like to work on this issue.

I also went through the codebase and I have a few clarification questions.

Since, we are talking about eval mode, do we only need to care about the inference folder inside src/export?

We right now have cache_x as false by default in some functions. My understanding is we need to set it to true as default and analyze the impact it has on the forward passes and execution time if there is any. Is my understanding correct?

Thanks,
Shivansh

[Giuseppe5]

Hello @shivansh980,

Thanks for your interest.
Recently I came across a potential downside of this approach.

When using groupwise quantization, especially with LLM with 70B parameters or more, the size of scale factors becomes non negligible anymore, and thus caching them all the time could mean that a model would suddenly not fit anymore in a given GPU(s).

If you want to do some experiments with this and evaluate the potential impact on execution time as well as memory consumption, we would greatly appreciate that but until we have a better picture of the potential side effect, I cannot guarantee that this change will make it to dev.

To answer your question:

do we only need to care about the inference folder inside src/export?

No the two ideas are not really related to each other. As you pointed out, there are a lot of internal flags about stuff that can be cached, and the idea is to set some of them to true, in particular everything that has to do with quantization metadata (and not quant tensor values themselves).

We right now have cache_x as false by default in some functions. My understanding is we need to set it to true as default and analyze the impact it has on the forward passes and execution time if there is any. Is my understanding correct?

As I mentioned above, we want to set to True only the caching of what we call metadata, i.e., scale factors and zero points mostly.
Also we should analysie the impact on memory utilization as well for different types of quantization choices (i.e., per-tensor vs per-channel vs per-group).

Thanks for you interest,
Giuseppe