optimum export openvino ,use gptq&scale_estimation error

[qihui-liu]

Exception has occurred: ValueError
could not broadcast input array from shape (84934656,) into shape (9216,)
File "C:\Users\admin\Desktop\convert_model.py", line 15, in
model = OVModelForCausalLM.from_pretrained(model_id, export=True,quantization_config=quantization_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: could not broadcast input array from shape (84934656,) into shape (9216,)

source code:

from optimum.intel import OVModelForCausalLM,OVWeightQuantizationConfig
from transformers import AutoTokenizer

model_id = "./Phi-3.5-mini-instruct"
quantization_config = OVWeightQuantizationConfig(
bits=4,
sym=True,
quant_method="awq",
scale_estimation=True,
group_size=-1,
gptq=True,
dataset="wikitext2"
)

model = OVModelForCausalLM.from_pretrained(model_id, export=True,quantization_config=quantization_config)
tokenizer = AutoTokenizer.from_pretrained(model_id)
save_directory = "./Phi3.5-ov-awq-gptq"
model.save_pretrained(save_directory)
tokenizer.save_pretrained(save_directory)

What shall I do?

[qihui-liu]

optimum-cli export openvino --model .\Phi-3.5-mini-instruct\ --task text-generation-with-past --weight-format int4 --group-size -1 --sym --awq --dataset wikitext2 --gptq --scale-estimation phi-3.5-mini-int4-awq-gptq-scale

It has the same effect.