Question about load_calibration_config function of the QAT quantizer

[Lycan1003]

    def load_calibration_config(self, calibration_config):
        modules_to_compress = self.get_modules_to_compress()
        for layer, _ in modules_to_compress:
            name, module = layer.name, layer.module
            if name not in calibration_config:
                if module.layer_quant_setting.weight or module.layer_quant_setting.input or module.layer_quant_setting.output:
                    logger.warning(f"Can not find module {name}'s parameter in input config.")
                continue
            if module.layer_quant_setting.weight:
                assert calibration_config[name]['weight_bits'] == module.layer_quant_setting.weight.bits, \
                    f"weight bits of module {name} fail to match"
            if module.layer_quant_setting.input:
                assert calibration_config[name]['input_bits'] == module.layer_quant_setting.input.bits, \
                    f"input bits of module {name} fail to match"
                module.tracked_min_input.data = torch.tensor([calibration_config[name]['tracked_min_input']])
                module.tracked_max_input.data = torch.tensor([calibration_config[name]['tracked_max_input']])
            if module.layer_quant_setting.output:
                assert calibration_config[name]['output_bits'] == module.layer_quant_setting.output.bits, \
                    f"output bits of module {name} fail to match"
                module.tracked_min_output.data = torch.tensor([calibration_config[name]['tracked_min_output']])
                module.tracked_max_output.data = torch.tensor([calibration_config[name]['tracked_max_output']])

The above code is from line 418 to 438 of qat_quantizer.py. As we can see, it can load tracked_min and tracked_max of input and output, but for the weight, it only checks the weight_bit match or not to the config list, and then do nothing.

In my understanding, maybe it is reasonable to load weight_scale and weight_zero_point from calibration_config show like following code?

            if module.layer_quant_setting.weight:
                assert calibration_config[name]['weight_bits'] == module.layer_quant_setting.weight.bits, \
                    f"weight bits of module {name} fail to match"
                module.weight_scale = calibration_config[name]['weight_scale'].data
                module.weight_zero_point = calibration_config[name]['weight_zero_point'].data

[liuzhe-lz]

Assigned to @linbinskn

[2730gf]

I have the same question with you

[scarlett2018]

@J-shang - is @Lycan1003 's proposal a reasonable improvement to the code? if so, we do encourage him/her to submit the PR.

[J-shang]

@scarlett2018, from @linbinskn, because the weight range can be obtained through the model, there is no need to calibrate separately, but the range of input and output changes dynamically with the input data, so calibration is required.

[Lycan1003]

@scarlett2018 @J-shang I am sorry that I don't remember every detail of this problem because of three months past.
When I load a pre-trained quantized NNI model to do testing, an error like "weight_scale and weight_zero_point do not exist" happens.
After changing the code shown before, it works normally.

I hope it will be helpful.

[J-shang]

@scarlett2018 @J-shang I am sorry that I don't remember every detail of this problem because of three months past. When I load a pre-trained quantized NNI model to do testing, an error like "weight_scale and weight_zero_point do not exist" happens. After changing the code shown before, it works normally.

I hope it will be helpful.

Thanks for your help, we will review this problem and give a better user experience.