Skip to content

Conversation

@meiravgri
Copy link

@meiravgri meiravgri commented Jul 30, 2024

Issue description

This PR addresses an issue with out-of-range values during scalar quantization. Previously, when quantizing data, values outside the calibration range could result in incorrect quantization.

Solution

The PR fixes this issue by introducing a clip operation in the quantization process. This operation ensures that quantized values are correctly bounded within the expected range, thus preventing out-of-range values from producing incorrect results.

Example

calibration_set = np.array([[1,20,-3], [4,5,-60]], dtype=np.float32)
dataset = np.array([[-1,15,1]], dtype=np.float32). 
sq_embeddings = quantize_embeddings(
    dataset,
    precision="int8",
    calibration_embeddings=my_np_array,
)
print("quantizing dataset = ", sq_embeddings)

Result:
incorrect quantized values for the first and third entries, respectively, which are outside the calibration range.

quantizing dataset =  [[ -42   42 -112]]

After the PR:
The quantized values are correctly bounded and thus match the expected output.

quantizing dataset =  [[-128   42  127]]

@meiravgri
Copy link
Author

Hi @tomaarsen, could you please review my PR? Thanks!

@meiravgri meiravgri changed the title Handle out of range values in scalar quantization [fix] Handle out of range values in scalar quantization Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant