VectorQuantizeTask's linear int8 path discards three of its eight bits (divides by ‖v‖₂, not max|v|)

[sroussey]

The defect

VectorQuantizeTask.quantizeToInt8 (packages/ai/src/task/VectorQuantizeTask.ts:322, called only from :286) normalizes to unit L2 length and then scales by 127:

private quantizeToInt8(values: number[]): Int8Array {
  // Assume values are in [-1, 1] range after normalization
  // Scale to [-127, 127] to avoid overflow at -128
  return new Int8Array(values.map((v) => Math.round(Math.max(-1, Math.min(1, v)) * 127)));
}

The comment's assumption is the problem. After normalizeNumberArray(values, false) the coordinates are not spread over [-1, 1] — they sum in square to 1, so on a well-spread vector each one sits near 1/√d. Multiplying by 127 therefore emits a very small code.

Measured on the branch's own generator (a[i] = rnd() - 0.5, mulberry32, 40 vectors), largest absolute code the path emits:

d	largest int8 code emitted	codes used of 255
768	8	17
1536	6	13
3072	4	9

Three of the eight bits are gone before anything downstream sees the vector. The resulting cosine RMSE against the exact similarity is 0.00269 at d=768, against 0.00024 for the same quantizer dividing by max|v| instead — an ~11x accuracy loss for nothing.

Why it was not fixed in the TurboQuant PR

Deliberately deferred out of #354's follow-up (PR link below). Changing the divisor from ‖v‖₂ to max|v| rescales every stored coordinate by ~16x at d=768:

• cosine is safe — the change is a positive scalar multiple per vector, and cosine is scale-invariant.
• l2 and ip are not. VectorDistanceMetric includes both, distances under them scale with the vector's magnitude, and there is no version marker on a stored int8 vector. A corpus written partly before and partly after the change is incoherent under those two metrics and there is no way to detect which rows are which.

So this is a migration-relevant change, and landing it silently inside an "add TurboQuant" PR would bury it where no reviewer is looking for it.

Suggested shape of the fix

1. Add a linearScale: "l2" | "max-abs" task input rather than flipping the divisor silently, so a deployment chooses when to move.
2. Record the scaling on the output (alongside the existing method / turboSeed / originalDimensions fields) so a consumer can tell the two generations apart.
3. Write a migration note covering the l2 / ip incoherence, and state whether a mixed corpus must be re-quantized wholesale.
4. Consider the same question for quantizeToInt16 (:337), which has the identical shape and the identical comment.

Guard already in place

packages/test/src/test/rag/VectorQuantizeTask.test.ts carries a test named records the shipped linear int8 path's largest emitted code at d=768, asserting the largest code is 8, commented as a recorded defect, not a desired property. When this is fixed that expectation changes to 127 and the comment is deleted — so the repair is a deliberate, reviewed change rather than a surprise.

The accuracy tests in that file deliberately compute their max-abs reference locally rather than calling the task's linear path, so fixing this cannot move their bounds.

[sroussey]

Filed from the review of #354. The guard test and the corrected claims that depend on this staying deferred land in #801.

---

Generated by Claude Code