Skip to content

Commit f7432b1

Browse files
committed
docs: describe the fused quantized_matmul path, not dequantize
The README quantization bullet and the qwen3_quantized livebook still described the quantized path as running through `dequantize_defn/1` (dequantize-then-dot). Since the fused quantized_matmul work (61b36f4) the native quantized path lowers to the fused `mx::quantized_matmul` kernel, streaming the packed low-bit weights; dequantize_defn is now only the non-MLX fallback. Update the prose to match.
1 parent 47266a4 commit f7432b1

2 files changed

Lines changed: 8 additions & 6 deletions

File tree

README.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,9 +65,11 @@ Emily backend smoke test.
6565
other backends.
6666
- **Affine group-wise quantization.** `Emily.QuantizedWeight` +
6767
`Emily.Quantization` wrap MLX `quantize` / `dequantize` /
68-
`quantized_matmul` for int2 / int4 / int8 inference. Includes a
69-
defn-native `dequantize_defn/1` for quantized layers inside Axon
70-
forward passes.
68+
`quantized_matmul` for int2 / int4 / int8 inference. Quantized dense
69+
layers lower to the fused `quantized_matmul` kernel — streaming the
70+
packed low-bit weights instead of dequantizing the whole weight per
71+
token — with a `dequantize_defn/1` + `Nx.dot` fallback on non-MLX
72+
backends.
7173
- **Mixed-precision training.** `Emily.MixedPrecision` provides the
7274
bf16 recipe (cast params for the forward, keep f32 master, dynamic
7375
loss scaling with overflow detection).

livebooks/qwen3_quantized.livemd

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ dense_text
5757
`native: true, native_fallback: :raise` on `:defn_options` lowers the
5858
per-token decode through Emily's native Expr compiler and asserts it
5959
stays fully native — the quantized serving below passes the same
60-
options, so the int4 dequantize path lowers native too.
60+
options, so the int4 quantized matmul lowers native too.
6161

6262
## Quantization transform
6363

@@ -168,8 +168,8 @@ end
168168
> recursive rewriter. The `transpose: true` default stores weights as
169169
> `[out, in]` (MLX / AWQ convention, groups along the reduction axis);
170170
> set `false` if you're feeding a checkpoint that's already laid out
171-
> the other way. Bits must be one of `[2, 3, 4, 6, 8]` for the
172-
> defn-native `dequantize_defn/1` path.
171+
> the other way. Bits must be one of `[2, 3, 4, 6, 8]` for MLX affine
172+
> quantization.
173173
174174
## Quantized inference
175175

0 commit comments

Comments
 (0)