diff --git a/README.md b/README.md index f016288..23a7668 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ Add `:emily` to your `mix.exs` deps: ```elixir def deps do [ - {:emily, "~> 0.7"} + {:emily, "~> 1.0"} ] end ``` @@ -65,9 +65,11 @@ Emily backend smoke test. other backends. - **Affine group-wise quantization.** `Emily.QuantizedWeight` + `Emily.Quantization` wrap MLX `quantize` / `dequantize` / - `quantized_matmul` for int2 / int4 / int8 inference. Includes a - defn-native `dequantize_defn/1` for quantized layers inside Axon - forward passes. + `quantized_matmul` for int2 / int4 / int8 inference. Quantized dense + layers lower to the fused `quantized_matmul` kernel — streaming the + packed low-bit weights instead of dequantizing the whole weight per + token — with a `dequantize_defn/1` + `Nx.dot` fallback on non-MLX + backends. - **Mixed-precision training.** `Emily.MixedPrecision` provides the bf16 recipe (cast params for the forward, keep f32 master, dynamic loss scaling with overflow detection). @@ -99,7 +101,7 @@ Releases on first `mix compile`. ### As a hex consumer -Add `{:emily, "~> 0.7"}` to `mix.exs`, then: +Add `{:emily, "~> 1.0"}` to `mix.exs`, then: ```sh mix deps.get diff --git a/lib/emily/bumblebee/fast_kernels.ex b/lib/emily/bumblebee/fast_kernels.ex index cd0ab1e..729f5f2 100644 --- a/lib/emily/bumblebee/fast_kernels.ex +++ b/lib/emily/bumblebee/fast_kernels.ex @@ -24,8 +24,8 @@ if Code.ensure_loaded?(Axon) and Code.ensure_loaded?(Bumblebee.Layers) do entirely when either dep is missing. To use the shim, add both to your own `deps/0`: - {:bumblebee, "~> 0.6"}, - {:axon, "~> 0.7"} + {:bumblebee, "~> 0.7"}, + {:axon, "~> 0.8"} ## Usage diff --git a/livebooks/distilbert_qa.livemd b/livebooks/distilbert_qa.livemd index 19cabf7..62d77af 100644 --- a/livebooks/distilbert_qa.livemd +++ b/livebooks/distilbert_qa.livemd @@ -3,7 +3,7 @@ ```elixir Mix.install( [ - {:emily, "~> 0.7"}, + {:emily, "~> 1.0"}, {:bumblebee, "~> 0.7"}, {:tokenizers, "~> 0.5"}, {:nx, "~> 0.12"}, diff --git a/livebooks/fast_kernels.livemd b/livebooks/fast_kernels.livemd index e121b7f..16c4bb0 100644 --- a/livebooks/fast_kernels.livemd +++ b/livebooks/fast_kernels.livemd @@ -3,7 +3,7 @@ ```elixir Mix.install( [ - {:emily, "~> 0.7"}, + {:emily, "~> 1.0"}, {:bumblebee, "~> 0.7"}, {:tokenizers, "~> 0.5"}, {:axon, "~> 0.8"}, diff --git a/livebooks/mnist_training.livemd b/livebooks/mnist_training.livemd index 780db83..31b9df0 100644 --- a/livebooks/mnist_training.livemd +++ b/livebooks/mnist_training.livemd @@ -3,7 +3,7 @@ ```elixir Mix.install( [ - {:emily, "~> 0.7"}, + {:emily, "~> 1.0"}, {:axon, "~> 0.8"}, {:scidata, "~> 0.1"}, {:nx, "~> 0.12"}, diff --git a/livebooks/modernbert_classification.livemd b/livebooks/modernbert_classification.livemd index 99fe637..53b7bdb 100644 --- a/livebooks/modernbert_classification.livemd +++ b/livebooks/modernbert_classification.livemd @@ -3,7 +3,7 @@ ```elixir Mix.install( [ - {:emily, "~> 0.7"}, + {:emily, "~> 1.0"}, {:bumblebee, "~> 0.7"}, {:tokenizers, "~> 0.5"}, {:nx, "~> 0.12"}, diff --git a/livebooks/nomic_embeddings.livemd b/livebooks/nomic_embeddings.livemd index 4ca194a..434709d 100644 --- a/livebooks/nomic_embeddings.livemd +++ b/livebooks/nomic_embeddings.livemd @@ -3,7 +3,7 @@ ```elixir Mix.install( [ - {:emily, "~> 0.7"}, + {:emily, "~> 1.0"}, {:bumblebee, "~> 0.7"}, {:tokenizers, "~> 0.5"}, {:nx, "~> 0.12"}, diff --git a/livebooks/qwen3_quantized.livemd b/livebooks/qwen3_quantized.livemd index 128501a..7fba122 100644 --- a/livebooks/qwen3_quantized.livemd +++ b/livebooks/qwen3_quantized.livemd @@ -3,7 +3,7 @@ ```elixir Mix.install( [ - {:emily, "~> 0.7"}, + {:emily, "~> 1.0"}, {:bumblebee, "~> 0.7"}, {:tokenizers, "~> 0.5"}, {:nx, "~> 0.12"}, @@ -57,7 +57,7 @@ dense_text `native: true, native_fallback: :raise` on `:defn_options` lowers the per-token decode through Emily's native Expr compiler and asserts it stays fully native — the quantized serving below passes the same -options, so the int4 dequantize path lowers native too. +options, so the int4 quantized matmul lowers native too. ## Quantization transform @@ -168,8 +168,8 @@ end > recursive rewriter. The `transpose: true` default stores weights as > `[out, in]` (MLX / AWQ convention, groups along the reduction axis); > set `false` if you're feeding a checkpoint that's already laid out -> the other way. Bits must be one of `[2, 3, 4, 6, 8]` for the -> defn-native `dequantize_defn/1` path. +> the other way. Bits must be one of `[2, 3, 4, 6, 8]` for MLX affine +> quantization. ## Quantized inference diff --git a/livebooks/smollm3_chat.livemd b/livebooks/smollm3_chat.livemd index ae595a1..a4e731b 100644 --- a/livebooks/smollm3_chat.livemd +++ b/livebooks/smollm3_chat.livemd @@ -3,7 +3,7 @@ ```elixir Mix.install( [ - {:emily, "~> 0.7"}, + {:emily, "~> 1.0"}, {:bumblebee, "~> 0.7"}, {:tokenizers, "~> 0.5"}, {:nx, "~> 0.12"}, diff --git a/livebooks/whisper_transcription.livemd b/livebooks/whisper_transcription.livemd index be2f14f..efb42d1 100644 --- a/livebooks/whisper_transcription.livemd +++ b/livebooks/whisper_transcription.livemd @@ -3,7 +3,7 @@ ```elixir Mix.install( [ - {:emily, "~> 0.7"}, + {:emily, "~> 1.0"}, {:bumblebee, "~> 0.7"}, {:tokenizers, "~> 0.5"}, {:nx, "~> 0.12"},