M23: public documentation & examples review

Pre-1.0 pass over the documentation surface users consume on HexDocs.

- Moduledoc pass: added Public API sections to Emily, Emily.Backend,
  and Emily.Compiler to distinguish user-facing helpers from
  Nx.Backend / Nx.Defn.Compiler callback implementations. Added
  iex> examples to Emily.{to_binary, shape, dtype}, Emily.Compiler,
  Emily.Stream, Emily.Quantization, Emily.QuantizedWeight,
  Emily.MixedPrecision, and Emily.Telemetry. Reorganised
  Emily.Backend's divergences into a dedicated "Divergences from
  Nx.BinaryBackend" subsection.
- Emily.to_binary/1 now documents the zero-copy semantics: the
  pinned buffer refcount disqualifies MLX's donation path, and the
  ProcBin accounting footgun is flagged with mitigations.
- HexDocs nav via mix.exs: extras now grouped into README and
  Notebooks; modules grouped into Core / Concurrency /
  Quantization / Training / Performance / Observability.
- README rewrite: Overview + Features bullets, Prerequisites with
  the xcodebuild -downloadComponent MetalToolchain step, Building
  with a "Why vendored?" subsection covering ml-explore/mlx#3348
  and #3405, and an expanded Concurrency model section covering
  the worker-thread dispatch architecture.
- Two runnable Livebooks under notebooks/: distilbert_qa.livemd
  (Bumblebee QA through Emily.Compiler) and qwen3_quantized.livemd
  (Qwen3-0.6B int4-quantized + Emily.Stream concurrent serving).
- CHANGELOG cutover: moved the M0-M22 release history from
  RELEASE.md into CHANGELOG.md, then drastically simplified it to
  a top-level feature summary under [0.1.0] - unreleased. Users
  who want per-milestone detail can consult the git history.
  RELEASE.md now carries only M23 notes.
- mix docs runs clean (zero warnings).

Author: ausimian

.gitignore

@@ -25,6 +25,7 @@ emily-*.tar
 # Native build artefacts
 /priv/libemily.*
 /priv/mlx/
+/priv/mlx.metallib
 
 # Dialyzer PLTs
 /priv/plts/

CHANGELOG.md

@@ -7,10 +7,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 <!-- %% CHANGELOG_ENTRIES %% -->
 
-## [Unreleased]
+## [0.1.0] - unreleased
+
+Initial release. See the git history for per-milestone detail.
 
 ### Added
 
-- Initial project scaffold (M0): mix project, MLX prebuilt fetcher,
-  Makefile, minimal NIF with `Tensor` resource and
-  `from_binary`/`to_binary`/`shape`/`dtype`/`eval` surface.
+- **Nx backend.** `Emily.Backend` implements every required
+  `Nx.Backend` callback against MLX, with transparent fallback to
+  `Nx.BinaryBackend` for ops without a native primitive.
+- **Defn compiler.** `Emily.Compiler` runs `defn` / `Nx.Serving` /
+  Bumblebee on Emily; pins the result backend and caps partition
+  concurrency so `Nx.Serving` stays compatible.
+- **Fused transformer kernels.** `Emily.Fast` exposes
+  `mx::fast::rms_norm`, `layer_norm`, `rope`, and scaled-dot-product
+  attention as defn-callable helpers with composed-defn fallbacks
+  for non-Emily backends.
+- **Affine group-wise quantization.** `Emily.QuantizedWeight` and
+  `Emily.Quantization` wrap MLX `quantize` / `dequantize` /
+  `quantized_matmul` for int2 / int4 / int8 inference.
+  `Emily.Quantization.dequantize_defn/1` provides a defn-native
+  dequantize for use inside Axon forward passes.
+- **Mixed-precision training.** `Emily.MixedPrecision` ships the
+  bf16 recipe: `cast_params` for the forward pass, f32 master
+  weights, dynamic loss scaling with overflow detection.
+- **Per-process Metal streams.** `Emily.Stream` lets each BEAM
+  process own its own Metal command queue, enabling concurrent
+  inference on a shared model.
+- **Zero-copy `to_binary`.** `Nx.to_binary/1` on an Emily tensor
+  returns a BEAM resource binary aliasing the MLX buffer — no memcpy.
+- **Native gradient + training primitives.** `gather`, `scatter`,
+  `scatter_add`, `conv`, and the window-reduction family lower
+  directly to MLX so `Nx.Defn.grad` and CNN training stay native.
+- **Native linalg.** `lu`, `svd`, `qr`, `cholesky`, `eigh`, `solve`,
+  and `triangular_solve` dispatch to `mx::linalg::*` instead of
+  rounding through `Nx.BinaryBackend`.
+- **Telemetry.** `[:emily, :eval, *]`, `[:emily, :to_binary, *]`,
+  `[:emily, :fallback, *]`, and `[:emily, :memory, :stats]` span
+  events; opt-in one-shot fallback warnings via
+  `config :emily, :warn_on_fallback, true`.
+- **Compile-time debug flags.** `:debug_bounds_check` and
+  `:debug_detect_nan_inf` re-enable runtime assertions on hot paths;
+  default off with zero runtime cost.
+- **Bumblebee conformance.** End-to-end suites for DistilBERT,
+  Qwen3-0.6B (dense and quantized), ViT-base, and Whisper-tiny,
+  pinned against HuggingFace reference values.
+- **Worker-thread dispatch.** Each MLX stream is owned by a
+  dedicated OS thread; NIFs hand work to the worker via a
+  promise/future, keeping per-thread Metal `CommandEncoder` state
+  consistent across BEAM scheduler migration.
+- **Vendored MLX build.** MLX is built from source via cmake from
+  `vendor/mlx` (git submodule); no prebuilt download. Build cache
+  keyed on the submodule SHA under `~/Library/Caches/emily/`.
+- **Documentation.** Per-module HexDocs, two runnable Livebooks
+  (`notebooks/distilbert_qa.livemd`,
+  `notebooks/qwen3_quantized.livemd`), and worked Bumblebee
+  examples in the conformance suite.

README.md

@@ -1,38 +1,140 @@
-# emily
+# Emily
 
-Elixir bindings and Nx backend for Apple's [MLX](https://github.com/ml-explore/mlx).
+Elixir bindings and Nx backend for Apple's
+[MLX](https://github.com/ml-explore/mlx).
 
-**Status: M14 — Serving concurrency.** Stream-per-process
-concurrent inference via `Emily.Stream`. See [`PLAN.md`](PLAN.md) for
-the full roadmap and [`RELEASE.md`](RELEASE.md) for unreleased-version
-notes.
+## Overview
 
-## Why
+Emily runs `Nx` computations on Apple Silicon through MLX. Installing
+it as the default Nx backend is enough to get Bumblebee models
+executing on the Metal GPU with no further integration work —
+DistilBERT, Qwen3, ViT, and Whisper all run against pinned reference
+outputs in the conformance suite today.
 
-To run Bumblebee models (notably Qwen3) on Apple Silicon with Metal
-acceleration, via a layered architecture that keeps each layer
-independently testable.
-
-## Architecture
+The library is structured as four thin layers, each independently
+testable against its own oracle:
 
 ```
 Emily.Compiler    (Nx.Defn.Compiler) — validates opts, pins the result backend
 Emily.Backend     (Nx.Backend)       — op-by-op translation to Native
-Emily.Native      (thin NIF shim)    — one function per MLX op, no policy
-MLX C++           (vendored binary)  — cocoa-xu/mlx-build prebuilts, pinned
+Emily.Native      (NIF shim)         — one function per MLX op, no policy
+MLX C++           (vendored source)  — built from `vendor/mlx` via cmake
 ```
 
-One-directional dispatch: Elixir → C++ → MLX. C++ never calls back
-into BEAM.
+Dispatch is one-directional (Elixir → C++ → MLX); the C++ side never
+calls back into the BEAM.
+
+## Features
+
+- **Nx backend.** Every `Nx.*` op dispatches to MLX; ops without a
+  native primitive fall back transparently to `Nx.BinaryBackend`
+  with a `[:emily, :fallback, *]` telemetry event. See
+  `Emily.Backend`.
+- **Defn compiler.** `Emily.Compiler` runs `defn` / `Nx.Serving` /
+  Bumblebee inference on MLX. Backs the results with lazy MLX graphs.
+- **Fused transformer kernels.** `Emily.Fast` exposes
+  `mx::fast::rms_norm`, `layer_norm`, `rope`, and scaled-dot-product
+  attention as defn-callable helpers with composed-defn fallbacks for
+  other backends.
+- **Affine group-wise quantization.** `Emily.QuantizedWeight` +
+  `Emily.Quantization` wrap MLX `quantize` / `dequantize` /
+  `quantized_matmul` for int2 / int4 / int8 inference. Includes a
+  defn-native `dequantize_defn/1` for quantized layers inside Axon
+  forward passes.
+- **Mixed-precision training.** `Emily.MixedPrecision` provides the
+  bf16 recipe (cast params for the forward, keep f32 master, dynamic
+  loss scaling with overflow detection).
+- **Per-process Metal streams.** `Emily.Stream` lets each BEAM
+  process own its own Metal command queue, so multiple processes can
+  share a model and run inference concurrently.
+- **Zero-copy `to_binary`.** `Nx.to_binary/1` on an Emily tensor
+  returns a BEAM resource binary aliasing the MLX buffer — no memcpy.
+- **Telemetry.** `[:emily, :eval, *]`, `[:emily, :to_binary, *]`,
+  `[:emily, :fallback, *]`, and `[:emily, :memory, :stats]` span
+  events. See `Emily.Telemetry`.
+- **Compile-time debug flags.** `:debug_bounds_check` and
+  `:debug_detect_nan_inf` re-enable runtime assertions on hot paths
+  that GPU backends skip by default. Both default off with zero
+  runtime cost.
+
+## Prerequisites
+
+- **macOS.** Apple Silicon recommended; x86_64 is supported but
+  without GPU acceleration.
+- **Elixir 1.18+ / OTP 27+.** Development is pinned to Elixir 1.19.5
+  / OTP 28.3 via `.tool-versions`.
+- **Xcode with the Metal toolchain.** The Command Line Tools alone
+  are not enough — the build invokes `xcrun -sdk macosx metal`,
+  which is only reachable from a full Xcode install. From a fresh
+  macOS:
+
+  ```sh
+  # 1. Install Xcode from the App Store, then point xcode-select at it:
+  sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
+
+  # 2. If Xcode is present but the Metal Toolchain component is missing:
+  xcodebuild -downloadComponent MetalToolchain
+  ```
+
+  `mix compile` will surface the correct command in its error message
+  if the toolchain is unreachable.
+- **cmake.** Used to build MLX from the vendored source tree. Install
+  via Homebrew (`brew install cmake`) or the equivalent for your
+  setup.
+
+## Building
+
+MLX is vendored as a git submodule under `vendor/mlx` and built from
+source during `mix compile`. There is no prebuilt download step.
+
+### Why vendored?
+
+The vendored pin sits ahead of the latest tagged MLX release because
+Emily depends on two thread-safety changes that were merged to MLX
+`main` after the last release:
+
+- [ml-explore/mlx#3348](https://github.com/ml-explore/mlx/pull/3348) —
+  thread-local `CommandEncoder`. Each MLX stream's Metal encoder lives
+  on the thread that created it, which lets Emily pin a stream to a
+  dedicated worker thread without colliding with other streams.
+- [ml-explore/mlx#3405](https://github.com/ml-explore/mlx/pull/3405) —
+  `ThreadLocalStream` API. Lets the worker thread set its stream as
+  the per-thread default so MLX ops dispatched on that thread go to
+  the right queue without explicit threading at every call site.
+
+Before these landed, concurrent dispatch from multiple OS threads
+would race the Metal driver (the conformance suite hit SIGABRTs and
+SIGSEGVs when the BEAM migrated processes between schedulers). With
+them, Emily can run one MLX stream per worker thread and let multiple
+BEAM processes drive inference concurrently — see the
+[Concurrency model](#concurrency-model) section below.
+
+Once a tagged MLX release contains both PRs, Emily can switch to a
+released version and drop the submodule. Until then, vendoring is the
+cleanest way to pin a known-good `main` commit reproducibly.
+
+### How to build
+
+```sh
+git clone --recurse-submodules https://github.com/ausimian/emily.git
+cd emily
+mix deps.get
+mix compile
+```
 
-## Requirements
+If you cloned without `--recurse-submodules`, initialise them:
 
-- macOS (Apple Silicon recommended; x86_64 supported)
-- Elixir 1.18+ / OTP 27+ (development pinned to 1.19.5 / OTP 28 via `.tool-versions`)
+```sh
+git submodule update --init --recursive
+```
 
-MLX 0.25.1 is fetched as a prebuilt from
-[cocoa-xu/mlx-build](https://github.com/cocoa-xu/mlx-build) during
-`mix compile`; no separate install step.
+The first build takes several minutes to compile MLX itself. The
+artefact (`libmlx.a` + the Metal shader library `mlx.metallib`) is
+cached under `$EMILY_CACHE` (default:
+`~/Library/Caches/emily/mlx-<submodule-hash>`) and reused across
+builds until the submodule pin changes. Override the cache location
+with `EMILY_CACHE=/some/path mix compile`, or force a rebuild with
+`mix compile.emily_mlx --force`.
 
 ## Usage
 
@@ -56,41 +158,54 @@ Nx.Defn.global_default_options(compiler: Emily.Compiler)
 Bumblebee inference works with no further configuration once the
 backend is installed — see the conformance suites under
 `test/emily/conformance/` for worked DistilBERT, Qwen3, ViT, and
-Whisper pipelines.
+Whisper pipelines, and the Notebooks section of the HexDocs nav for
+runnable Livebooks.
 
 The low-level tensor API (`Emily.from_binary/3`, `to_binary/1`,
-`shape/1`, `dtype/1`, `eval/1`) remains available for diagnostics and
-direct MLX round-trips, but most users should go through Nx.
+`shape/1`, `dtype/1`, `eval/1`) remains available for diagnostics
+and direct MLX round-trips, but most users should go through Nx.
 
-## Concurrency
+## Concurrency model
 
-MLX dispatches GPU work through Metal command queues. By default all
-ops share one queue (the default stream), which is not safe for
-concurrent dispatch from multiple OS threads.
+MLX dispatches GPU work through Metal command queues. Emily owns one
+worker thread per command queue; each worker is a dedicated OS thread
+that runs the MLX ops on behalf of BEAM schedulers. NIFs hand their
+work to a worker via a `run_sync` promise (blocks the caller for
+~1–10 µs) rather than executing on the scheduler thread directly,
+which keeps MLX's per-thread `CommandEncoder` consistent and lets
+the BEAM migrate Elixir processes freely.
+
+By default, every op uses the **default worker** owned by the
+`Emily.MlxStream.Default` GenServer under the application supervisor.
+That single queue serialises all GPU work across the VM — correct
+and simple, but a bottleneck under concurrent inference.
 
 **Stream-per-process** — for concurrent inference on a shared model:
 
 ```elixir
 stream = Emily.Stream.new(:gpu)
 
 Emily.Stream.with_stream(stream, fn ->
-  # All Emily ops here dispatch on this stream's command queue.
+  # Every Emily op in this block dispatches on `stream`'s
+  # Metal command queue — concurrent with other streams.
   model.(input)
 end)
 ```
 
-Each stream maps to its own Metal command queue. Multiple processes
-can run inference concurrently — one shared model, no weight
-duplication. Create streams at init time (one per serving process),
-not per-request.
+Each `Emily.Stream` maps to its own `WorkerThread` and its own Metal
+command queue. Weights are shared across streams (MLX arrays are
+refcounted and thread-safe for reads), so the per-stream cost is the
+command buffer, not the model. Create streams once at serving-worker
+init, not per-request.
 
-**Pooled servings** — for simpler setups with small models, start K
-`Nx.Serving` instances behind a pool (poolboy, Registry, etc.). Each
-instance loads its own weights and runs on the default stream. No
-`Emily.Stream` needed. Trade-off: each pool member holds its own
-weight copy.
+**Pooled servings** — for small models where duplicating weights is
+cheap, start K `Nx.Serving` instances behind poolboy / Registry /
+etc. Each instance holds its own weights and runs on the default
+stream. No `Emily.Stream` needed. Trade-off: memory scales linearly
+with K.
 
-See `Emily.Stream` moduledoc for details.
+See `Emily.Stream` for details and the `qwen3_quantized` notebook
+under Notebooks for a worked multi-stream example.
 
 ## Observability
 
@@ -140,41 +255,16 @@ Each check is a per-op MLX reduction plus a scalar readback — a
 worker sync that breaks lazy-graph fusion. Leave off in release
 builds. See the `Emily` moduledoc for the full opt-in snippet.
 
-## Milestones shipped
-
-- **M0** — NIF scaffold, MLX prebuilt fetch, tensor round-trip.
-- **M1** — `Emily.Native` op inventory (creation, unary, binary,
-  reductions, shape, indexing, sort, linalg, FFT, random, memory).
-- **M2** — `Emily.Backend` (`Nx.Backend`). StreamData property oracle
-  vs. `Nx.BinaryBackend`; soak + concurrency harnesses.
-- **M3** — DistilBERT end-to-end on Bumblebee; native batched `dot`,
-  type promotion, `bitcast`.
-- **M4** — Qwen3 (`Qwen/Qwen3-0.6B`) greedy decode end-to-end; native
-  `put_slice` for KV-cache.
-- **M5** — `Emily.Compiler` (`Nx.Defn.Compiler`): validates opts, pins
-  the result backend, delegates the walk to `Nx.Defn.Evaluator`.
-- **M6** — **dropped** after Phase-1 de-risk.
-  `mlx::core::compile` wrapping measured <1.20× on transformer-shaped
-  workloads (regression on CPU) and was cut. Microbench harness
-  retained at `bench/native/compile_microbench.cpp` /
-  `mix bench.native` so the decision can be re-measured against
-  future MLX releases. Full results:
-  [`bench/compile_microbench.md`](bench/compile_microbench.md).
-- **M7** — Bumblebee conformance breadth. ViT
-  (`google/vit-base-patch16-224`) and Whisper (`openai/whisper-tiny`)
-  each ship tiny-random (`@moduletag :conformance`) and
-  full-checkpoint (`@moduletag :vit_full` / `:whisper_full`) tiers.
-  `mix test --only conformance` now aggregates 14 tiny-random tests
-  across DistilBERT, Qwen3, ViT, and Whisper.
-
 ## Testing
 
 ```bash
 mix test                           # fast suite (unit + property)
 mix test --only conformance        # + Bumblebee tiny-random suites
 mix test --only qwen3_full         # full Qwen3-0.6B checkpoint (~1.5 GB)
+mix test --only qwen3_quant_full   # quantized Qwen3-0.6B end-to-end
 mix test --only vit_full           # full ViT-base (~330 MB)
 mix test --only whisper_full       # full whisper-tiny (~150 MB)
+mix test --only training_full      # MNIST convergence canary
 mix test --only soak               # memory + concurrency soak harnesses
 ```