docs(bench): refresh benchmarks against emlx 0.4.1

Bump the emlx pin from ~> 0.3.1 to ~> 0.4.0 in both bench scripts and
rerun the full five-tier suite plus the Qwen3-4B addendum. EMLX 0.4 is
a major performance upgrade: its DistilBERT time roughly halved and its
Qwen3 decode throughput jumped ~5x, while the EXLA and Emily lanes were
flat, so the published Emily-vs-EMLX ratios were badly stale.

- regenerate bench/emily_vs_exla_results.md (emlx 0.4.1, emily 1.0.0)
  and re-append the Qwen3-4B addendum section
- rewrite bench/emily_vs_exla_report.md with the new numbers and a
  'changes since 0.3.1' section
- update the README EMLX comparison: 2.72x/5.82x/~3.2x becomes
  1.44x (DistilBERT) / 1.26x (Qwen3-0.6B) / 1.17x (Qwen3-4B)
- EMLX 0.4.1 still does not complete the ViT-base or Whisper-tiny
  tiers; Emily-vs-EXLA numbers are essentially unchanged
- add RELEASE.md entry

Author: ausimian

README.md

@@ -294,11 +294,11 @@ tensor size**, not model kind:
 | Workload (M4 Pro, f32)                   | Best Emily lane vs EXLA-CPU |
 | ---------------------------------------- | --------------------------- |
 | Large matmul (2048²)                     | **5.0× faster**             |
-| ViT-base image classification            | **2.35× faster**            |
-| Qwen3-0.6B greedy decode                 | **1.67× faster** (`fuse`)   |
-| DistilBERT QA (one encoder forward)      | **1.27× faster**            |
+| ViT-base image classification            | **2.56× faster**            |
+| Qwen3-0.6B greedy decode                 | **1.68× faster** (`fuse`)   |
+| DistilBERT QA (one encoder forward)      | **1.28× faster**            |
 | Whisper-tiny transcription               | **11× slower**              |
-| Elementwise / matmul ≤ ~512 per dim      | up to ~2.3× slower          |
+| Elementwise / matmul ≤ ~512 per dim      | up to ~2.2× slower          |
 
 **Rule of thumb: reach for Emily when the per-op tensors are large** —
 hidden dim ≥ 768 and matmuls ≥ 1024 per dimension, which covers Qwen3,
@@ -311,17 +311,19 @@ with zero fallbacks**, so these gaps are kernel/dispatch efficiency,
 not coverage holes.
 
 **Versus EMLX (GPU-vs-GPU).** The same benchmark runs an `emlx` lane —
-the older MLX-backed Nx backend, also on the Metal GPU — so the
-comparison isn't only against the CPU. Here Emily's *compiler* is the
-differentiator: eager Emily is roughly EMLX-like, but native/fuse pull
-far ahead — **2.72× faster on DistilBERT QA**, **5.82× faster on
-Qwen3-0.6B decode**, and ~3.2× faster on the Qwen3-4B addendum. (EMLX
-did not complete the ViT-base or Whisper-tiny tiers in this harness.)
+the other MLX-backed Nx backend, also on the Metal GPU — so the
+comparison isn't only against the CPU. EMLX 0.4 is a much stronger
+baseline than 0.3 was (its times roughly halved or better), making this
+a compiler-vs-compiler contest: emlx comfortably beats Emily's *eager*
+lane on decode, while Emily's native/fuse lanes keep a consistent lead —
+**1.44× faster on DistilBERT QA**, **1.26× faster on Qwen3-0.6B
+decode**, and **1.17× faster on the Qwen3-4B addendum**. (EMLX did not
+complete the ViT-base or Whisper-tiny tiers in this harness.)
 
 For decode, use the native compiler rather than the eager backend:
-eager Qwen3 decode is 3.2× *slower* than EXLA, while native is 1.5×
-faster and fuse 1.67× — a 5.3× throughput swing from eager to fuse
-(12.51 → 66.42 tok/s) that is purely the per-op dispatch floor, paid
+eager Qwen3 decode is 3.4× *slower* than EXLA, while native is 1.5×
+faster and fuse 1.68× — a 5.6× throughput swing from eager to fuse
+(11.96 → 67.57 tok/s) that is purely the per-op dispatch floor, paid
 once per tiny op across thousands of decode steps.
 
 Re-run the benchmark with `elixir bench/emily_vs_exla.exs`; the full

RELEASE.md

@@ -0,0 +1,8 @@
+### Changed
+
+- Refreshed the published benchmarks against emlx 0.4.1 (was 0.3.1). EMLX 0.4
+  is a much stronger GPU baseline, so the Emily-vs-EMLX comparisons in the
+  README and benchmark report now read 1.44× (DistilBERT QA), 1.26×
+  (Qwen3-0.6B decode) and 1.17× (Qwen3-4B) in Emily's favour, down from
+  2.72×/5.82×/~3.2×. Emily's own lane numbers are essentially unchanged; EMLX
+  still does not complete the ViT-base or Whisper-tiny tiers.

bench/emily_vs_exla.exs

@@ -62,7 +62,7 @@
 Mix.install([
   {:emily, path: Path.expand("..", __DIR__)},
   {:exla, "~> 0.12"},
-  {:emlx, "~> 0.3.1"},
+  {:emlx, "~> 0.4.0"},
   {:bumblebee, "~> 0.7"},
   {:axon, "~> 0.8"},
   {:tokenizers, "~> 0.5"},

bench/emily_vs_exla_report.md

@@ -3,7 +3,7 @@
 This report compares Emily against two Nx backend baselines:
 
 * **EXLA**: XLA host/CPU backend on Apple Silicon.
-* **EMLX**: older MLX-backed Nx backend on the Metal GPU.
+* **EMLX**: MLX-backed Nx backend on the Metal GPU.
 * **Emily**: local MLX/Metal backend, reported across eager, native, and fuse
   lanes. For conclusions, use the best Emily lane for the workload.
 
@@ -21,7 +21,7 @@ Those lanes provide two complementary baselines:
 | Baseline | Question answered |
 | --- | --- |
 | EXLA CPU | Is MLX/Metal GPU faster than XLA host CPU for this workload? |
-| EMLX GPU | Is Emily's compiler/runtime faster than the older MLX-backed Nx stack? |
+| EMLX GPU | Is Emily's compiler/runtime faster than the other MLX-backed Nx stack? |
 
 The focused Qwen3-4B script includes an EXLA lane for explicit experiments, but
 its default run is GPU-only. On this 24 GB M4 Pro, Qwen3-4B bf16 on EXLA-CPU was
@@ -36,33 +36,50 @@ Run on an Apple M4 Pro MacBook Pro with 24 GB RAM.
 | --- | --- |
 | Elixir / OTP | 1.19.5 / 28 |
 | Nx | 0.12.1 |
-| Emily | 0.7.0 local checkout |
-| EMLX | 0.3.1, Metal GPU |
+| Emily | 1.0.0 local checkout |
+| EMLX | 0.4.1, Metal GPU |
 | EXLA | 0.12.0, host CPU client |
 
+## Changes since the emlx 0.3.1 run (2026-06-13)
+
+EMLX 0.4 is a major performance upgrade over 0.3.1, and it reshapes the
+Emily-vs-EMLX story. On the tiers EMLX completes, its times roughly halved
+(DistilBERT 19.19 ms -> 9.97 ms) or better (Qwen3-0.6B decode 11.42 -> 53.81
+tok/s, Qwen3-4B 7.33 -> 20.28 tok/s). The EXLA and Emily lanes moved only a few
+percent between runs, so the delta is the emlx upgrade itself, not machine
+drift.
+
+Concretely, best-Emily-vs-EMLX went from 2.72x to 1.44x on DistilBERT, from
+5.82x to 1.26x on Qwen3-0.6B decode, and from 3.20x to 1.17x on the Qwen3-4B
+addendum. EMLX 0.4.1 still does not complete the ViT-base or Whisper-tiny
+tiers in this harness.
+
 ## Executive summary
 
 Emily's best lane wins the main model tiers that are GPU-friendly:
 
 | Tier | EXLA | EMLX | Best Emily | Best lane | vs EXLA | vs EMLX |
 | --- | ---: | ---: | ---: | --- | ---: | ---: |
-| DistilBERT QA | 8.99 ms | 19.19 ms | 7.06 ms | native | 1.27x faster | 2.72x faster |
-| Qwen3-0.6B decode | 39.84 tok/s | 11.42 tok/s | 66.42 tok/s | fuse | 1.67x faster | 5.82x faster |
-| ViT-base image classification | 56.19 ms | ERR | 23.93 ms | native | 2.35x faster | n/a |
-| Whisper-tiny transcription | 88.49 ms | ERR | 961.68 ms | native | 10.9x slower | n/a |
+| DistilBERT QA | 8.89 ms | 9.97 ms | 6.94 ms | fuse | 1.28x faster | 1.44x faster |
+| Qwen3-0.6B decode | 40.15 tok/s | 53.81 tok/s | 67.57 tok/s | fuse | 1.68x faster | 1.26x faster |
+| ViT-base image classification | 55.88 ms | ERR | 21.79 ms | fuse | 2.56x faster | n/a |
+| Whisper-tiny transcription | 87.83 ms | ERR | 928.35 ms | native | 10.6x slower | n/a |
 
-The Qwen3-4B addendum shows the qualitative Emily-vs-EMLX difference clearly on
-the largest practical Bumblebee model for this machine:
+The Qwen3-4B addendum on the largest practical Bumblebee model for this
+machine:
 
 | Lane | Qwen3-4B tok/s | vs EMLX |
 | --- | ---: | ---: |
-| EMLX | 7.33 | 1.00x |
-| Emily eager | 8.03 | 1.10x |
-| Emily native | 22.27 | 3.04x |
-| Emily fuse | 23.46 | 3.20x |
+| EMLX | 20.28 | 1.00x |
+| Emily eager | 7.88 | 0.39x |
+| Emily native | 22.63 | 1.12x |
+| Emily fuse | 23.78 | 1.17x |
 
-So the headline is: **Emily's compiler path is the differentiator.** Eager is
-roughly EMLX-like on Qwen3-4B; native/fuse are about 3.2x faster.
+The headline is still: **Emily's compiler path is the differentiator** — but
+the shape has changed with emlx 0.4. EMLX's own compiled lane now decisively
+beats op-by-op execution (including Emily's eager lane) on decode, so the
+contest is compiler-vs-compiler, and Emily native/fuse hold a consistent
+1.2-1.4x lead on every tier both stacks complete.
 
 ## Visual summary
 
@@ -71,31 +88,31 @@ xychart-beta
   title "Qwen3-0.6B decode throughput"
   x-axis ["EXLA", "EMLX", "Eager", "Native", "Fuse"]
   y-axis "tokens/sec" 0 --> 70
-  bar [39.84, 11.42, 12.51, 59.96, 66.42]
+  bar [40.15, 53.81, 11.96, 61.70, 67.57]
 ```
 
 ```mermaid
 xychart-beta
   title "Qwen3-4B decode throughput"
   x-axis ["EMLX", "Eager", "Native", "Fuse"]
   y-axis "tokens/sec" 0 --> 25
-  bar [7.33, 8.03, 22.27, 23.46]
+  bar [20.28, 7.88, 22.63, 23.78]
 ```
 
 ```mermaid
 xychart-beta
   title "Best Emily lane vs EXLA"
   x-axis ["DistilBERT", "Qwen3", "ViT", "Whisper"]
   y-axis "speedup x" 0 --> 3
-  bar [1.27, 1.67, 2.35, 0.09]
+  bar [1.28, 1.68, 2.56, 0.09]
 ```
 
 ```mermaid
 xychart-beta
   title "Matmul crossover: best Emily / EXLA"
   x-axis ["128", "512", "1024", "2048"]
   y-axis "ratio (lower is faster)" 0 --> 2
-  line [1.80, 1.12, 0.31, 0.20]
+  line [1.87, 1.20, 0.32, 0.20]
 ```
 
 ## Tier 1 - op microbenchmarks
@@ -108,98 +125,106 @@ Examples from the fresh run:
 
 | Op | Size | Winner | Signal |
 | --- | ---: | --- | --- |
-| add | 256 | EXLA | best Emily is 2.28x slower than EXLA |
-| add | 4096 | Emily | best Emily is 2.0x faster than EXLA |
-| exp | 4096 | Emily | best Emily is 3.2x faster than EXLA |
-| softmax | 4096 | Emily fuse | best Emily is 2.4x faster than EXLA and 1.29x faster than EMLX |
+| add | 256 | EXLA | best Emily is 2.08x slower than EXLA |
+| add | 4096 | Emily | best Emily is 2.2x faster than EXLA |
+| exp | 4096 | Emily | best Emily is 3.3x faster than EXLA |
+| softmax | 4096 | Emily/EMLX tie | both MLX lanes are ~2.6x faster than EXLA |
 | matmul | 2048 | Emily/EMLX tie | both MLX lanes are about 5x faster than EXLA |
 
 Against EMLX, Emily's best op lane is usually close: sometimes a little faster,
-sometimes a little slower. The bigger EMLX-vs-Emily separation appears in traced
-model execution, especially Qwen decode, where Emily native/fuse avoid the
-op-by-op execution shape.
+sometimes a little slower. Per-op parity is expected — both stacks dispatch the
+same MLX kernels — so the EMLX-vs-Emily separation appears in traced model
+execution, where compiler and dispatch strategy differ.
 
 ## Tier 2 - DistilBERT QA
 
-DistilBERT is a clean three-way win for Emily native:
+DistilBERT is a three-way win for Emily native/fuse:
 
 | Lane | ms/call |
 | --- | ---: |
-| EXLA CPU | 8.99 |
-| EMLX GPU | 19.19 |
-| Emily eager | 16.45 |
-| Emily native | 7.06 |
-| Emily fuse | 8.50 |
+| EXLA CPU | 8.89 |
+| EMLX GPU | 9.97 |
+| Emily eager | 14.82 |
+| Emily native | 7.02 |
+| Emily fuse | 6.94 |
 
-Native is the right Emily option here. Fuse is not universally better; it helps
-most when a compiled body is reused, such as decode loops.
+Native and fuse are within a few percent of each other here; either is the
+right Emily option for a single-forward workload. The EMLX lane roughly halved
+its 0.3.1 time but showed high run-to-run variance in this run (11.7 / 12.4 /
+5.8 ms), so treat its mean with some caution.
 
 ## Tier 3 - Qwen3-0.6B decode
 
 Qwen3-0.6B is the canonical completed three-way generation benchmark:
 
 | Lane | tok/s |
 | --- | ---: |
-| EXLA CPU | 39.84 |
-| EMLX GPU | 11.42 |
-| Emily eager | 12.51 |
-| Emily native | 59.96 |
-| Emily fuse | 66.42 |
+| EXLA CPU | 40.15 |
+| EMLX GPU | 53.81 |
+| Emily eager | 11.96 |
+| Emily native | 61.70 |
+| Emily fuse | 67.57 |
 
-This is the clearest main-suite result. Emily eager is near EMLX, while
-native/fuse jump far ahead. Fuse is the best choice for decode loops.
+The 0.4 emlx lane now comfortably beats both EXLA-CPU and Emily's eager lane —
+op-by-op decode is simply not competitive from either stack. Emily native/fuse
+stay ahead at 1.15-1.26x over EMLX. Fuse is the best choice for decode loops.
 
 ## Tier 4 - ViT-base image classification
 
-ViT-base strongly favors Emily native:
+ViT-base strongly favors Emily:
 
 | Lane | ms/call |
 | --- | ---: |
-| EXLA CPU | 56.19 |
+| EXLA CPU | 55.88 |
 | EMLX GPU | ERR |
-| Emily eager | 38.78 |
-| Emily native | 23.93 |
-| Emily fuse | 27.50 |
+| Emily eager | 35.58 |
+| Emily native | 23.96 |
+| Emily fuse | 21.79 |
 
 This tier is GPU-friendly: larger matrix multiplies and enough work per forward
-for the GPU path to dominate. The EMLX lane did not complete in this harness, so
-the meaningful comparison here is Emily vs EXLA.
+for the GPU path to dominate. The EMLX lane did not complete in this harness
+(on 0.3.1 or 0.4.1), so the meaningful comparison here is Emily vs EXLA.
 
 ## Tier 5 - Whisper-tiny transcription
 
 Whisper-tiny remains Emily's bad case:
 
 | Lane | ms/call |
 | --- | ---: |
-| EXLA CPU | 88.49 |
+| EXLA CPU | 87.83 |
 | EMLX GPU | ERR |
-| Emily eager | 1815.80 |
-| Emily native | 961.68 |
-| Emily fuse | 981.89 |
+| Emily eager | 1920.57 |
+| Emily native | 928.35 |
+| Emily fuse | 955.36 |
 
 This is not a coverage win for EXLA; the Emily lanes reported zero fallbacks in
 the live run. It is a workload-shape problem: Whisper-tiny is made of many small
 kernels where CPU launch overhead and cache locality beat GPU dispatch. Native
 cuts eager roughly in half, but still cannot remove the underlying small-kernel
-cost. Fuse does not help this workload.
+cost. Fuse does not help this workload. The EMLX lane did not complete this
+tier on 0.3.1 or 0.4.1.
 
 ## Qwen3-4B addendum
 
-The focused Qwen3-4B script was rerun after the three-way reorganization. Its
-safe default remains GPU-only:
+The focused Qwen3-4B script's safe default remains GPU-only:
 
 ```sh
 elixir bench/qwen3_4b_emily_vs_emlx.exs
 ```
 
-Fresh result:
+Fresh result (emlx 0.4.1):
 
 | Lane | mean tok/s | min | max | vs EMLX |
 | --- | ---: | ---: | ---: | ---: |
-| EMLX | 7.33 | 7.31 | 7.37 | 1.00x |
-| Emily eager | 8.03 | 7.98 | 8.08 | 1.10x |
-| Emily native | 22.27 | 22.16 | 22.32 | 3.04x |
-| Emily fuse | 23.46 | 23.35 | 23.58 | 3.20x |
+| EMLX | 20.28 | 20.07 | 20.44 | 1.00x |
+| Emily eager | 7.88 | 7.81 | 7.97 | 0.39x |
+| Emily native | 22.63 | 22.61 | 22.64 | 1.12x |
+| Emily fuse | 23.78 | 23.55 | 23.97 | 1.17x |
+
+All four lanes produced the identical greedy completion, so the throughput
+comparison is on the same decode. Note the Emily lanes are essentially
+unchanged from the 0.3.1-era run (native 22.27 -> 22.63, fuse 23.46 -> 23.78)
+while EMLX jumped 7.33 -> 20.28 tok/s.
 
 An explicit EXLA smoke attempt on Qwen3-4B:
 
@@ -218,9 +243,13 @@ used as the canonical three-way comparison on this 24 GB machine.
 Use `emily-fuse` for autoregressive decode loops. It is best on Qwen3-0.6B and
 Qwen3-4B because the loop body is reused.
 
-Use `emily-native` as the default best Emily lane for single-forward model
-benchmarks. It wins DistilBERT and ViT here; fuse can be neutral or slower when
-there is no repeated body to amortize.
+Use `emily-native` or `emily-fuse` for single-forward model workloads — they
+are within a few percent of each other on DistilBERT and ViT. Fuse's edge grows
+when a compiled body is reused, as in decode.
+
+Avoid op-by-op execution for decode from either stack: Emily eager and
+pre-compiler EMLX both sit near 12 tok/s on Qwen3-0.6B where the compiled lanes
+reach 54-68 tok/s.
 
 Keep EXLA out of the default Qwen3-4B run on 24 GB machines. The EXLA 4B smoke
 was killed by the OS, while Qwen3-0.6B gives a completed three-way generation