diff --git a/README.md b/README.md index 23a7668..0fdae2a 100644 --- a/README.md +++ b/README.md @@ -294,11 +294,11 @@ tensor size**, not model kind: | Workload (M4 Pro, f32) | Best Emily lane vs EXLA-CPU | | ---------------------------------------- | --------------------------- | | Large matmul (2048²) | **5.0× faster** | -| ViT-base image classification | **2.35× faster** | -| Qwen3-0.6B greedy decode | **1.67× faster** (`fuse`) | -| DistilBERT QA (one encoder forward) | **1.27× faster** | +| ViT-base image classification | **2.56× faster** | +| Qwen3-0.6B greedy decode | **1.68× faster** (`fuse`) | +| DistilBERT QA (one encoder forward) | **1.28× faster** | | Whisper-tiny transcription | **11× slower** | -| Elementwise / matmul ≤ ~512 per dim | up to ~2.3× slower | +| Elementwise / matmul ≤ ~512 per dim | up to ~2.2× slower | **Rule of thumb: reach for Emily when the per-op tensors are large** — hidden dim ≥ 768 and matmuls ≥ 1024 per dimension, which covers Qwen3, @@ -311,17 +311,19 @@ with zero fallbacks**, so these gaps are kernel/dispatch efficiency, not coverage holes. **Versus EMLX (GPU-vs-GPU).** The same benchmark runs an `emlx` lane — -the older MLX-backed Nx backend, also on the Metal GPU — so the -comparison isn't only against the CPU. Here Emily's *compiler* is the -differentiator: eager Emily is roughly EMLX-like, but native/fuse pull -far ahead — **2.72× faster on DistilBERT QA**, **5.82× faster on -Qwen3-0.6B decode**, and ~3.2× faster on the Qwen3-4B addendum. (EMLX -did not complete the ViT-base or Whisper-tiny tiers in this harness.) +the other MLX-backed Nx backend, also on the Metal GPU — so the +comparison isn't only against the CPU. EMLX 0.4 is a much stronger +baseline than 0.3 was (its times roughly halved or better), making this +a compiler-vs-compiler contest: emlx comfortably beats Emily's *eager* +lane on decode, while Emily's native/fuse lanes keep a consistent lead — +**1.44× faster on DistilBERT QA**, **1.26× faster on Qwen3-0.6B +decode**, and **1.17× faster on the Qwen3-4B addendum**. (EMLX did not +complete the ViT-base or Whisper-tiny tiers in this harness.) For decode, use the native compiler rather than the eager backend: -eager Qwen3 decode is 3.2× *slower* than EXLA, while native is 1.5× -faster and fuse 1.67× — a 5.3× throughput swing from eager to fuse -(12.51 → 66.42 tok/s) that is purely the per-op dispatch floor, paid +eager Qwen3 decode is 3.4× *slower* than EXLA, while native is 1.5× +faster and fuse 1.68× — a 5.6× throughput swing from eager to fuse +(11.96 → 67.57 tok/s) that is purely the per-op dispatch floor, paid once per tiny op across thousands of decode steps. Re-run the benchmark with `elixir bench/emily_vs_exla.exs`; the full diff --git a/RELEASE.md b/RELEASE.md index e69de29..d2da9b7 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -0,0 +1,8 @@ +### Changed + +- Refreshed the published benchmarks against emlx 0.4.1 (was 0.3.1). EMLX 0.4 + is a much stronger GPU baseline, so the Emily-vs-EMLX comparisons in the + README and benchmark report now read 1.44× (DistilBERT QA), 1.26× + (Qwen3-0.6B decode) and 1.17× (Qwen3-4B) in Emily's favour, down from + 2.72×/5.82×/~3.2×. Emily's own lane numbers are essentially unchanged; EMLX + still does not complete the ViT-base or Whisper-tiny tiers. diff --git a/bench/emily_vs_exla.exs b/bench/emily_vs_exla.exs index ca7cf72..03d97e7 100644 --- a/bench/emily_vs_exla.exs +++ b/bench/emily_vs_exla.exs @@ -62,7 +62,7 @@ Mix.install([ {:emily, path: Path.expand("..", __DIR__)}, {:exla, "~> 0.12"}, - {:emlx, "~> 0.3.1"}, + {:emlx, "~> 0.4.0"}, {:bumblebee, "~> 0.7"}, {:axon, "~> 0.8"}, {:tokenizers, "~> 0.5"}, diff --git a/bench/emily_vs_exla_report.md b/bench/emily_vs_exla_report.md index c480981..0beb74d 100644 --- a/bench/emily_vs_exla_report.md +++ b/bench/emily_vs_exla_report.md @@ -3,7 +3,7 @@ This report compares Emily against two Nx backend baselines: * **EXLA**: XLA host/CPU backend on Apple Silicon. -* **EMLX**: older MLX-backed Nx backend on the Metal GPU. +* **EMLX**: MLX-backed Nx backend on the Metal GPU. * **Emily**: local MLX/Metal backend, reported across eager, native, and fuse lanes. For conclusions, use the best Emily lane for the workload. @@ -21,7 +21,7 @@ Those lanes provide two complementary baselines: | Baseline | Question answered | | --- | --- | | EXLA CPU | Is MLX/Metal GPU faster than XLA host CPU for this workload? | -| EMLX GPU | Is Emily's compiler/runtime faster than the older MLX-backed Nx stack? | +| EMLX GPU | Is Emily's compiler/runtime faster than the other MLX-backed Nx stack? | The focused Qwen3-4B script includes an EXLA lane for explicit experiments, but its default run is GPU-only. On this 24 GB M4 Pro, Qwen3-4B bf16 on EXLA-CPU was @@ -36,33 +36,50 @@ Run on an Apple M4 Pro MacBook Pro with 24 GB RAM. | --- | --- | | Elixir / OTP | 1.19.5 / 28 | | Nx | 0.12.1 | -| Emily | 0.7.0 local checkout | -| EMLX | 0.3.1, Metal GPU | +| Emily | 1.0.0 local checkout | +| EMLX | 0.4.1, Metal GPU | | EXLA | 0.12.0, host CPU client | +## Changes since the emlx 0.3.1 run (2026-06-13) + +EMLX 0.4 is a major performance upgrade over 0.3.1, and it reshapes the +Emily-vs-EMLX story. On the tiers EMLX completes, its times roughly halved +(DistilBERT 19.19 ms -> 9.97 ms) or better (Qwen3-0.6B decode 11.42 -> 53.81 +tok/s, Qwen3-4B 7.33 -> 20.28 tok/s). The EXLA and Emily lanes moved only a few +percent between runs, so the delta is the emlx upgrade itself, not machine +drift. + +Concretely, best-Emily-vs-EMLX went from 2.72x to 1.44x on DistilBERT, from +5.82x to 1.26x on Qwen3-0.6B decode, and from 3.20x to 1.17x on the Qwen3-4B +addendum. EMLX 0.4.1 still does not complete the ViT-base or Whisper-tiny +tiers in this harness. + ## Executive summary Emily's best lane wins the main model tiers that are GPU-friendly: | Tier | EXLA | EMLX | Best Emily | Best lane | vs EXLA | vs EMLX | | --- | ---: | ---: | ---: | --- | ---: | ---: | -| DistilBERT QA | 8.99 ms | 19.19 ms | 7.06 ms | native | 1.27x faster | 2.72x faster | -| Qwen3-0.6B decode | 39.84 tok/s | 11.42 tok/s | 66.42 tok/s | fuse | 1.67x faster | 5.82x faster | -| ViT-base image classification | 56.19 ms | ERR | 23.93 ms | native | 2.35x faster | n/a | -| Whisper-tiny transcription | 88.49 ms | ERR | 961.68 ms | native | 10.9x slower | n/a | +| DistilBERT QA | 8.89 ms | 9.97 ms | 6.94 ms | fuse | 1.28x faster | 1.44x faster | +| Qwen3-0.6B decode | 40.15 tok/s | 53.81 tok/s | 67.57 tok/s | fuse | 1.68x faster | 1.26x faster | +| ViT-base image classification | 55.88 ms | ERR | 21.79 ms | fuse | 2.56x faster | n/a | +| Whisper-tiny transcription | 87.83 ms | ERR | 928.35 ms | native | 10.6x slower | n/a | -The Qwen3-4B addendum shows the qualitative Emily-vs-EMLX difference clearly on -the largest practical Bumblebee model for this machine: +The Qwen3-4B addendum on the largest practical Bumblebee model for this +machine: | Lane | Qwen3-4B tok/s | vs EMLX | | --- | ---: | ---: | -| EMLX | 7.33 | 1.00x | -| Emily eager | 8.03 | 1.10x | -| Emily native | 22.27 | 3.04x | -| Emily fuse | 23.46 | 3.20x | +| EMLX | 20.28 | 1.00x | +| Emily eager | 7.88 | 0.39x | +| Emily native | 22.63 | 1.12x | +| Emily fuse | 23.78 | 1.17x | -So the headline is: **Emily's compiler path is the differentiator.** Eager is -roughly EMLX-like on Qwen3-4B; native/fuse are about 3.2x faster. +The headline is still: **Emily's compiler path is the differentiator** — but +the shape has changed with emlx 0.4. EMLX's own compiled lane now decisively +beats op-by-op execution (including Emily's eager lane) on decode, so the +contest is compiler-vs-compiler, and Emily native/fuse hold a consistent +1.2-1.4x lead on every tier both stacks complete. ## Visual summary @@ -71,7 +88,7 @@ xychart-beta title "Qwen3-0.6B decode throughput" x-axis ["EXLA", "EMLX", "Eager", "Native", "Fuse"] y-axis "tokens/sec" 0 --> 70 - bar [39.84, 11.42, 12.51, 59.96, 66.42] + bar [40.15, 53.81, 11.96, 61.70, 67.57] ``` ```mermaid @@ -79,7 +96,7 @@ xychart-beta title "Qwen3-4B decode throughput" x-axis ["EMLX", "Eager", "Native", "Fuse"] y-axis "tokens/sec" 0 --> 25 - bar [7.33, 8.03, 22.27, 23.46] + bar [20.28, 7.88, 22.63, 23.78] ``` ```mermaid @@ -87,7 +104,7 @@ xychart-beta title "Best Emily lane vs EXLA" x-axis ["DistilBERT", "Qwen3", "ViT", "Whisper"] y-axis "speedup x" 0 --> 3 - bar [1.27, 1.67, 2.35, 0.09] + bar [1.28, 1.68, 2.56, 0.09] ``` ```mermaid @@ -95,7 +112,7 @@ xychart-beta title "Matmul crossover: best Emily / EXLA" x-axis ["128", "512", "1024", "2048"] y-axis "ratio (lower is faster)" 0 --> 2 - line [1.80, 1.12, 0.31, 0.20] + line [1.87, 1.20, 0.32, 0.20] ``` ## Tier 1 - op microbenchmarks @@ -108,31 +125,33 @@ Examples from the fresh run: | Op | Size | Winner | Signal | | --- | ---: | --- | --- | -| add | 256 | EXLA | best Emily is 2.28x slower than EXLA | -| add | 4096 | Emily | best Emily is 2.0x faster than EXLA | -| exp | 4096 | Emily | best Emily is 3.2x faster than EXLA | -| softmax | 4096 | Emily fuse | best Emily is 2.4x faster than EXLA and 1.29x faster than EMLX | +| add | 256 | EXLA | best Emily is 2.08x slower than EXLA | +| add | 4096 | Emily | best Emily is 2.2x faster than EXLA | +| exp | 4096 | Emily | best Emily is 3.3x faster than EXLA | +| softmax | 4096 | Emily/EMLX tie | both MLX lanes are ~2.6x faster than EXLA | | matmul | 2048 | Emily/EMLX tie | both MLX lanes are about 5x faster than EXLA | Against EMLX, Emily's best op lane is usually close: sometimes a little faster, -sometimes a little slower. The bigger EMLX-vs-Emily separation appears in traced -model execution, especially Qwen decode, where Emily native/fuse avoid the -op-by-op execution shape. +sometimes a little slower. Per-op parity is expected — both stacks dispatch the +same MLX kernels — so the EMLX-vs-Emily separation appears in traced model +execution, where compiler and dispatch strategy differ. ## Tier 2 - DistilBERT QA -DistilBERT is a clean three-way win for Emily native: +DistilBERT is a three-way win for Emily native/fuse: | Lane | ms/call | | --- | ---: | -| EXLA CPU | 8.99 | -| EMLX GPU | 19.19 | -| Emily eager | 16.45 | -| Emily native | 7.06 | -| Emily fuse | 8.50 | +| EXLA CPU | 8.89 | +| EMLX GPU | 9.97 | +| Emily eager | 14.82 | +| Emily native | 7.02 | +| Emily fuse | 6.94 | -Native is the right Emily option here. Fuse is not universally better; it helps -most when a compiled body is reused, such as decode loops. +Native and fuse are within a few percent of each other here; either is the +right Emily option for a single-forward workload. The EMLX lane roughly halved +its 0.3.1 time but showed high run-to-run variance in this run (11.7 / 12.4 / +5.8 ms), so treat its mean with some caution. ## Tier 3 - Qwen3-0.6B decode @@ -140,30 +159,31 @@ Qwen3-0.6B is the canonical completed three-way generation benchmark: | Lane | tok/s | | --- | ---: | -| EXLA CPU | 39.84 | -| EMLX GPU | 11.42 | -| Emily eager | 12.51 | -| Emily native | 59.96 | -| Emily fuse | 66.42 | +| EXLA CPU | 40.15 | +| EMLX GPU | 53.81 | +| Emily eager | 11.96 | +| Emily native | 61.70 | +| Emily fuse | 67.57 | -This is the clearest main-suite result. Emily eager is near EMLX, while -native/fuse jump far ahead. Fuse is the best choice for decode loops. +The 0.4 emlx lane now comfortably beats both EXLA-CPU and Emily's eager lane — +op-by-op decode is simply not competitive from either stack. Emily native/fuse +stay ahead at 1.15-1.26x over EMLX. Fuse is the best choice for decode loops. ## Tier 4 - ViT-base image classification -ViT-base strongly favors Emily native: +ViT-base strongly favors Emily: | Lane | ms/call | | --- | ---: | -| EXLA CPU | 56.19 | +| EXLA CPU | 55.88 | | EMLX GPU | ERR | -| Emily eager | 38.78 | -| Emily native | 23.93 | -| Emily fuse | 27.50 | +| Emily eager | 35.58 | +| Emily native | 23.96 | +| Emily fuse | 21.79 | This tier is GPU-friendly: larger matrix multiplies and enough work per forward -for the GPU path to dominate. The EMLX lane did not complete in this harness, so -the meaningful comparison here is Emily vs EXLA. +for the GPU path to dominate. The EMLX lane did not complete in this harness +(on 0.3.1 or 0.4.1), so the meaningful comparison here is Emily vs EXLA. ## Tier 5 - Whisper-tiny transcription @@ -171,35 +191,40 @@ Whisper-tiny remains Emily's bad case: | Lane | ms/call | | --- | ---: | -| EXLA CPU | 88.49 | +| EXLA CPU | 87.83 | | EMLX GPU | ERR | -| Emily eager | 1815.80 | -| Emily native | 961.68 | -| Emily fuse | 981.89 | +| Emily eager | 1920.57 | +| Emily native | 928.35 | +| Emily fuse | 955.36 | This is not a coverage win for EXLA; the Emily lanes reported zero fallbacks in the live run. It is a workload-shape problem: Whisper-tiny is made of many small kernels where CPU launch overhead and cache locality beat GPU dispatch. Native cuts eager roughly in half, but still cannot remove the underlying small-kernel -cost. Fuse does not help this workload. +cost. Fuse does not help this workload. The EMLX lane did not complete this +tier on 0.3.1 or 0.4.1. ## Qwen3-4B addendum -The focused Qwen3-4B script was rerun after the three-way reorganization. Its -safe default remains GPU-only: +The focused Qwen3-4B script's safe default remains GPU-only: ```sh elixir bench/qwen3_4b_emily_vs_emlx.exs ``` -Fresh result: +Fresh result (emlx 0.4.1): | Lane | mean tok/s | min | max | vs EMLX | | --- | ---: | ---: | ---: | ---: | -| EMLX | 7.33 | 7.31 | 7.37 | 1.00x | -| Emily eager | 8.03 | 7.98 | 8.08 | 1.10x | -| Emily native | 22.27 | 22.16 | 22.32 | 3.04x | -| Emily fuse | 23.46 | 23.35 | 23.58 | 3.20x | +| EMLX | 20.28 | 20.07 | 20.44 | 1.00x | +| Emily eager | 7.88 | 7.81 | 7.97 | 0.39x | +| Emily native | 22.63 | 22.61 | 22.64 | 1.12x | +| Emily fuse | 23.78 | 23.55 | 23.97 | 1.17x | + +All four lanes produced the identical greedy completion, so the throughput +comparison is on the same decode. Note the Emily lanes are essentially +unchanged from the 0.3.1-era run (native 22.27 -> 22.63, fuse 23.46 -> 23.78) +while EMLX jumped 7.33 -> 20.28 tok/s. An explicit EXLA smoke attempt on Qwen3-4B: @@ -218,9 +243,13 @@ used as the canonical three-way comparison on this 24 GB machine. Use `emily-fuse` for autoregressive decode loops. It is best on Qwen3-0.6B and Qwen3-4B because the loop body is reused. -Use `emily-native` as the default best Emily lane for single-forward model -benchmarks. It wins DistilBERT and ViT here; fuse can be neutral or slower when -there is no repeated body to amortize. +Use `emily-native` or `emily-fuse` for single-forward model workloads — they +are within a few percent of each other on DistilBERT and ViT. Fuse's edge grows +when a compiled body is reused, as in decode. + +Avoid op-by-op execution for decode from either stack: Emily eager and +pre-compiler EMLX both sit near 12 tok/s on Qwen3-0.6B where the compiled lanes +reach 54-68 tok/s. Keep EXLA out of the default Qwen3-4B run on 24 GB machines. The EXLA 4B smoke was killed by the OS, while Qwen3-0.6B gives a completed three-way generation diff --git a/bench/emily_vs_exla_results.md b/bench/emily_vs_exla_results.md index 3cc40d8..6667455 100644 --- a/bench/emily_vs_exla_results.md +++ b/bench/emily_vs_exla_results.md @@ -9,11 +9,11 @@ and EMLX baselines._ | Field | Value | | ----- | ----- | -| date | 2026-06-13 12:24 | -| host | Apple M4 Pro (BEAM total 116 MB at write) | +| date | 2026-07-22 18:56 | +| host | Apple M4 Pro (BEAM total 118 MB at write) | | elixir / otp | 1.19.5 / 28 | -| emily | 0.6.1 | -| emlx | 0.3.1 | +| emily | 1.0.0 | +| emlx | 0.4.1 | | exla | 0.12.0 (host (1 device)) | | nx | 0.12.1 | | smoke run? | false | @@ -24,65 +24,65 @@ _Ratios are `best Emily / baseline`; <1 means Emily is faster._ | op | size | exla (CPU) | emlx (GPU) | emily-eager | emily-native | emily-fuse | best-emily/exla | best-emily/emlx | | -- | ---- | ---------- | ---------- | ----------- | ------------ | ---------- | --------------- | --------------- | -| add | 256 | 112.0 | 343.6 | 276.9 | 296.7 | 255.2 | 2.28x | 0.74x | -| add | 1024 | 258.7 | 417.4 | 423.6 | 491.8 | 486.3 | 1.64x | 1.02x | -| add | 4096 | 2972.8 | 1536.0 | 1489.3 | 1523.0 | 1545.2 | 0.5x | 0.97x | -| mul | 256 | 107.0 | 324.1 | 247.2 | 228.9 | 220.1 | 2.06x | 0.68x | -| mul | 1024 | 242.1 | 438.4 | 418.0 | 489.5 | 480.2 | 1.73x | 0.95x | -| mul | 4096 | 2764.0 | 1542.5 | 1639.6 | 1461.4 | 1478.5 | 0.53x | 0.95x | -| exp | 256 | 134.2 | 297.1 | 219.1 | 246.0 | 211.1 | 1.57x | 0.71x | -| exp | 1024 | 381.4 | 467.7 | 420.1 | 517.2 | 515.6 | 1.1x | 0.9x | -| exp | 4096 | 3634.1 | 1223.5 | 1231.6 | 1140.6 | 1234.7 | 0.31x | 0.93x | -| sum | 256 | 94.0 | 223.4 | 176.4 | 194.1 | 203.1 | 1.88x | 0.79x | -| sum | 1024 | 203.4 | 351.0 | 331.9 | 315.6 | 355.0 | 1.55x | 0.9x | -| sum | 4096 | 1882.1 | 649.2 | 776.9 | 714.3 | 817.8 | 0.38x | 1.1x | -| softmax | 256 | 164.2 | 416.8 | 283.9 | 246.8 | 295.3 | 1.5x | 0.59x | -| softmax | 1024 | 503.8 | 786.0 | 758.1 | 743.5 | 721.0 | 1.43x | 0.92x | -| softmax | 4096 | 5433.8 | 2891.4 | 2889.6 | 2771.8 | 2244.5 | 0.41x | 0.78x | -| matmul | 128 | 106.5 | 215.1 | 191.8 | 200.8 | 204.8 | 1.8x | 0.89x | -| matmul | 512 | 459.6 | 692.0 | 642.3 | 515.5 | 643.2 | 1.12x | 0.74x | -| matmul | 1024 | 2577.2 | 866.2 | 937.4 | 1028.0 | 801.9 | 0.31x | 0.93x | -| matmul | 2048 | 17864.3 | 3578.4 | 3557.2 | 3593.9 | 3582.5 | 0.2x | 0.99x | +| add | 256 | 105.0 | 252.8 | 240.2 | 225.3 | 218.1 | 2.08x | 0.86x | +| add | 1024 | 232.9 | 359.8 | 343.7 | 377.7 | 361.8 | 1.48x | 0.96x | +| add | 4096 | 3083.4 | 1459.2 | 1629.4 | 1435.1 | 1423.9 | 0.46x | 0.98x | +| mul | 256 | 109.2 | 196.9 | 200.0 | 210.0 | 221.9 | 1.83x | 1.02x | +| mul | 1024 | 229.1 | 381.1 | 377.9 | 350.2 | 353.4 | 1.53x | 0.92x | +| mul | 4096 | 2893.7 | 1400.2 | 1410.0 | 1379.8 | 1426.2 | 0.48x | 0.99x | +| exp | 256 | 108.2 | 194.8 | 175.8 | 198.4 | 198.8 | 1.62x | 0.9x | +| exp | 1024 | 328.5 | 327.2 | 322.1 | 330.0 | 336.7 | 0.98x | 0.98x | +| exp | 4096 | 3682.9 | 1101.0 | 1156.0 | 1112.4 | 1175.2 | 0.3x | 1.01x | +| sum | 256 | 81.8 | 215.3 | 180.6 | 191.0 | 187.7 | 2.21x | 0.84x | +| sum | 1024 | 194.1 | 253.0 | 227.2 | 250.6 | 248.3 | 1.17x | 0.9x | +| sum | 4096 | 2017.7 | 722.3 | 730.3 | 588.7 | 824.0 | 0.29x | 0.82x | +| softmax | 256 | 138.7 | 250.0 | 276.5 | 276.2 | 272.0 | 1.96x | 1.09x | +| softmax | 1024 | 472.4 | 518.9 | 598.9 | 569.2 | 513.0 | 1.09x | 0.99x | +| softmax | 4096 | 5697.2 | 2172.1 | 2765.2 | 2700.8 | 2196.1 | 0.39x | 1.01x | +| matmul | 128 | 100.8 | 209.4 | 189.0 | 214.9 | 188.4 | 1.87x | 0.9x | +| matmul | 512 | 449.2 | 540.6 | 537.6 | 559.2 | 560.6 | 1.2x | 0.99x | +| matmul | 1024 | 2484.9 | 819.4 | 931.5 | 797.8 | 830.0 | 0.32x | 0.97x | +| matmul | 2048 | 17717.1 | 3533.9 | 3503.6 | 3512.6 | 3545.0 | 0.2x | 0.99x | ## Tier 2 — DistilBERT QA (mean us/call) | lane | us/call | | ---- | ------------- | -| exla (CPU) | 8985.0 | -| emlx (GPU) | 19194.3 | -| emily-eager (GPU) | 16452.7 | -| emily-native (GPU) | 7058.7 | -| emily-fuse (GPU) | 8504.3 | +| exla (CPU) | 8890.0 | +| emlx (GPU) | 9968.7 | +| emily-eager (GPU) | 14820.7 | +| emily-native (GPU) | 7020.7 | +| emily-fuse (GPU) | 6937.7 | -_best Emily lane vs EXLA (>1 = Emily faster): 1.27x_ +_best Emily lane vs EXLA (>1 = Emily faster): 1.28x_ -_best Emily lane vs EMLX (>1 = Emily faster): 2.72x_ +_best Emily lane vs EMLX (>1 = Emily faster): 1.44x_ ## Tier 3 — Qwen3-0.6B decode (tokens/sec) | lane | tok/s | | ---- | ------------- | -| exla (CPU) | 39.84 | -| emlx (GPU) | 11.42 | -| emily-eager (GPU) | 12.51 | -| emily-native (GPU) | 59.96 | -| emily-fuse (GPU) | 66.42 | +| exla (CPU) | 40.15 | +| emlx (GPU) | 53.81 | +| emily-eager (GPU) | 11.96 | +| emily-native (GPU) | 61.7 | +| emily-fuse (GPU) | 67.57 | -_best Emily lane vs EXLA (>1 = Emily faster): 1.67x_ +_best Emily lane vs EXLA (>1 = Emily faster): 1.68x_ -_best Emily lane vs EMLX (>1 = Emily faster): 5.82x_ +_best Emily lane vs EMLX (>1 = Emily faster): 1.26x_ ## Tier 4 — ViT-base image classification (mean us/call) | lane | us/call | | ---- | ------------- | -| exla (CPU) | 56191.3 | +| exla (CPU) | 55880.3 | | emlx (GPU) | ERR | -| emily-eager (GPU) | 38780.3 | -| emily-native (GPU) | 23934.3 | -| emily-fuse (GPU) | 27498.0 | +| emily-eager (GPU) | 35582.3 | +| emily-native (GPU) | 23958.7 | +| emily-fuse (GPU) | 21790.7 | -_best Emily lane vs EXLA (>1 = Emily faster): 2.35x_ +_best Emily lane vs EXLA (>1 = Emily faster): 2.56x_ _best Emily lane vs EMLX (>1 = Emily faster): —_ @@ -90,38 +90,27 @@ _best Emily lane vs EMLX (>1 = Emily faster): —_ | lane | us/call | | ---- | ------------- | -| exla (CPU) | 88488.7 | +| exla (CPU) | 87834.0 | | emlx (GPU) | ERR | -| emily-eager (GPU) | 1815796.0 | -| emily-native (GPU) | 961683.7 | -| emily-fuse (GPU) | 981888.0 | +| emily-eager (GPU) | 1920572.3 | +| emily-native (GPU) | 928346.0 | +| emily-fuse (GPU) | 955357.3 | _best Emily lane vs EXLA (>1 = Emily faster): 0.09x_ _best Emily lane vs EMLX (>1 = Emily faster): —_ -## Addendum — Qwen3-4B decode (GPU-focused) +## Qwen3-4B addendum (tokens/sec) -_Manual addendum from `bench/qwen3_4b_emily_vs_emlx.exs`, rerun after the main -three-way suite. Higher is better. The script has an explicit EXLA lane, but the -safe default for Qwen3-4B on this 24 GB M4 Pro is GPU-only._ +_From `bench/qwen3_4b_emily_vs_emlx.exs` (same date/host, bf16, 32 new tokens, +3 runs; GPU-only default — the EXLA lane is OOM-killed on this 24 GB machine)._ -| lane | mean tok/s | min | max | speedup vs EMLX | -| ---- | ----------: | --: | --: | --------------: | -| emlx (GPU) | 7.33 | 7.31 | 7.37 | 1.00x | -| emily-eager (GPU) | 8.03 | 7.98 | 8.08 | 1.10x | -| emily-native (GPU) | 22.27 | 22.16 | 22.32 | 3.04x | -| emily-fuse (GPU) | 23.46 | 23.35 | 23.58 | 3.20x | +| lane | mean tok/s | min | max | vs EMLX | +| ---- | ---------: | --: | --: | ------: | +| emlx (GPU) | 20.28 | 20.07 | 20.44 | 1.00x | +| emily-eager (GPU) | 7.88 | 7.81 | 7.97 | 0.39x | +| emily-native (GPU) | 22.63 | 22.61 | 22.64 | 1.12x | +| emily-fuse (GPU) | 23.78 | 23.55 | 23.97 | 1.17x | -_Best Emily lane vs EMLX: emily-fuse at 23.46 tok/s, 3.20x faster._ +_best Emily lane vs EMLX (>1 = Emily faster): 1.17x_ -### Qwen3-4B EXLA note - -An explicit EXLA smoke attempt was run with: - -`EMILY_BENCH_NEW_TOKENS=4 EMILY_BENCH_RUNS=1 EMILY_BENCH_WARMUP=0 EMILY_BENCH_LANES=exla,emlx,emily-fuse elixir bench/qwen3_4b_emily_vs_emlx.exs` - -It loaded `Qwen/Qwen3-4B` on `EXLA.Backend` as `:bf16` in 4.86 s, then the -process was killed with exit 137 during the EXLA compile/run. No Qwen3-4B EXLA -throughput number is reported; `bench/emily_vs_exla.exs` uses Qwen3-0.6B for the -canonical completed three-way generation comparison. diff --git a/bench/qwen3_4b_emily_vs_emlx.exs b/bench/qwen3_4b_emily_vs_emlx.exs index e6a8856..6bed405 100644 --- a/bench/qwen3_4b_emily_vs_emlx.exs +++ b/bench/qwen3_4b_emily_vs_emlx.exs @@ -46,7 +46,7 @@ Mix.install([ {:emily, path: Path.expand("..", __DIR__)}, {:exla, "~> 0.12"}, - {:emlx, "~> 0.3.1"}, + {:emlx, "~> 0.4.0"}, # Versions track Emily's own deps (Bumblebee 0.7 / Axon 0.8 / Nx 0.12); # keep them in sync with mix.exs so the standalone install resolves. {:bumblebee, "~> 0.7"},