Skip to content

Commit 165148b

Browse files
committed
docs: add EMLX baseline to perf section and fix report version
Reframe the README Performance section around both benchmark baselines: EXLA (host CPU) and EMLX (the older MLX-backed Nx backend on the Metal GPU), and add a dedicated EMLX comparison. EMLX previously appeared only in Acknowledgements. Reconcile the rule-of-thumb table and decode prose with the report's current numbers: ViT 2.35x, DistilBERT 1.27x, and the eager->fuse decode swing. In the benchmark report, correct the Emily version to 0.7.0 and drop the run timestamp, since the checkout tag is the relevant provenance.
1 parent 9f74cf8 commit 165148b

3 files changed

Lines changed: 34 additions & 9 deletions

File tree

README.md

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -273,9 +273,15 @@ KV-cache threading, stop conditions, and per-token streaming.
273273
## Performance
274274

275275
Emily targets **GPU-friendly model inference on Apple Silicon**. The
276-
[Emily-vs-EXLA benchmark][bench-report] compares Emily (MLX, Metal GPU)
277-
against EXLA — which on macOS arm64 ships no GPU client and runs on the
278-
**CPU**. So the practical choice most Elixir-on-Apple-Silicon users
276+
[benchmark][bench-report] compares Emily (MLX, Metal GPU) against two
277+
baselines: **EXLA** — which on macOS arm64 ships no GPU client and runs
278+
on the **CPU** — and **EMLX**, the older MLX-backed Nx backend, which
279+
like Emily runs on the **Metal GPU**. EXLA answers the cross-hardware
280+
question (is the GPU faster than XLA on the CPU here?); EMLX answers the
281+
same-hardware one (is Emily's compiler/runtime faster than the older
282+
MLX-backed Nx stack?).
283+
284+
Against EXLA, the practical choice most Elixir-on-Apple-Silicon users
279285
face is GPU-via-Emily vs CPU-via-EXLA, and the two have opposite cost
280286
structures: the GPU has a higher fixed per-op latency floor
281287
(~160–280 µs — a BEAM↔worker hop, a Metal command-buffer commit, and a
@@ -286,9 +292,9 @@ tensor size**, not model kind:
286292
| Workload (M4 Pro, f32) | Best Emily lane vs EXLA-CPU |
287293
| ---------------------------------------- | --------------------------- |
288294
| Large matmul (2048²) | **5.0× faster** |
289-
| ViT-base image classification | **2.2× faster** |
295+
| ViT-base image classification | **2.35× faster** |
290296
| Qwen3-0.6B greedy decode | **1.67× faster** (`fuse`) |
291-
| DistilBERT QA (one encoder forward) | ~parity (1.06×) |
297+
| DistilBERT QA (one encoder forward) | **1.27× faster** |
292298
| Whisper-tiny transcription | **11× slower** |
293299
| Elementwise / matmul ≤ ~512 per dim | up to ~2.3× slower |
294300

@@ -302,9 +308,18 @@ unused. Notably, every model in the benchmark lowered **fully native
302308
with zero fallbacks**, so these gaps are kernel/dispatch efficiency,
303309
not coverage holes.
304310

311+
**Versus EMLX (GPU-vs-GPU).** The same benchmark runs an `emlx` lane —
312+
the older MLX-backed Nx backend, also on the Metal GPU — so the
313+
comparison isn't only against the CPU. Here Emily's *compiler* is the
314+
differentiator: eager Emily is roughly EMLX-like, but native/fuse pull
315+
far ahead — **2.72× faster on DistilBERT QA**, **5.82× faster on
316+
Qwen3-0.6B decode**, and ~3.2× faster on the Qwen3-4B addendum. (EMLX
317+
did not complete the ViT-base or Whisper-tiny tiers in this harness.)
318+
305319
For decode, use the native compiler rather than the eager backend:
306-
eager Qwen3 decode is 3.5× *slower* than EXLA where native is 1.67×
307-
faster — a 5.8× swing that is purely the per-op dispatch floor, paid
320+
eager Qwen3 decode is 3.2× *slower* than EXLA, while native is 1.5×
321+
faster and fuse 1.67× — a 5.3× throughput swing from eager to fuse
322+
(12.51 → 66.42 tok/s) that is purely the per-op dispatch floor, paid
308323
once per tiny op across thousands of decode steps.
309324

310325
Re-run the benchmark with `elixir bench/emily_vs_exla.exs`; the full

RELEASE.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
### Fixed
2+
3+
- The README performance section now compares Emily against both
4+
benchmark baselines — EXLA (host CPU) and EMLX (the older MLX-backed
5+
Nx backend on the Metal GPU) — instead of EXLA alone, and its
6+
rule-of-thumb figures (ViT-base, DistilBERT) are reconciled with the
7+
current benchmark report.
8+
- The benchmark report's environment block now records the Emily
9+
version the numbers were produced on (0.7.0) and drops a misleading
10+
run timestamp.

bench/emily_vs_exla_report.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,13 +30,13 @@ comparison is therefore Qwen3-0.6B in the main suite.
3030

3131
## Environment
3232

33-
Fresh run: 2026-06-13 12:24, Apple M4 Pro MacBook Pro with 24 GB RAM.
33+
Run on an Apple M4 Pro MacBook Pro with 24 GB RAM.
3434

3535
| Component | Version / backend |
3636
| --- | --- |
3737
| Elixir / OTP | 1.19.5 / 28 |
3838
| Nx | 0.12.1 |
39-
| Emily | 0.6.1 local checkout |
39+
| Emily | 0.7.0 local checkout |
4040
| EMLX | 0.3.1, Metal GPU |
4141
| EXLA | 0.12.0, host CPU client |
4242

0 commit comments

Comments
 (0)