Merge pull request #198 from ausimian/docs/perf-emlx-and-version

docs: add EMLX baseline to perf section and fix report version

Author: ausimian

CHANGELOG.md

@@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 <!-- %% CHANGELOG_ENTRIES %% -->
 
+## 0.7.2 - 2026-06-13
+
+### Fixed
+
+- The README performance section now compares Emily against both
+  benchmark baselines — EXLA (host CPU) and EMLX (the older MLX-backed
+  Nx backend on the Metal GPU) — instead of EXLA alone, and its
+  rule-of-thumb figures (ViT-base, DistilBERT) are reconciled with the
+  current benchmark report.
+- The benchmark report's environment block now records the Emily
+  version the numbers were produced on (0.7.0) and drops a misleading
+  run timestamp.
+- The `MAINTAINING.md` release runbook is corrected: `mix publisho` is
+  no longer described as pushing (it only commits and tags), and the
+  obsolete manual draft-promotion step is dropped — `release-nif.yml`
+  now publishes the release automatically once the NIFs are built.
+
 ## 0.7.1 - 2026-06-13
 
 ### Fixed

MAINTAINING.md

@@ -50,11 +50,11 @@ default `aot`) through `config/config.exs` and stash the atom as
 ## Cutting a release
 
 Consumers verify each NIF tarball against the checksum pinned in
-`native_checksums.txt`, which ships in the hex package. Publishing is two
-commands (step 4): `mix emily.publish` regenerates that file from the
-freshly-built release artifacts, then `mix hex.publish` publishes — so
-there is nothing to update or commit by hand (the file is git-ignored and
-can't go stale).
+`native_checksums.txt`, which ships in the hex package. The publish step
+itself is two commands (steps 4–5): `mix emily.publish` regenerates that
+file from the built release artifacts, then `mix hex.publish` publishes —
+so there is nothing to update or commit by hand (the file is git-ignored
+and can't go stale).
 
 ### 1. Land changes on `main`
 
@@ -68,8 +68,16 @@ mix publisho patch   # or minor / major
 ```
 
 Bumps `@version`, rolls `RELEASE.md` into `CHANGELOG.md` under a
-dated `## <v>` heading, commits `Version <v>`, tags bare semver (no
-`v` prefix), and pushes both the commit and the tag.
+dated `## <v>` heading, commits `Version <v>`, and creates an annotated
+bare-semver tag (no `v` prefix). It does **not** push — that's the next
+step.
+
+### 3. Push the commit and the tag
+
+```sh
+git push              # the `Version <v>` commit
+git push origin <v>   # the tag — this is what fires the release workflow
+```
 
 The tag push fires `.github/workflows/release-nif.yml`, which fans
 out `{variant × target}`:
@@ -89,9 +97,46 @@ sidecar (informational — consumers verify against the pinned
 `https://github.com/ausimian/emily/releases/tag/<v>` — the URL the
 consumer's `compile.emily_nif` step fetches from.
 
-### 3. Verify end-to-end
+Once both cells finish, the workflow's `publish-release` job flips the
+release out of draft **automatically** (`gh release edit <v> --draft=false`),
+so its assets become public with no manual step. That job is gated to the
+tag-push path, so a manual `workflow_dispatch` rebuild leaves the release
+untouched — see *Rebuilding without retagging*.
+
+### 4. Pin the checksums
+
+Once the workflow has finished and the release is public:
 
-In a throwaway project:
+```sh
+mix emily.publish   # alias for `mix emily.checksums`
+```
+
+Downloads each tarball from the now-public release and records its SHA256
+into `native_checksums.txt`. It does **not** trust the `.sha256` sidecars —
+it hashes the bytes itself. If the release isn't public yet, the download
+404s and the task aborts before writing anything.
+
+### 5. Publish to Hex
+
+```sh
+mix hex.publish   # publishes the package (incl. native_checksums.txt) + docs
+```
+
+Packages `native_checksums.txt` — so it's covered by Hex's package hash in
+the consumer's `mix.lock`, a trust root independent of the mutable GitHub
+release — and publishes the package + docs.
+
+Steps 4 and 5 are two separate commands on purpose — they can't be folded
+into one alias. Mix only loads the Hex archive for the task *named on the
+command line*, so a `hex.publish` step chained inside an alias (whose CLI
+name is `emily.publish`) fails with `** (Mix) The task "hex.publish" could
+not be found`. Running `mix hex.publish` directly is what loads Hex.
+`hex.publish` keeps its `:docs` `preferred_env`, so it publishes docs as
+well as the package.
+
+### 6. Verify end-to-end (recommended)
+
+After publishing, in a throwaway project:
 
 ```sh
 mix new /tmp/emily-verify && cd /tmp/emily-verify
@@ -106,39 +151,9 @@ iex -S mix
 downloads the tarball, verifies, validates entries, extracts. A
 variant-mismatched consumer (`config :emily, variant: :jit`) should
 download the JIT tarball instead — worth spot-checking both lanes on
-the first release of a bump. (The pin only ever exists in the
-*published* package — `native_checksums.txt` is git-ignored and
-generated during `mix hex.publish` — so run this end-to-end verify
-against the published package, i.e. after step 4.)
-
-### 4. Promote the draft and publish
-
-Promote the release so its assets are public, then publish:
-
-```sh
-gh release edit <v> --repo ausimian/emily --draft=false   # assets go public
-mix emily.publish                                          # regenerate native_checksums.txt
-mix hex.publish                                            # publish package + docs
-```
-
-`mix emily.publish` runs `mix emily.checksums`: it downloads each tarball
-from the (now-public) release and records its SHA256 into
-`native_checksums.txt`. Then `mix hex.publish` packages that file and
-publishes the package + docs.
-
-These are two separate commands on purpose — they can't be folded into one
-alias. Mix only loads the Hex archive for the task *named on the command
-line*, so a `hex.publish` step chained inside an alias (whose CLI name is
-`emily.publish`) fails with `** (Mix) The task "hex.publish" could not be
-found`. Running `mix hex.publish` directly is what loads Hex. `hex.publish`
-keeps its `:docs` `preferred_env`, so it publishes docs as well as the
-package.
-
-So the consumer verifies downloads against a trust root that lives in the
-immutable Hex package, not the mutable GitHub release — with no file to
-maintain and nothing to commit. The file is git-ignored and regenerated on
-every publish, so it can't go stale. If the draft isn't public yet,
-`mix emily.checksums` 404s and aborts before you publish.
+the first release of a bump. The checksum pin only exists in the
+*published* package, so this must run against the published package,
+i.e. after step 5.
 
 ### Rebuilding without retagging
 

README.md

@@ -273,9 +273,15 @@ KV-cache threading, stop conditions, and per-token streaming.
 ## Performance
 
 Emily targets **GPU-friendly model inference on Apple Silicon**. The
-[Emily-vs-EXLA benchmark][bench-report] compares Emily (MLX, Metal GPU)
-against EXLA — which on macOS arm64 ships no GPU client and runs on the
-**CPU**. So the practical choice most Elixir-on-Apple-Silicon users
+[benchmark][bench-report] compares Emily (MLX, Metal GPU) against two
+baselines: **EXLA** — which on macOS arm64 ships no GPU client and runs
+on the **CPU** — and **EMLX**, the older MLX-backed Nx backend, which
+like Emily runs on the **Metal GPU**. EXLA answers the cross-hardware
+question (is the GPU faster than XLA on the CPU here?); EMLX answers the
+same-hardware one (is Emily's compiler/runtime faster than the older
+MLX-backed Nx stack?).
+
+Against EXLA, the practical choice most Elixir-on-Apple-Silicon users
 face is GPU-via-Emily vs CPU-via-EXLA, and the two have opposite cost
 structures: the GPU has a higher fixed per-op latency floor
 (~160–280 µs — a BEAM↔worker hop, a Metal command-buffer commit, and a
@@ -286,9 +292,9 @@ tensor size**, not model kind:
 | Workload (M4 Pro, f32)                   | Best Emily lane vs EXLA-CPU |
 | ---------------------------------------- | --------------------------- |
 | Large matmul (2048²)                     | **5.0× faster**             |
-| ViT-base image classification            | **2.2× faster**             |
+| ViT-base image classification            | **2.35× faster**            |
 | Qwen3-0.6B greedy decode                 | **1.67× faster** (`fuse`)   |
-| DistilBERT QA (one encoder forward)      | ~parity (1.06×)             |
+| DistilBERT QA (one encoder forward)      | **1.27× faster**            |
 | Whisper-tiny transcription               | **11× slower**              |
 | Elementwise / matmul ≤ ~512 per dim      | up to ~2.3× slower          |
 
@@ -302,9 +308,18 @@ unused. Notably, every model in the benchmark lowered **fully native
 with zero fallbacks**, so these gaps are kernel/dispatch efficiency,
 not coverage holes.
 
+**Versus EMLX (GPU-vs-GPU).** The same benchmark runs an `emlx` lane —
+the older MLX-backed Nx backend, also on the Metal GPU — so the
+comparison isn't only against the CPU. Here Emily's *compiler* is the
+differentiator: eager Emily is roughly EMLX-like, but native/fuse pull
+far ahead — **2.72× faster on DistilBERT QA**, **5.82× faster on
+Qwen3-0.6B decode**, and ~3.2× faster on the Qwen3-4B addendum. (EMLX
+did not complete the ViT-base or Whisper-tiny tiers in this harness.)
+
 For decode, use the native compiler rather than the eager backend:
-eager Qwen3 decode is 3.5× *slower* than EXLA where native is 1.67×
-faster — a 5.8× swing that is purely the per-op dispatch floor, paid
+eager Qwen3 decode is 3.2× *slower* than EXLA, while native is 1.5×
+faster and fuse 1.67× — a 5.3× throughput swing from eager to fuse
+(12.51 → 66.42 tok/s) that is purely the per-op dispatch floor, paid
 once per tiny op across thousands of decode steps.
 
 Re-run the benchmark with `elixir bench/emily_vs_exla.exs`; the full

bench/emily_vs_exla_report.md

@@ -30,13 +30,13 @@ comparison is therefore Qwen3-0.6B in the main suite.
 
 ## Environment
 
-Fresh run: 2026-06-13 12:24, Apple M4 Pro MacBook Pro with 24 GB RAM.
+Run on an Apple M4 Pro MacBook Pro with 24 GB RAM.
 
 | Component | Version / backend |
 | --- | --- |
 | Elixir / OTP | 1.19.5 / 28 |
 | Nx | 0.12.1 |
-| Emily | 0.6.1 local checkout |
+| Emily | 0.7.0 local checkout |
 | EMLX | 0.3.1, Metal GPU |
 | EXLA | 0.12.0, host CPU client |
 

mix.exs

@@ -2,7 +2,7 @@ defmodule Emily.MixProject do
   use Mix.Project
 
   @app :emily
-  @version "0.7.1"
+  @version "0.7.2"
   @source_url "https://github.com/ausimian/emily"
 
   # MLX pin. Drives the git tag the `:mlx_src` dep is cloned at (see