Profile Emily against the EXLA backend

[ausimian]

Summary

Benchmark and profile Emily (the Nx backend for Apple MLX) against the existing EXLA backend to understand where we win, where we lose, and where to focus optimization effort.

Both are Nx.Backend implementations, so the same Nx programs can run on either by swapping the default backend — making a like-for-like comparison straightforward.

Goals

• Establish a baseline performance comparison between Emily.Backend and EXLA.Backend on Apple Silicon
• Identify operations / workloads where Emily is significantly slower than EXLA so they can be prioritized
• Capture both compiled (defn / Emily.Compiler) and eager-mode numbers

Suggested workloads

• Core tensor ops: elementwise, reductions, matmul/dot across a range of shapes & dtypes
• Nx.Defn compiled functions (exercise Emily.Compiler vs EXLA JIT)
• Convolutions (conv_general) at typical CNN sizes
• Quantized matmul paths (Emily's Emily.Quantization vs EXLA equivalents, where comparable)
• An end-to-end model forward pass via Bumblebee (e.g. one of the notebook models — DistilBERT QA, Qwen3, embeddings)

Method

• Use benchee for wall-clock timing (warm up to exclude first-call JIT/compile cost; also report cold-start separately)
• Pin MLX :variant (:aot vs :jit) and note which was measured
• Record device, OTP/Elixir versions, MLX version (@mlx_version), and EXLA/XLA version
• Watch memory too (Emily.Memory telemetry) — not just latency
• Land the benchmark scripts in the repo (e.g. bench/) so results are reproducible

Notes

• EXLA is not currently a dependency; it would be added as a only: [:dev, :test] dep purely for benchmarking.
• Emily is Apple-Silicon / macOS only; EXLA on the same host runs CPU (or Metal via XLA if available) — note the comparison target explicitly so numbers aren't misread.