Skip to content

Commit 3683751

Browse files
ausimianausimian
committed
Version 0.7.0
1 parent 1e7b38f commit 3683751

3 files changed

Lines changed: 113 additions & 110 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,118 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
77

88
<!-- %% CHANGELOG_ENTRIES %% -->
99

10+
## 0.7.0 - 2026-06-13
11+
12+
### Added
13+
14+
- **Native Expr compiler — on by default under
15+
`compiler: Emily.Compiler`.** Lowers a traced `Nx.Defn.Expr` to a
16+
flat IR once and replays the whole forward graph in a **single NIF
17+
call per invocation**, collapsing the per-op BEAM↔worker round-trips
18+
a step-evaluated decode loop would otherwise pay. Weights cross the
19+
NIF boundary once (captured by the compiled program) and are never
20+
re-serialised per call. It is the default, so a bare
21+
`compiler: Emily.Compiler` compiles native:
22+
23+
Nx.Defn.jit(&forward/1, compiler: Emily.Compiler).(input)
24+
25+
Coverage is the full Nx primitive set (with `Emily.Backend`'s
26+
dtype-coercion and op-composition semantics ported into the
27+
lowering), the fused `Emily.Fast.*` kernels (RMSNorm, LayerNorm,
28+
RoPE, scaled dot-product attention and its mask / sink / mask+sink
29+
variants), `Nx.Block.*` including the full `LinAlg` family
30+
(`cholesky` / `solve` / `qr` / `eigh` / `lu` / `svd` /
31+
`determinant`), `Nx.Random`, and the control flow `cond` /
32+
`defn while` (with the host loop driven entirely from the worker
33+
thread). Anything the IR can't lower yet routes through
34+
`Nx.Defn.Evaluator` under the default `native_fallback: :eval` (with
35+
a one-shot `[:emily, :compiler, :fallback]` telemetry event), so the
36+
native lane is safe as the default on any model. The default is read
37+
from `config :emily, :native` (defaulting to `true`), so
38+
`config :emily, native: false` opts every defn out of the native lane
39+
application-wide — e.g. on a memory-constrained host where the
40+
one-shot compile peak is too large; a per-call `native:` option
41+
always wins over the app-env default.
42+
43+
`native_fallback: :raise` fails instead — the conformance suites use
44+
this to prove a model lowers fully native.
45+
46+
End-to-end: DistilBERT (question answering with `Nx.Serving`), ViT,
47+
Whisper (`speech_to_text` end-to-end including the featurizer STFT,
48+
encoder/decoder, and autoregressive decode loop), and Bumblebee
49+
`Text.generation` (greedy *and* multinomial sampling) all compile
50+
fully native under `native_fallback: :raise`. Bumblebee generation
51+
on Qwen3-0.6B measures **~5× the evaluator's decode throughput**
52+
(~61 vs ~12 tok/s on an M-series Mac), with byte-identical
53+
completions. Native training drives Axon end-to-end — a LeNet CNN
54+
and a dense MLP train on real MNIST entirely through the single-NIF
55+
path (forward, categorical-cross-entropy, backward, Adam) to the
56+
same >97% / >96% accuracy as the evaluator.
57+
58+
- **`Emily.Compiler` — `:fuse` opt-in.** Adds `mx::compile` fusion on
59+
top of the replay, fusing elementwise runs (RMSNorm, softmax, SiLU
60+
gating, residual adds) the plain replay leaves as separate kernels.
61+
For a `defn while`, the loop body is fused under `mx::compile` and
62+
cached per stream so it cache-hits across iterations rather than
63+
recompiling per step. Enable on top of the native generation path:
64+
65+
Nx.Defn.jit(&forward/1,
66+
compiler: Emily.Compiler, native: true, fuse: true)
67+
68+
On Qwen3-0.6B this lifts greedy decode to **~5.4× the evaluator
69+
(~1.1× over the plain native lane)**, ~68 vs ~62 tok/s; in
70+
isolation on a decode-shaped transformer block, fusion measures
71+
~1.5–1.6× over the plain replay. Trade-off: `mx::compile`
72+
reassociates f32 to within a few ULP, so output is **not**
73+
bit-identical to the evaluator. Greedy argmax is robust to that
74+
empirically (Qwen3-0.6B token ids matched the evaluator exactly in
75+
our run), but the match is empirical, not guaranteed — a near-tie
76+
top-2 logit can flip a token. **Sampling strategies will diverge
77+
from the evaluator under fusion** even with a fixed seed.
78+
79+
- **`Emily.Generation` — a model-agnostic decode-loop driver.**
80+
JIT-compiles a caller-supplied shape-stable per-token forward
81+
(`fn token, offset, cache, params -> {logits, cache} end`) with the
82+
native single-NIF compiler and drives the autoregressive loop from
83+
Elixir — offset bookkeeping, KV-cache threading, stop conditions,
84+
next-token selection (greedy by default), and per-token streaming
85+
via `:on_token`. The forward runs fully native; the loop stays in
86+
Elixir, so token streaming and host-side control are preserved.
87+
Emily supplies only the mechanism — the model (forward + cache) is
88+
the caller's.
89+
90+
- `Emily.async_eval/1` (and `Emily.Native.async_eval/2`) schedule
91+
evaluation of one or more lazy graphs **without blocking on the
92+
GPU**, wrapping `mlx::core::async_eval`. The work is handed to the
93+
device's command queue and the call returns as soon as it is
94+
enqueued — not when it finishes. Lets a caller keep dispatching the
95+
next step's ops while the device computes the current one (e.g. an
96+
autoregressive decode loop), blocking only when a value is actually
97+
read back on the host via `to_binary/1` / `eval/1`. Pass every
98+
output of a step (logits plus all KV-cache buffers) in one call.
99+
100+
- `Emily.Native.fast_rope_int/8` — RoPE with an **integer**
101+
absolute-position `offset` (routing to MLX's int-offset `rope`
102+
overload), for incremental decode where the caller tracks position
103+
host-side. Complements the existing tensor-offset `fast_rope/8`.
104+
Note: feed the kernel the 4-D `{batch, heads, seq, head_dim}`
105+
layout — in 3-D, MLX 0.31 mis-rotates single-token (`seq == 1`)
106+
inputs.
107+
108+
### Fixed
109+
110+
- **Dilated window reductions (`window_dilations > 1`) returned wrong
111+
values.** `window_sum`/`window_max`/`window_min`/`window_product`
112+
with a dilated kernel silently produced garbage for windows past the
113+
first stride positions, on both the eager backend and the native
114+
compiler (they share the window-reduce core). A dilated kernel axis
115+
gets an `as_strided` stride > 1, so the sliding-window view aliases
116+
fewer physical elements than its logical size; MLX's strided-reduce
117+
fast path then read past the aliased buffer. The view is now
118+
materialised contiguously before the reduce when any dilation > 1
119+
(the common non-dilated pooling path is unchanged and stays
120+
copy-free).
121+
10122
## 0.6.1 - 2026-05-31
11123

12124
### Changed

‎RELEASE.md‎

Lines changed: 0 additions & 109 deletions
Original file line numberDiff line numberDiff line change
@@ -1,109 +0,0 @@
1-
### Added
2-
3-
- **Native Expr compiler — on by default under
4-
`compiler: Emily.Compiler`.** Lowers a traced `Nx.Defn.Expr` to a
5-
flat IR once and replays the whole forward graph in a **single NIF
6-
call per invocation**, collapsing the per-op BEAM↔worker round-trips
7-
a step-evaluated decode loop would otherwise pay. Weights cross the
8-
NIF boundary once (captured by the compiled program) and are never
9-
re-serialised per call. It is the default, so a bare
10-
`compiler: Emily.Compiler` compiles native:
11-
12-
Nx.Defn.jit(&forward/1, compiler: Emily.Compiler).(input)
13-
14-
Coverage is the full Nx primitive set (with `Emily.Backend`'s
15-
dtype-coercion and op-composition semantics ported into the
16-
lowering), the fused `Emily.Fast.*` kernels (RMSNorm, LayerNorm,
17-
RoPE, scaled dot-product attention and its mask / sink / mask+sink
18-
variants), `Nx.Block.*` including the full `LinAlg` family
19-
(`cholesky` / `solve` / `qr` / `eigh` / `lu` / `svd` /
20-
`determinant`), `Nx.Random`, and the control flow `cond` /
21-
`defn while` (with the host loop driven entirely from the worker
22-
thread). Anything the IR can't lower yet routes through
23-
`Nx.Defn.Evaluator` under the default `native_fallback: :eval` (with
24-
a one-shot `[:emily, :compiler, :fallback]` telemetry event), so the
25-
native lane is safe as the default on any model. The default is read
26-
from `config :emily, :native` (defaulting to `true`), so
27-
`config :emily, native: false` opts every defn out of the native lane
28-
application-wide — e.g. on a memory-constrained host where the
29-
one-shot compile peak is too large; a per-call `native:` option
30-
always wins over the app-env default.
31-
32-
`native_fallback: :raise` fails instead — the conformance suites use
33-
this to prove a model lowers fully native.
34-
35-
End-to-end: DistilBERT (question answering with `Nx.Serving`), ViT,
36-
Whisper (`speech_to_text` end-to-end including the featurizer STFT,
37-
encoder/decoder, and autoregressive decode loop), and Bumblebee
38-
`Text.generation` (greedy *and* multinomial sampling) all compile
39-
fully native under `native_fallback: :raise`. Bumblebee generation
40-
on Qwen3-0.6B measures **~5× the evaluator's decode throughput**
41-
(~61 vs ~12 tok/s on an M-series Mac), with byte-identical
42-
completions. Native training drives Axon end-to-end — a LeNet CNN
43-
and a dense MLP train on real MNIST entirely through the single-NIF
44-
path (forward, categorical-cross-entropy, backward, Adam) to the
45-
same >97% / >96% accuracy as the evaluator.
46-
47-
- **`Emily.Compiler` — `:fuse` opt-in.** Adds `mx::compile` fusion on
48-
top of the replay, fusing elementwise runs (RMSNorm, softmax, SiLU
49-
gating, residual adds) the plain replay leaves as separate kernels.
50-
For a `defn while`, the loop body is fused under `mx::compile` and
51-
cached per stream so it cache-hits across iterations rather than
52-
recompiling per step. Enable on top of the native generation path:
53-
54-
Nx.Defn.jit(&forward/1,
55-
compiler: Emily.Compiler, native: true, fuse: true)
56-
57-
On Qwen3-0.6B this lifts greedy decode to **~5.4× the evaluator
58-
(~1.1× over the plain native lane)**, ~68 vs ~62 tok/s; in
59-
isolation on a decode-shaped transformer block, fusion measures
60-
~1.5–1.6× over the plain replay. Trade-off: `mx::compile`
61-
reassociates f32 to within a few ULP, so output is **not**
62-
bit-identical to the evaluator. Greedy argmax is robust to that
63-
empirically (Qwen3-0.6B token ids matched the evaluator exactly in
64-
our run), but the match is empirical, not guaranteed — a near-tie
65-
top-2 logit can flip a token. **Sampling strategies will diverge
66-
from the evaluator under fusion** even with a fixed seed.
67-
68-
- **`Emily.Generation` — a model-agnostic decode-loop driver.**
69-
JIT-compiles a caller-supplied shape-stable per-token forward
70-
(`fn token, offset, cache, params -> {logits, cache} end`) with the
71-
native single-NIF compiler and drives the autoregressive loop from
72-
Elixir — offset bookkeeping, KV-cache threading, stop conditions,
73-
next-token selection (greedy by default), and per-token streaming
74-
via `:on_token`. The forward runs fully native; the loop stays in
75-
Elixir, so token streaming and host-side control are preserved.
76-
Emily supplies only the mechanism — the model (forward + cache) is
77-
the caller's.
78-
79-
- `Emily.async_eval/1` (and `Emily.Native.async_eval/2`) schedule
80-
evaluation of one or more lazy graphs **without blocking on the
81-
GPU**, wrapping `mlx::core::async_eval`. The work is handed to the
82-
device's command queue and the call returns as soon as it is
83-
enqueued — not when it finishes. Lets a caller keep dispatching the
84-
next step's ops while the device computes the current one (e.g. an
85-
autoregressive decode loop), blocking only when a value is actually
86-
read back on the host via `to_binary/1` / `eval/1`. Pass every
87-
output of a step (logits plus all KV-cache buffers) in one call.
88-
89-
- `Emily.Native.fast_rope_int/8` — RoPE with an **integer**
90-
absolute-position `offset` (routing to MLX's int-offset `rope`
91-
overload), for incremental decode where the caller tracks position
92-
host-side. Complements the existing tensor-offset `fast_rope/8`.
93-
Note: feed the kernel the 4-D `{batch, heads, seq, head_dim}`
94-
layout — in 3-D, MLX 0.31 mis-rotates single-token (`seq == 1`)
95-
inputs.
96-
97-
### Fixed
98-
99-
- **Dilated window reductions (`window_dilations > 1`) returned wrong
100-
values.** `window_sum`/`window_max`/`window_min`/`window_product`
101-
with a dilated kernel silently produced garbage for windows past the
102-
first stride positions, on both the eager backend and the native
103-
compiler (they share the window-reduce core). A dilated kernel axis
104-
gets an `as_strided` stride > 1, so the sliding-window view aliases
105-
fewer physical elements than its logical size; MLX's strided-reduce
106-
fast path then read past the aliased buffer. The view is now
107-
materialised contiguously before the reduce when any dilation > 1
108-
(the common non-dilated pooling path is unchanged and stays
109-
copy-free).

‎mix.exs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ defmodule Emily.MixProject do
22
use Mix.Project
33

44
@app :emily
5-
@version "0.6.1"
5+
@version "0.7.0"
66
@source_url "https://github.com/ausimian/emily"
77

88
# MLX pin. Drives the git tag the `:mlx_src` dep is cloned at (see

0 commit comments

Comments
 (0)