M5: Emily.Compiler — Nx.Defn.Compiler on Emily.Backend

Implements the Nx.Defn.Compiler behaviour as a thin wrapper over
Nx.Defn.Evaluator. The Evaluator already walks Nx.Defn.Expr in Elixir
and dispatches each op via Nx.Shared.list_impl!/1, which routes to
Emily.Backend whenever the operands carry it. Emily.Compiler validates
options, points __to_backend__/1 at Emily.Backend (honouring :device),
pins __partitions_options__/1 to a single partition (MLX kernel
dispatch isn't thread-safe), and delegates the walk.

The compile result is held in the closure returned by __compile__/4
rather than in an external cache. PLAN.md originally specified an ETS
cache keyed by {mfa, input_signature}; deviated after accounting for
the per-call ETS deep-copy cost on a Qwen3-sized expression tree —
copying the closure-captured plan via the closure environment is
free, copying it out of ETS is not. Callers that want reuse across
invocations use Nx.Defn.compile/3 (Bumblebee / Nx.Serving already do
this on warmup). PLAN.md updated to record the rationale.

mlx::core::compile wrapping is M6, deliberately not landed here.

Tests:
  - compiler_test.exs: callback contracts (__to_backend__ device
    routing, partition pinning, unknown-opt and max_concurrency>1
    rejection); op equivalence across elementwise / reduction / shape /
    linalg / container outputs; defn-only `while` (the construct
    Qwen3's KV-cache update relies on) and `cond`; Nx.Defn.compile/3
    closure reuse.
  - compiler_axon_test.exs: M5 exit criterion. 3-layer Axon MLP under
    Emily.Compiler matches Nx.Defn.Evaluator on the same backend
    within float tolerance, plus a multi-input compile-and-reuse case.

:axon added as explicit only:test dep — already transitive via
Bumblebee, but the Axon MLP test reaches for it directly and shouldn't
be hostage to a Bumblebee dep change.

Author: ausimian

PLAN.md

@@ -138,21 +138,40 @@ throughput number.
 
 ### M5 — `Emily.Compiler`: `Nx.Defn.Compiler` implementation
 
-- Walk `Nx.Defn.Expr` in Elixir, dispatching each node to `Emily.Backend`
-- Cache the walk result keyed by input signature in ETS
+- Walk `Nx.Defn.Expr` in Elixir, dispatching each node to `Emily.Backend`.
+  In practice this is what `Nx.Defn.Evaluator` already does — it dispatches
+  via `Nx.Shared.list_impl!/1` which finds whichever backend the operands
+  carry. `Emily.Compiler` validates options, points `__to_backend__/1` at
+  `Emily.Backend`, pins partitions to 1 (MLX kernel dispatch isn't
+  thread-safe), and delegates the walk.
+- Hold the walked plan in the closure returned by `__compile__/4`; the
+  closure *is* the cache. Callers that want reuse across invocations use
+  `Nx.Defn.compile/3` and hold the returned function — Bumblebee /
+  `Nx.Serving` already do this on warmup.
+  - *Earlier draft proposed an ETS cache keyed by `{mfa, input_signature}`;
+    rejected once we accounted for the per-call ETS deep-copy cost on a
+    Qwen3-sized expression tree. The closure-capture path avoids the copy
+    and matches the upstream Evaluator pattern.*
 - **Do not use `mlx::core::compile` yet.** Lazy eval at the Backend layer suffices.
 
 **Testing — Layer 3 (Compiler):**
 
-- **Equivalence tests**: every Backend test run wrapped in `defn` with
-  `compiler: Emily.Compiler`, assert identical output.
-- **Cache tests**: same call → hit; different shape → miss.
-- **Recompilation**: changed dtype → recompile; same shape+dtype → no
-  recompile.
+- **Equivalence tests**: a representative sample of ops (creation,
+  binary, reduction, shape, dot, container output) plus the `defn`-only
+  constructs `while` and `cond`; assert `compiler: Emily.Compiler` matches
+  raw Backend execution (and `Nx.Defn.Evaluator` for the `defn`-only
+  cases). The full Backend property suite isn't re-run per op — the
+  Backend already passes its own oracle suite, and the Compiler test
+  is structural ("did the walk reach the right backend with the right
+  args").
+- **Reuse**: a `Nx.Defn.compile/3` closure runs many inputs of the same
+  signature without re-walking the expression.
+- **Callback contracts**: `__to_backend__`, `__partitions_options__`,
+  unknown-option rejection, `:max_concurrency > 1` refusal.
 
 **Exit:** Axon MLPs forward with `compiler: Emily.Compiler`; results
-match Backend-only mode within float tolerance. (Training is out of
-scope for v1.)
+match `Nx.Defn.Evaluator` running on the same backend within float
+tolerance. (Training is out of scope for v1.)
 
 ### M6 — `mlx::core::compile` wrapping
 

RELEASE.md

@@ -2,6 +2,49 @@
 
 ## Added
 
+- M5 — `Emily.Compiler`, an `Nx.Defn.Compiler` implementation that runs
+  `defn` computations on `Emily.Backend`. Wraps `Nx.Defn.Evaluator` after
+  validating options and pinning the result backend; the Evaluator
+  already walks `Nx.Defn.Expr` in Elixir and dispatches each op via
+  `Nx.Shared.list_impl!/1`, which finds `Emily.Backend` whenever the
+  operands carry it.
+  - **`__to_backend__/1`** returns `{Emily.Backend, [device: …]}` so
+    `Nx.Defn.to_backend/1` (consulted by `Nx.Serving` and friends)
+    allocates inputs and outputs on Emily rather than the process default
+    backend. Honours `:device` opt; defaults to `:gpu`.
+  - **`__partitions_options__/1`** pins to a single partition. MLX's
+    Metal runtime is not safe for concurrent kernel dispatch from
+    multiple OS threads (the same constraint that forces
+    `max_cases: 1` in `test_helper.exs`); a multi-partition serving
+    would race the driver. `:max_concurrency` is accepted for
+    `Nx.Serving` API compatibility but values >1 raise.
+  - **No external compile cache.** `__compile__/4` returns a closure
+    that captures the walked plan; the closure *is* the cache. Callers
+    that want reuse across invocations use `Nx.Defn.compile/3` and hold
+    the returned function — Bumblebee / `Nx.Serving` already do this on
+    warmup. PLAN.md originally specified an ETS cache keyed by
+    `{mfa, input_signature}`; deliberately deviated after accounting
+    for the per-call ETS deep-copy cost on a Qwen3-sized expression
+    tree. PLAN.md updated to record the rationale.
+  - **No `mlx::core::compile` wrapping.** That is M6; lazy evaluation
+    at the Backend layer suffices for correctness.
+  - **`test/emily/compiler_test.exs`** — callback-contract tests
+    (`__to_backend__` device routing, partition pinning, unknown-option
+    rejection, `:max_concurrency > 1` refusal); op-equivalence tests
+    across elementwise / reduction / shape / linalg / container-output
+    paths; control-flow equivalence under `defn` for `while` (the
+    construct Qwen3's KV-cache update relies on) and `cond`; a
+    `Nx.Defn.compile/3` reuse test confirming the closure executes
+    repeatedly without re-walking.
+  - **`test/emily/compiler_axon_test.exs`** — the M5 exit criterion. A
+    3-layer Axon MLP forward pass under `Emily.Compiler` matches
+    `Nx.Defn.Evaluator` on the same backend within float tolerance,
+    plus a `Nx.Defn.compile/3` reuse case driving multiple inputs
+    through one walk.
+  - **`:axon`** added as an explicit `only: :test` dep — already
+    transitively available via Bumblebee, but the Axon MLP test reaches
+    for it directly and shouldn't be hostage to a Bumblebee dep change.
+
 - M0 scaffold: mix project, MLX 0.25.1 prebuilt fetch pipeline,
   Makefile wiring `fine` + MLX, `Emily.Native` NIF surface for tensor
   round-trip, application supervisor skeleton, smoke test suite.

lib/emily/compiler.ex

@@ -0,0 +1,105 @@
+defmodule Emily.Compiler do
+  @moduledoc """
+  `Nx.Defn.Compiler` implementation that runs `defn` computations on
+  `Emily.Backend`.
+
+  The compiler walks `Nx.Defn.Expr` in Elixir and dispatches each node
+  through the active backend — exactly what `Nx.Defn.Evaluator` already
+  does — with two adjustments specific to Emily:
+
+    * `c:__to_backend__/1` returns `{Emily.Backend, [device: …]}` so
+      `Nx.Defn.to_backend/1` (and the callers that consult it, including
+      `Nx.Serving`) allocate inputs and outputs on Emily rather than the
+      process-default backend.
+    * `c:__partitions_options__/1` always returns a single partition.
+      MLX's Metal runtime is not safe for concurrent kernel dispatch
+      from multiple OS threads (see `test/soak/backend_concurrency_test.exs`
+      for the SIGSEGV story); a multi-partition serving would race the
+      driver. `:max_concurrency` is accepted for API compatibility with
+      `Nx.Serving` but capped at 1.
+
+  ## Why this is so thin
+
+  M5 deliberately avoids two pieces of complexity:
+
+    1. **No external cache.** `__compile__/4` walks the expression once
+       and returns a closure that captures the walked plan; the closure
+       *is* the cache. Callers that want reuse across invocations use
+       `Nx.Defn.compile/3` and hold the returned function (Bumblebee /
+       `Nx.Serving` already do this on warmup). The PLAN.md note about
+       caching the walk in ETS was rejected once we accounted for the
+       per-call ETS deep-copy cost on a Qwen3-sized expression tree.
+
+    2. **No `mlx::core::compile` wrapping.** Lazy evaluation at the
+       Backend layer suffices for correctness; kernel-fusion via
+       `mlx::core::compile` is M6.
+
+  Concretely, `__jit__/5` and `__compile__/4` delegate to
+  `Nx.Defn.Evaluator` after option validation. The Evaluator dispatches
+  every op via `Nx.Shared.list_impl!/1`, which finds `Emily.Backend`
+  whenever the operands carry it — and `c:__to_backend__/1` ensures the
+  operands do.
+
+  ## Options
+
+    * `:device` — `:gpu` (default) or `:cpu`. Forwarded to `Emily.Backend`
+      via the `c:__to_backend__/1` callback.
+    * `:hooks`, `:debug_options`, `:garbage_collect` — passed through to
+      `Nx.Defn.Evaluator` unchanged. See its moduledoc.
+    * `:max_concurrency` — accepted for `Nx.Serving` compatibility, but
+      multi-partition serving is rejected because MLX kernel dispatch
+      isn't thread-safe. Pass `1` (the default) to silence.
+  """
+
+  @behaviour Nx.Defn.Compiler
+
+  alias Nx.Defn.Evaluator
+
+  @valid_opts [:device, :hooks, :debug_options, :garbage_collect, :max_concurrency]
+
+  @impl true
+  def __jit__(key, vars, fun, args_list, opts) do
+    opts = validate_opts!(opts)
+    Evaluator.__jit__(key, vars, fun, args_list, opts)
+  end
+
+  @impl true
+  def __compile__(key, vars, fun, opts) do
+    opts = validate_opts!(opts)
+    Evaluator.__compile__(key, vars, fun, opts)
+  end
+
+  @impl true
+  def __partitions_options__(opts) do
+    opts = validate_opts!(opts)
+
+    case Keyword.get(opts, :max_concurrency, 1) do
+      n when n in [nil, 1] ->
+        [opts]
+
+      n when is_integer(n) and n > 1 ->
+        raise ArgumentError,
+              "Emily.Compiler does not support :max_concurrency > 1 — MLX's Metal " <>
+                "runtime isn't safe for concurrent kernel dispatch from multiple OS " <>
+                "threads. Got: #{n}"
+    end
+  end
+
+  @impl true
+  def __to_backend__(opts) do
+    opts = validate_opts!(opts)
+    {Emily.Backend, [device: Keyword.get(opts, :device, :gpu)]}
+  end
+
+  defp validate_opts!(opts) do
+    case Enum.reject(Keyword.keys(opts), &(&1 in @valid_opts)) do
+      [] ->
+        opts
+
+      unknown ->
+        raise ArgumentError,
+              "Emily.Compiler received unknown option(s): #{inspect(unknown)}. " <>
+                "Valid options: #{inspect(@valid_opts)}"
+    end
+  end
+end

mix.exs

@@ -59,6 +59,11 @@ defmodule Emily.MixProject do
       {:bumblebee,
        github: "elixir-nx/bumblebee", ref: "273805e95507dc7866b958d90e0012a3abad1761", only: :test},
       {:tokenizers, "~> 0.5", only: :test},
+      # Axon is already pulled in transitively by Bumblebee, but the M5
+      # exit-criterion test (Axon MLP forward under `Emily.Compiler`)
+      # reaches for it directly — pin it explicitly so the test isn't
+      # hostage to a Bumblebee dep change.
+      {:axon, "~> 0.7", only: :test},
       {:stream_data, "~> 1.1", only: [:dev, :test]},
       {:credo, "~> 1.7", only: [:dev, :test], runtime: false},
       {:ex_doc, "~> 0.34", only: :docs, runtime: false}

test/emily/compiler_axon_test.exs

@@ -0,0 +1,85 @@
+defmodule Emily.CompilerAxonTest do
+  @moduledoc """
+  M5 exit-criterion test: an Axon MLP forward pass under
+  `Emily.Compiler` produces results matching `Nx.Defn.Evaluator` on the
+  same backend within float tolerance.
+
+  Both compilers execute through `Emily.Backend`; the comparison is
+  structural — does our compiler walk the expression and reach the same
+  ops with the same operands as the reference walker.
+  """
+
+  use ExUnit.Case, async: false
+
+  import Emily.BackendGenerators, only: [assert_close: 2]
+
+  setup do
+    prev = Nx.default_backend()
+    Nx.default_backend(Emily.Backend)
+    on_exit(fn -> Nx.default_backend(prev) end)
+    :ok
+  end
+
+  test "3-layer MLP forward pass matches Nx.Defn.Evaluator" do
+    model =
+      Axon.input("input", shape: {nil, 16})
+      |> Axon.dense(32, activation: :relu)
+      |> Axon.dense(16, activation: :relu)
+      |> Axon.dense(10)
+
+    {init_fn, predict_fn} = Axon.build(model)
+
+    template = Nx.template({1, 16}, :f32)
+    params = init_fn.(template, Axon.ModelState.empty())
+
+    # Deterministic input — we don't care about the values, only that
+    # both compilers see the same numbers.
+    input =
+      Nx.iota({4, 16}, type: :f32)
+      |> Nx.divide(64.0)
+      |> Nx.subtract(0.5)
+
+    eval =
+      Nx.Defn.jit_apply(predict_fn, [params, input], compiler: Nx.Defn.Evaluator)
+
+    emily =
+      Nx.Defn.jit_apply(predict_fn, [params, input], compiler: Emily.Compiler)
+
+    assert Nx.shape(emily) == {4, 10}
+    assert_close(emily, eval)
+  end
+
+  test "Nx.Defn.compile reuse — closure runs many inputs through one walk" do
+    model =
+      Axon.input("input", shape: {nil, 8})
+      |> Axon.dense(8, activation: :tanh)
+      |> Axon.dense(4)
+
+    {init_fn, predict_fn} = Axon.build(model)
+    template = Nx.template({1, 8}, :f32)
+    params = init_fn.(template, Axon.ModelState.empty())
+
+    compiled =
+      Nx.Defn.compile(
+        predict_fn,
+        [params, Nx.template({2, 8}, :f32)],
+        compiler: Emily.Compiler
+      )
+
+    inputs = [
+      Nx.iota({2, 8}, type: :f32),
+      Nx.broadcast(0.25, {2, 8}),
+      Nx.tensor(for _ <- 1..2, do: for(_ <- 1..8, do: 0.5))
+    ]
+
+    for input <- inputs do
+      out = compiled.(params, input)
+      assert Nx.shape(out) == {2, 4}
+
+      ref =
+        Nx.Defn.jit_apply(predict_fn, [params, input], compiler: Nx.Defn.Evaluator)
+
+      assert_close(out, ref)
+    end
+  end
+end