M7: Bumblebee conformance breadth — ViT + Whisper - #14
Merged
Conversation
DistilBERT and Qwen3 conformance suites each carry an identical setup_all block (swap to Emily.Backend, restore on exit) and an identical assert_all_close/3. Before adding ViT + Whisper suites (which would be a third and fourth copy), lift both into test/support/conformance_helper.ex. Net change ~zero LOC; keeps future suites terse.
Ports Bumblebee.Vision.VitTest verbatim — three tiny-random
architectures (:base, :for_image_classification,
:for_masked_image_modeling), same synthetic pixel input, same
expected slices. First conformance suite to exercise the conv
fallback path in anger.
Surfaced a latent bug in Emily.Backend.via_binary/via_binary_tuple:
the helpers transferred input tensors to BinaryBackend but left the
global default backend as-is. When a fallback op (Nx.conv in this
case) constructs a scalar tensor internally — e.g. the zero-pad
scalar via `Nx.pad(t, 0, ...)` — that scalar landed on
Emily.Backend whenever the conformance setup_all had swapped the
global default. The resulting mixed-backend operand list crashed
inside BinaryBackend's op with a FunctionClauseError on
to_binary/1 (the Emily-backed scalar's `data` doesn't match the
`%{state: data}` pattern BinaryBackend uses for its own tensors).
Fix: pin the default backend to Nx.BinaryBackend for the duration
of the fun call in both via_binary and via_binary_tuple. Never
surfaced before because test/emily/backend_fallbacks_test.exs
doesn't install Emily as the global default — it constructs
operands with an explicit `backend: Emily.Backend` — and every
earlier Bumblebee suite (DistilBERT, Qwen3) had already moved its
hot-path ops off the fallback by M4.
Full-size ViT-Base forward pass on Emily.Backend, pinned against a deterministic synthetic pixel input. Structure mirrors qwen3_full_test.exs: @moduletag :vit_full, excluded even from --only conformance because the checkpoint is ~330 MB. Run with `mix test --only vit_full`. Uses a constant-gray pixel tensor rather than a checked-in JPEG fixture so the repo stays free of binary test assets and the featurizer preprocessing path doesn't enter the assertion surface — the intent is to catch numerical drift on real-size weight tensors, not verify image preprocessing. Pinned values: leading 5 logits + argmax index 763 (ImageNet class "revolver") on the gray input. Also update test/test_helper.exs to exclude :vit_full and :whisper_full (landing next) from the default test run; expand the comment to document each heavyweight tag and its cache footprint.
Ports Bumblebee.Audio.WhisperTest verbatim — two tiny-random
architectures (:base, :for_conditional_generation), same synthetic
mel features (Nx.sin(Nx.iota({1, 60, 80}))), same decoder inputs,
same expected slices.
First conformance suite to exercise encoder-decoder cross-attention
on Emily.Backend, and the first with strided 1-D conv in the
encoder frontend. Both pass first try against Bumblebee's
HF-Transformers-produced reference slices, which means the
cross-attention KV-cache update and the two stacked 1-D convs are
numerically consistent with PyTorch — despite conv still routing
through the BinaryBackend fallback (M8).
Full-size Whisper-tiny forward pass on Emily.Backend, pinned
against a deterministic synthetic 30 s mel window (sin(iota({1,
3000, 80}) * 0.01)) and a short special-token decoder prompt.
Asserts the leading 3×3 logits slice at 1e-3 tolerance plus the
decoder-last-step argmax (== 50257, <|endoftext|>, which is what
the synthetic input collapses to — that's fine for a pin; the
assertion is "same backend + same weights + same input reproduces
the same token", not "the model says something interesting").
Runtime is ~7 minutes on Apple Silicon because Emily.Backend.conv
still routes through Nx.BinaryBackend (deferred to M8). Whisper
encoder has two 1-D convs (kernel=3, out=384) over 3000 time-steps,
~940M multiply-adds executed as BEAM loops. The full test is
:whisper_full tagged, excluded from --only conformance, and opt-in
only; M8 wiring Backend.conv to Native.conv_general will bring this
to sub-second.
Also picks up formatter fixups on lib/emily/backend.ex (the
via_binary assignment from the previous commit) and trailing blank
lines on distilbert_test.exs / qwen3_test.exs.
- Renumber milestones: M7 = conformance breadth (this), M8 = native conv (lift Backend.conv off the BinaryBackend fallback), M9 = 1.0 release (was M7). - M7 section documents the two new models and the rationale (vision, encoder-decoder, conv coverage) and records that MoE / Mixtral is deferred pending upstream Bumblebee support. - RELEASE.md accumulates the added test suites, the shared ConformanceHelper, the test_helper exclude-list update, and the via_binary default-backend fix surfaced by ViT tiny-random.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Emily.ConformanceHelper(test/support/conformance_helper.ex) —setup_allbackend swap +assert_all_close/3— and migrates DistilBERT, Qwen3, ViT, Whisper onto it. Net change ~zero LOC today, terser for future suites.Emily.Backend.via_binary: pin the default backend toNx.BinaryBackendfor the duration of the fallbackfun. Surfaced by ViT tiny-random exercisingconv:Nx.convbuilds a zero-pad scalar internally, which landed onEmily.Backendwhenever the conformancesetup_allwas active, then crashed inside BinaryBackend on the mixed-backend operand list. Never surfaced before becausebackend_fallbacks_test.exsdoesn't install Emily as the global default.PLAN.mdrenumbered: M7 = conformance breadth (this), M8 = native conv (liftBackend.convoff BinaryBackend), M9 = 1.0 release (was M7). MoE / Mixtral tracked as deferred pending upstream Bumblebee support.test_helper.exsexclude list extended with:vit_full,:whisper_full(both opt-in viamix test --only <tag>).Test plan
mix test— 183 tests, 56 properties, 1 doctest, 0 failuresmix test --only conformance— 16 tests, 0 failures (DistilBERT + Qwen3 + ViT + Whisper tiny)mix test --only vit_full— 1 test, 0 failures (~21 s; pins argmax == 763 + leading-5 logits slice)mix test --only whisper_full— 1 test, 0 failures (~7 min; pinned values verified on re-run)mix format --check-formattedcleanmix credo --strictcleanmix test --only qwen3_full— not re-run this session; the only touchpoint is the via_binary fix, which tightens a correctness case Qwen3 doesn't hit (Qwen3's hot path went native in M4). Worth a manual pass before merging if you want to be belt-and-braces.Notes
Whisper full runs ~7 minutes on Apple Silicon because
Emily.Backend.convstill routes throughNx.BinaryBackend— ~940M multiply-adds through BEAM loops across two encoder 1-D convs. M8 (wiringBackend.convto the existingNative.conv_general) brings this to sub-second; intentionally out of scope for this MR to keep it focused on conformance.