Context
CM13's test/emily/conformance/generation_native_test.exs gates native Bumblebee.Text.generation (greedy + multinomial sampling) against the evaluator on a tiny-random Qwen3 causal LM (default :conformance), driven through build_generate on in-vocab input_ids (no tokenizer, to dodge the OOB-gather pitfall of pairing a tiny model with a full-vocab tokenizer).
The original plan called for a real-model opt-in tier too ("gemma/qwen3 full as opt-in"), which wasn't added. bench/qwen3_tokens_per_sec.exs already loads the real Qwen3-0.6B and shows native and evaluator decode produce byte-identical completions — so promoting that into a gated conformance assertion is cheap and worthwhile.
Ask
Add a @moduletag :generation_full test (opt-in, not default CI — mirrors the existing :qwen3_full / :*_full discipline) that:
- loads the real
Qwen/Qwen3-0.6B (model + real tokenizer + generation config),
- runs
Bumblebee.Text.generation end-to-end through the serving (real tokenization → generate → detokenization) under compiler: Emily.Compiler, native: true, native_fallback: :raise,
- asserts the generated token ids / text are identical to the evaluator path (greedy; optionally multinomial with a fixed seed).
Run via mix test --only generation_full.
Value
The tiny-random model can't catch issues that only show up with a real vocab, real shapes, longer sequences, or the real generation config (e.g. real EOS handling, larger KV windows). This is the end-to-end "it actually generates correct text natively" gate, complementing the fast tiny-random gate on default CI.
Where
test/emily/conformance/ — likely extend generation_native_test.exs with a :generation_full describe, or a sibling file.
Follow-up to the CM13 native-generation work (PRs #156–#162).
Context
CM13's
test/emily/conformance/generation_native_test.exsgates nativeBumblebee.Text.generation(greedy + multinomial sampling) against the evaluator on a tiny-random Qwen3 causal LM (default:conformance), driven throughbuild_generateon in-vocabinput_ids(no tokenizer, to dodge the OOB-gather pitfall of pairing a tiny model with a full-vocab tokenizer).The original plan called for a real-model opt-in tier too ("gemma/qwen3 full as opt-in"), which wasn't added.
bench/qwen3_tokens_per_sec.exsalready loads the real Qwen3-0.6B and shows native and evaluator decode produce byte-identical completions — so promoting that into a gated conformance assertion is cheap and worthwhile.Ask
Add a
@moduletag :generation_fulltest (opt-in, not default CI — mirrors the existing:qwen3_full/:*_fulldiscipline) that:Qwen/Qwen3-0.6B(model + real tokenizer + generation config),Bumblebee.Text.generationend-to-end through the serving (real tokenization → generate → detokenization) undercompiler: Emily.Compiler, native: true, native_fallback: :raise,Run via
mix test --only generation_full.Value
The tiny-random model can't catch issues that only show up with a real vocab, real shapes, longer sequences, or the real generation config (e.g. real EOS handling, larger KV windows). This is the end-to-end "it actually generates correct text natively" gate, complementing the fast tiny-random gate on default CI.
Where
test/emily/conformance/— likely extendgeneration_native_test.exswith a:generation_fulldescribe, or a sibling file.Follow-up to the CM13 native-generation work (PRs #156–#162).