Skip to content

Commit 758700a

Browse files
committed
fix(gemma4): temperature/top-p/repeat-penalty sampling — kills degen loops
v0.1.1 served Gemma 4 26B-A4B MoE in greedy-only mode: temperature, top_p, and repeat_penalty from the request were all dropped on the backend side ("_temperature: f32" — silently underscored). On aggressively quantized builds (gemma-4-26b-a4b-mlx-3bit / MXFP4) greedy decoding catastrophically locks onto repeated-token attractors once quantization noise nudges argmax off the correct path — the canonical reproducer was a Korean prompt drifting into Turkish and then looping "karın-karın-karın-..." until max_tokens. This patch lands the standard HF / llama.cpp sampling pipeline behind a temperature/repeat_penalty guard so the existing GPU-pipelined greedy path stays bit-identical when the request is greedy (temperature == 0 AND repeat_penalty == 1.0). ## What lands * `lumen-core::sampling` — new module (backend-agnostic): - `SamplingConfig` (temperature, top_p, repeat_penalty, repeat_penalty_last_n, seed) + `is_greedy()` helper. - `apply_repeat_penalty` (HF-style: divide positive, multiply negative for the last-N tokens). - `softmax_inplace` with max-subtraction stability + uniform fallback on `-inf` to avoid NaNs. - `sample_top_p` (O(V log V) sort + cumulative cutoff, always keeps at least one token). - `Xorshift64` deterministic PRNG (no `rand` crate dep). - 7 unit tests, all passing. Pure `&mut [f32]` API so the Candle Qwen/Gemma/GemmaGguf legacy paths (currently greedy too) can adopt it in a follow-up without pulling in mlx-rs. * `lumen-mlx::gemma4_sampling` — thin MLX bridge re-exports `lumen_core::sampling::*` and adds `last_logits_to_cpu_f32` / `sample_next_token` to pull the last-position logits off the GPU into a CPU buffer before delegating to the shared sampler. * `lumen-mlx::gemma4_moe::GenerateConfig` — new `sampling: Option<SamplingConfig>` field. When `Some(non_greedy)`, `generate_with_cache` takes a CPU-sampled branch (no async pipelining, but the ~1-2 ms / step CPU cost is bounded vs the ~30 ms / step GPU step time, so net impact < 5%). MTP + lookup-spec paths remain greedy-only — sampling drafted decoding is a Phase 2 follow-up. * `lumen-mlx::gemma4_backend` — new private `build_sampling_config` helper reads `REPEAT_PENALTY` / `LUMEN_REPEAT_LAST_N` / `LUMEN_SAMPLE_SEED` env at chat-call time and combines with request-supplied temperature/top_p. Returns `None` when greedy so callers route to the fast path. - `chat()`, `chat_streaming()`, `chat_with_prefix_cache()`, `generate()` now take `top_p: f32` and use sampling when appropriate. - `chat_streaming()` gets a dedicated sampled decode loop that mirrors the existing greedy loop's prefill setup but replaces `argmax_last_token_lazy` with `sample_next_token`. * `lumen-mlx::lib::MlxBackend::{chat, chat_streaming, generate}` — signature extended with `top_p: f32`; threaded through to the Gemma 4 backend. Qwen35Family arm drops it (its own sampling lives in `lumen-server::engine.rs` SeqState path with REPEAT_PENALTY env already wired). * `lumen-server::engine.rs` ModelBackend trait + chat / chat_streaming call sites — `top_p` threaded through. Candle paths (Qwen, Gemma, GemmaGguf, Qwen35Moe) silently ignore via `let _ = top_p;` — follow-up PR will plumb sampling for those (lumen-model already has a CPU sampler in `sampling.rs`; goal is to move it to lumen-core and share with mlx-native). * `lumen-server::types::AnthropicRequest` — gains `top_p` field with the same `default_top_p()` default as OpenAI chat. /v1/messages callers now honor top_p. ## Verification * `cargo test -p lumen-core --lib sampling` — 7/7 passing. * Release build clean on default features (mlx-native + qwen3_5_moe + turboquant-gpu). * End-to-end smoke against `gemma-4-26b-a4b-mlx-3bit` with REPEAT_PENALTY=1.1, temperature=0.7, top_p=0.9 — Korean prompt "넌 누구야?" produces coherent Korean answer + EOS-terminates normally. v0.1.1's Turkish-drift-then-"karın-" loop fully resolved. ## Default behavior preserved When the operator hasn't set REPEAT_PENALTY (default 1.0) and the client doesn't send a temperature (or sends temperature=0), `is_greedy()` returns true and decode takes the existing GPU- pipelined argmax path — bit-identical to v0.1.1. Sampling activates only on opt-in via REPEAT_PENALTY env (operator-set in the SERVER card) OR temperature > 0 (default for OpenAI / Anthropic clients, including Moltis). ## Knobs * `REPEAT_PENALTY` — float, default 1.0 (off). 1.1-1.15 recommended for 3-bit MXFP4 builds. Set via SERVER card "repeat_penalty" field in the desktop app. * `LUMEN_REPEAT_LAST_N` — usize, default 64. Sliding window the penalty applies to. * `LUMEN_SAMPLE_SEED` — u64, default = wall-clock nanos. Deterministic when explicitly set. ## Versions * crates/lumen-app/Cargo.toml 0.1.1 → 0.1.2 * crates/lumen-app/tauri.conf.json 0.1.1 → 0.1.2 * crates/lumen-app/frontend/package.json 0.1.1 → 0.1.2
1 parent 58795c4 commit 758700a

17 files changed

Lines changed: 634 additions & 27 deletions

Cargo.lock

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

crates/lumen-app/Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[package]
22
name = "lumen-app"
3-
version = "0.1.1"
3+
version = "0.1.2"
44
edition.workspace = true
55
rust-version.workspace = true
66
license.workspace = true

crates/lumen-app/frontend/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"name": "lumen-app-frontend",
33
"private": true,
4-
"version": "0.1.1",
4+
"version": "0.1.2",
55
"type": "module",
66
"scripts": {
77
"dev": "vite --port 5173 --strictPort",

crates/lumen-app/tauri.conf.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"$schema": "https://schema.tauri.app/config/2",
33
"productName": "Lumen",
4-
"version": "0.1.1",
4+
"version": "0.1.2",
55
"identifier": "ai.lumen.app",
66
"build": {
77
"frontendDist": "frontend/dist",

crates/lumen-core/src/lib.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,4 @@ pub mod config;
44
pub mod lloyd_max;
55
pub mod qjl;
66
pub mod rotation;
7+
pub mod sampling;

crates/lumen-core/src/sampling.rs

Lines changed: 300 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,300 @@
1+
//! Backend-agnostic token sampling.
2+
//!
3+
//! Implements the standard HF / llama.cpp-style sampling pipeline:
4+
//!
5+
//! 1. Repeat penalty over a sliding window of recently emitted tokens
6+
//! (divide positive logits, multiply negative — pushes mass off
7+
//! self-reinforcing attractors that cause infinite repetition on
8+
//! aggressively quantized models).
9+
//! 2. Temperature scaling (`logits /= T`).
10+
//! 3. Softmax with the max-subtraction stability trick.
11+
//! 4. Top-p nucleus filtering.
12+
//! 5. Multinomial draw via a deterministic xorshift64* PRNG.
13+
//!
14+
//! The whole pipeline operates on `&mut [f32]` so it can be shared
15+
//! across backends: the GPU runtime (mlx, candle) is responsible for
16+
//! computing per-step logits and pulling the last-position vector into
17+
//! a CPU buffer; from there `sample_from_logits` returns the next
18+
//! token id.
19+
20+
/// Tuning knobs for one sampling step. `is_greedy()` returns true when
21+
/// the config collapses to argmax (temperature 0 AND no repeat
22+
/// penalty), letting callers skip the CPU pipeline entirely.
23+
#[derive(Debug, Clone, Copy)]
24+
pub struct SamplingConfig {
25+
/// Softmax temperature. `<= 0` collapses to argmax (greedy).
26+
pub temperature: f32,
27+
/// Nucleus sampling cutoff in `(0, 1]`. `>= 1.0` disables top-p
28+
/// (full distribution).
29+
pub top_p: f32,
30+
/// HF / llama.cpp-style repeat penalty applied to the last
31+
/// `repeat_penalty_last_n` tokens. `1.0` = no penalty.
32+
pub repeat_penalty: f32,
33+
/// Sliding-window length the penalty applies to. `0` disables.
34+
pub repeat_penalty_last_n: usize,
35+
/// PRNG seed. Same prompt + seed → bit-identical output.
36+
pub seed: u64,
37+
}
38+
39+
impl Default for SamplingConfig {
40+
fn default() -> Self {
41+
Self {
42+
temperature: 0.0,
43+
top_p: 1.0,
44+
repeat_penalty: 1.0,
45+
repeat_penalty_last_n: 64,
46+
seed: 0,
47+
}
48+
}
49+
}
50+
51+
impl SamplingConfig {
52+
pub fn is_greedy(&self) -> bool {
53+
self.temperature <= 0.0 && (self.repeat_penalty - 1.0).abs() < 1e-6
54+
}
55+
}
56+
57+
/// xorshift64* — deterministic, no external crate, perfectly adequate
58+
/// for token sampling. Cryptographic strength is not a goal; cheap
59+
/// seeded reproducibility is.
60+
pub struct Xorshift64 {
61+
state: u64,
62+
}
63+
64+
impl Xorshift64 {
65+
pub fn new(seed: u64) -> Self {
66+
// Reject the all-zero state — xorshift would lock at 0.
67+
let state = if seed == 0 { 0x9E3779B97F4A7C15 } else { seed };
68+
Self { state }
69+
}
70+
71+
pub fn next_u64(&mut self) -> u64 {
72+
let mut x = self.state;
73+
x ^= x >> 12;
74+
x ^= x << 25;
75+
x ^= x >> 27;
76+
self.state = x;
77+
x.wrapping_mul(0x2545F4914F6CDD1D)
78+
}
79+
80+
/// Uniform `f32` in `[0, 1)`.
81+
pub fn next_f32(&mut self) -> f32 {
82+
((self.next_u64() >> 40) as f32) / (1u32 << 24) as f32
83+
}
84+
}
85+
86+
/// HF-style repeat penalty applied in place: divide positive logits and
87+
/// multiply negative logits of recently-emitted tokens by `penalty`.
88+
/// Pushes probability mass off repeated-token attractors that
89+
/// catastrophically dominate greedy decoding on aggressive 3-bit
90+
/// quantization (the original `karın-karın-...` bug class).
91+
pub fn apply_repeat_penalty(logits: &mut [f32], recent: &[u32], penalty: f32) {
92+
if (penalty - 1.0).abs() < 1e-6 {
93+
return;
94+
}
95+
for &tok in recent {
96+
let i = tok as usize;
97+
if i >= logits.len() {
98+
continue;
99+
}
100+
let v = logits[i];
101+
logits[i] = if v >= 0.0 { v / penalty } else { v * penalty };
102+
}
103+
}
104+
105+
/// In-place softmax with the standard max-subtraction trick for
106+
/// numerical stability. After this call `logits` is a valid probability
107+
/// distribution summing to ~1.0. Falls back to uniform on degenerate
108+
/// input (all `-inf`) instead of producing NaNs.
109+
pub fn softmax_inplace(logits: &mut [f32]) {
110+
let max = logits.iter().copied().fold(f32::NEG_INFINITY, f32::max);
111+
if !max.is_finite() {
112+
let u = 1.0 / logits.len() as f32;
113+
for v in logits.iter_mut() {
114+
*v = u;
115+
}
116+
return;
117+
}
118+
let mut sum = 0.0_f32;
119+
for v in logits.iter_mut() {
120+
*v = (*v - max).exp();
121+
sum += *v;
122+
}
123+
if sum <= 0.0 {
124+
let u = 1.0 / logits.len() as f32;
125+
for v in logits.iter_mut() {
126+
*v = u;
127+
}
128+
return;
129+
}
130+
let inv = 1.0 / sum;
131+
for v in logits.iter_mut() {
132+
*v *= inv;
133+
}
134+
}
135+
136+
/// Sample a token id from `probs` after applying top-p nucleus
137+
/// filtering. `probs` must sum to ~1 (call `softmax_inplace` first).
138+
/// `top_p >= 1.0` skips the filter and samples from the full
139+
/// distribution. Always keeps at least one token (the argmax) so a
140+
/// degenerate `top_p = 0` doesn't deadlock.
141+
pub fn sample_top_p(probs: &[f32], top_p: f32, rng: &mut Xorshift64) -> u32 {
142+
debug_assert!(!probs.is_empty());
143+
144+
if top_p >= 1.0 || top_p <= 0.0 {
145+
return categorical(probs, rng);
146+
}
147+
148+
let mut idx: Vec<u32> = (0..probs.len() as u32).collect();
149+
idx.sort_unstable_by(|&a, &b| {
150+
probs[b as usize]
151+
.partial_cmp(&probs[a as usize])
152+
.unwrap_or(std::cmp::Ordering::Equal)
153+
});
154+
155+
let mut cum = 0.0_f32;
156+
let mut cutoff = idx.len();
157+
for (rank, &i) in idx.iter().enumerate() {
158+
cum += probs[i as usize];
159+
if cum >= top_p {
160+
cutoff = rank + 1;
161+
break;
162+
}
163+
}
164+
cutoff = cutoff.max(1);
165+
166+
let kept = &idx[..cutoff];
167+
let mass: f32 = kept.iter().map(|&i| probs[i as usize]).sum();
168+
if mass <= 0.0 {
169+
return kept[0];
170+
}
171+
let r = rng.next_f32() * mass;
172+
let mut acc = 0.0_f32;
173+
for &i in kept {
174+
acc += probs[i as usize];
175+
if r < acc {
176+
return i;
177+
}
178+
}
179+
*kept.last().unwrap()
180+
}
181+
182+
fn categorical(probs: &[f32], rng: &mut Xorshift64) -> u32 {
183+
let r = rng.next_f32();
184+
let mut acc = 0.0_f32;
185+
for (i, &p) in probs.iter().enumerate() {
186+
acc += p;
187+
if r < acc {
188+
return i as u32;
189+
}
190+
}
191+
(probs.len() - 1) as u32
192+
}
193+
194+
/// One-shot helper: run the full pipeline (penalty → temperature →
195+
/// softmax → top-p → sample) on a CPU logit buffer. Mutates `logits`
196+
/// in place (caller may discard or reuse). The caller owns
197+
/// `recent_tokens` (sliding window for the repeat penalty).
198+
pub fn sample_from_logits(
199+
logits: &mut [f32],
200+
recent_tokens: &[u32],
201+
cfg: &SamplingConfig,
202+
rng: &mut Xorshift64,
203+
) -> u32 {
204+
// Repeat penalty restricted to the trailing window — older context
205+
// shouldn't dampen tokens we naturally want to emit again.
206+
let n = cfg.repeat_penalty_last_n.min(recent_tokens.len());
207+
if cfg.repeat_penalty != 1.0 && n > 0 {
208+
let window = &recent_tokens[recent_tokens.len() - n..];
209+
apply_repeat_penalty(logits, window, cfg.repeat_penalty);
210+
}
211+
212+
// Temperature scaling before softmax. `<=0` would mean greedy but
213+
// the caller is responsible for routing greedy elsewhere; clamp to
214+
// a tiny epsilon as a safety net.
215+
let t = cfg.temperature.max(1e-5);
216+
if (t - 1.0).abs() > 1e-6 {
217+
let inv = 1.0 / t;
218+
for v in logits.iter_mut() {
219+
*v *= inv;
220+
}
221+
}
222+
223+
softmax_inplace(logits);
224+
sample_top_p(logits, cfg.top_p, rng)
225+
}
226+
227+
#[cfg(test)]
228+
mod tests {
229+
use super::*;
230+
231+
#[test]
232+
fn xorshift_deterministic() {
233+
let mut a = Xorshift64::new(42);
234+
let mut b = Xorshift64::new(42);
235+
for _ in 0..16 {
236+
assert_eq!(a.next_u64(), b.next_u64());
237+
}
238+
}
239+
240+
#[test]
241+
fn softmax_sums_to_one() {
242+
let mut v = vec![1.0_f32, 2.0, 3.0, 4.0];
243+
softmax_inplace(&mut v);
244+
let s: f32 = v.iter().sum();
245+
assert!((s - 1.0).abs() < 1e-5);
246+
}
247+
248+
#[test]
249+
fn softmax_handles_neg_inf() {
250+
let mut v = vec![f32::NEG_INFINITY; 4];
251+
softmax_inplace(&mut v);
252+
assert!((v.iter().sum::<f32>() - 1.0).abs() < 1e-5);
253+
}
254+
255+
#[test]
256+
fn repeat_penalty_pushes_positive_down() {
257+
let mut v = vec![2.0_f32, 0.5, -0.5, -2.0];
258+
apply_repeat_penalty(&mut v, &[0, 2], 1.5);
259+
assert!(v[0] < 2.0, "positive logit should be divided");
260+
assert!(v[2] < -0.5, "negative logit should be multiplied");
261+
assert_eq!(v[1], 0.5);
262+
assert_eq!(v[3], -2.0);
263+
}
264+
265+
#[test]
266+
fn top_p_keeps_at_least_one() {
267+
let probs = vec![0.4, 0.3, 0.2, 0.1];
268+
let mut rng = Xorshift64::new(1);
269+
let tok = sample_top_p(&probs, 0.0, &mut rng);
270+
assert!(tok < probs.len() as u32);
271+
}
272+
273+
#[test]
274+
fn greedy_flag() {
275+
let g = SamplingConfig::default();
276+
assert!(g.is_greedy());
277+
let s = SamplingConfig {
278+
temperature: 0.7,
279+
..g
280+
};
281+
assert!(!s.is_greedy());
282+
}
283+
284+
#[test]
285+
fn end_to_end_seeded_is_deterministic() {
286+
let mut probs = vec![1.0_f32, 2.0, 3.0, 4.0, 5.0];
287+
let cfg = SamplingConfig {
288+
temperature: 0.7,
289+
top_p: 0.9,
290+
repeat_penalty: 1.0,
291+
repeat_penalty_last_n: 0,
292+
seed: 12345,
293+
};
294+
let mut rng1 = Xorshift64::new(cfg.seed);
295+
let t1 = sample_from_logits(&mut probs.clone(), &[], &cfg, &mut rng1);
296+
let mut rng2 = Xorshift64::new(cfg.seed);
297+
let t2 = sample_from_logits(&mut probs, &[], &cfg, &mut rng2);
298+
assert_eq!(t1, t2);
299+
}
300+
}

crates/lumen-mlx/examples/bench_gemma4_native_e2e.rs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ fn main() -> Result<()> {
7878
let warm_cfg = GenerateConfig {
7979
max_new_tokens: warmup.max(1),
8080
stop_on_eos: false,
81+
sampling: None,
8182
};
8283
// hide `LUMEN_METAL_CAPTURE` from the warmup generate so
8384
// the .gputrace bundle only contains the timed run. We re-set after.
@@ -146,6 +147,7 @@ fn main() -> Result<()> {
146147
let cfg = GenerateConfig {
147148
max_new_tokens: steps,
148149
stop_on_eos: false,
150+
sampling: None,
149151
};
150152
// Marker line for external profilers (e.g. capture_metal_systrace.sh) to
151153
// sync attach with the start of the timed decode window.

crates/lumen-mlx/examples/gemma4_backend_prefix_cache_smoke.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ fn main() -> Result<()> {
7676
&messages,
7777
max_new_tokens,
7878
/* temperature */ 0.0,
79+
/* top_p */ 1.0,
7980
/* thinking */ false,
8081
prefix_key,
8182
)

crates/lumen-mlx/examples/gemma4_lookup_real_prompt_smoke.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ fn main() -> Result<()> {
104104
let cfg = GenerateConfig {
105105
max_new_tokens: max_tokens,
106106
stop_on_eos,
107+
sampling: None,
107108
};
108109

109110
// ── OFF: standard decode ──

crates/lumen-mlx/examples/gemma4_mtp_generate_smoke.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ fn main() -> Result<()> {
5151
let cfg = GenerateConfig {
5252
max_new_tokens: max_tokens,
5353
stop_on_eos: false,
54+
sampling: None,
5455
};
5556

5657
// Baseline (MTP-off).

0 commit comments

Comments
 (0)