b10091: ~35% generation speed regression on 8GB VRAM — 3 root causes identified (fit+no-mmap incompatibility, GPU not boosting, CPU MoE regression) + 30+ tuning experiments

[Sensenkawa]

b10091: ~36% generation speed regression on 8GB VRAM — GPU not boosting to max clock, fit abort limits offload

Environment

Field	Value
GPU	NVIDIA GeForce RTX 4060 Laptop 8GB (driver 610.62)
CPU	Intel Core Ultra 5 125H (16 logical cores)
RAM	24 GB DDR5
OS	Windows 11 24H2
llama.cpp	b10091 (commit b4d6c7d8f), CUDA 13.3 build
Previous working	b9536 (commit 308f61c31), CUDA 13.3 build
Model	Ornith-1.0-35B-MTP-APEX-I-Quality.gguf (35B MoE, 21GB, IQ3_S, MTP-enabled)

Test configuration (identical on both versions)

llama-server.exe ^
  -m "models\Ornith-1.0-35B-MTP-APEX-I-Quality.gguf" ^
  -ngl 999 ^
  --n-cpu-moe 35 ^
  --flash-attn on ^
  --jinja ^
  -c 102400 ^
  -np 1 ^
  --no-warmup ^
  --no-mmap ^
  --fit on ^
  --fit-target 640 ^
  -t 14 ^
  --cpu-range 0-15 ^
  --cpu-strict 1 ^
  -tb 8 ^
  -b 512 ^
  -ub 512 ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --cache-type-k-draft q8_0 ^
  --cache-type-v-draft q8_0 ^
  --prio 2 ^
  --cache-ram 2048 ^
  --mlock ^
  --poll 50 ^
  --spec-type draft-mtp ^
  --spec-draft-n-max 4 ^
  --spec-draft-p-min 0.55 ^
  --temp 0.6 ^
  --top-k 20 ^
  --top-p 0.95 ^
  --presence-penalty 0.3 ^
  --frequency-penalty 0.3 ^
  --repeat-penalty 1.0 ^
  --port 8080

Speed comparison (V2 benchmark, Hot phase)

Metric	b9536 🏆	b10091 ❌	Change
Hot Short gen	37.9 t/s	25.3 t/s	-33%
Hot Long gen	29.3 t/s	22.4 t/s	-24%
Hot Agent gen	34.9 t/s	10.2 t/s	-71%
Hot RealSim gen	27.4 t/s	25.0 t/s	-9%
Hot average	32.4 t/s	20.7 t/s	-36%
MTP draft acceptance	69-93%	69-88%	Similar

Note: The Agent test's 71% drop is likely due to the short multi-turn nature of the test amplifying the GPU clock issue. RealSim (longest, most sustained) is least affected at -9%.

GPU clock analysis

The GPU's maximum clock is 3105 MHz, but during inference on b10091 it only reaches 1200–1455 MHz — less than half of its potential. The GPU does enter P0 (the highest performance state), but the actual clock within P0 is far below the card's rated maximum.

Phase	P-state	Actual clock	Max possible	Power
Idle	P8	210 MHz	3105 MHz	1.4 W
Cold Short	P8→P4	210→465 MHz	3105 MHz	12 W
Cold Long	P4	465→900 MHz	3105 MHz	12 W
Warmup	P0	1200 MHz	3105 MHz	19 W
Hot Short	P0	1200 MHz	3105 MHz	19 W
Hot RealSim	P0	1455 MHz	3105 MHz	23 W

The GPU is in P0 but clocked at 1200-1455 MHz instead of the expected 2500-3105 MHz. This is the primary cause of the ~36% speed regression.

Fit abort analysis

b10091's --fit allocates less VRAM than b9536:

Metric	b9536	b10091
VRAM used after fit	~7901 MiB	6581 MiB
Offload headroom	~180 MiB	~1500 MiB wasted

The --fit algorithm on b10091 aborts early, leaving ~1.5 GB of VRAM unused. This means fewer layers are offloaded to the GPU, and more computation runs on the CPU, compounding the GPU clock issue.

Summary

The ~36% regression is caused by two factors:

1. GPU not reaching max boost clock: P0 is entered but clocks only reach 1200-1455 MHz instead of 2500-3105 MHz. This suggests a change in how b10091's CUDA kernel launch pattern interacts with the NVIDIA driver's boost algorithm.

2. Fit abort leaves VRAM unused: --fit only allocates 6581 MiB vs 7901 MiB on b9536, wasting ~1.5 GB of VRAM and reducing GPU offload.

Both issues are absent in b9536 on the exact same hardware, configuration, and model.

Correction note

An earlier version of this report claimed 70-80% regression. That was a false alarm caused by missing cudart DLLs in the test environment — the server fell back to full CPU execution. After installing the correct CUDA runtime DLLs, the real regression is ~36% as documented above.

[Sensenkawa]

Consolidated update: 3 root causes identified, 30+ tuning experiments, zooming in on the fix

Correction note

Earlier reports claiming 70-80% regression were false alarms caused by missing cudart64_13.dll/cublas64_13.dll in the bin directory — the server fell back to full CPU inference (~2 t/s). After installing the correct CUDA runtime DLLs, the real regression is ~35% (32.4 → 21.1 t/s). All data below is from the corrected test environment.

Real regression (same config, same model, same hardware)

Metric	b9536 🏆	b10091 ❌	Change
Hot Short gen	37.9 t/s	26.6 t/s	-30%
Hot Long gen	29.3 t/s	8.0 t/s	-73%
Hot Agent gen	34.9 t/s	26.3 t/s	-25%
Hot RealSim gen	27.4 t/s	23.5 t/s	-14%
Hot average	32.4 t/s	21.1 t/s	-35%
GPU clock (Hot)	2625 MHz / 47W	1275-1455 MHz / 18W	-45% freq
VRAM used	7901 MiB	7901 MiB	same

3 root causes identified

Root cause 1: + incompatibility (new in b10091)

When is set (which sets ), refuses to work:

b9536 had no such restriction — fit worked normally and dynamically offloaded non-critical layers to CPU. This is the single behavioral change between b9536 and b10091 that we can directly attribute. Removing and restores fit but doesn't free VRAM (model + 100K KV cache naturally occupies 7.9GB).

Root cause 2: GPU not boosting

GPU enters P0 but only reaches 1275-1455 MHz (max 3105 MHz), power ~18W (max 47W). We tried forcing clock with ERROR: Option -lgc is missing its value. Please run 'nvidia-smi -h' for help. — only +3% improvement (noise level), because the bottleneck is CPU MoE, not GPU frequency. The GPU boost algorithm is simply not triggered by b10091's CUDA kernel launch pattern.

Root cause 3: CPU MoE path regression

Long generation is the worst hit (8.0 vs 29.3, -73%). The 200-token sustained generation exposes a regression in the CPU-side MoE routing (ggml-backend scheduling). Short and Agent are less affected (30% and 25% drops) because they involve fewer consecutive tokens.

What we tried: 30+ tuning experiments

Best b10091 config found (non-champion bat): v5 baseline + + → 24.8 avg (only -23% from 32.4). But this sacrifices 100K context (uses 16K).

Best champion-bat config: champion bat + → 22.3 avg (-31%). Keeps 100K context but RealSim times out.

Full tuning table (30+ experiments):

Config	Change	Short	Long	Agent	RealSim	avg
v5 baseline	-c16384, moe35, -t14	29.0	8.7	28.7	18.9	21.3
P1a 🏆	-t 8 --cpu-range 0-7	30.8	25.8	21.9	20.6	24.8
P1b	-t 16 (all cores)	25.9	19.4	9.6	22.6	19.4
P4a 🟢	moe33, -c16384, -t8	32.6	27.1	22.6	23.8	26.5
Champion bat	-c102400, -ngl999, --no-mmap, -t14	26.6	8.0	26.3	23.5	21.1
t12 🟢	-t 12 --cpu-range 0-11	29.1	4.6	31.5	23.9	22.3
MTP off	speculative off	22.4	22.0	8.8	18.2	17.9

What we're asking for

Three specific asks for the maintainers:

1. Fix + incompatibility (most impactful)

The log message says . Either:

• (a) Allow fit to work even when is set (b9536 behavior), or
• (b) If there's a technical reason for the restriction, document it clearly in --fit help text and suggest the workaround

This is the single most impactful fix — without it, 8GB VRAM users with MoE models can't use to dynamically manage GPU/CPU layer allocation.

2. Investigate why GPU doesn't boost

Something between b9536 (308f61c) and b10091 (b4d6c7d) changed the CUDA kernel launch pattern enough that the NVIDIA driver's boost algorithm doesn't trigger. The GPU enters P0 but at 1275-1455 MHz instead of 2625 MHz. This is reproducible on a fresh cold boot, not a sleep/wake issue.

3. Review CPU MoE scheduling regression

Long generation shows -73% regression specifically. The CPU-side MoE routing path (ggml-backend) appears to have degraded. This is the hardest to isolate without a git bisect, but the impact is clear.

Summary

Version	Config	Hot avg	vs b9536
b9536 🏆	MTP champion, 100K ctx	32.4 t/s	—
b10091	Champion bat (same)	21.1 t/s	-35%
b10091 best	P-Core tuned, 16K ctx	24.8 t/s	-23%

The most impactful single fix would be restoring compatibility with (root cause 1). The GPU boost issue (root cause 2) and CPU MoE regression (root cause 3) are secondary but still significant.

Full report: ISSUE-b10091-regression-consolidated.md (will upload)

[sanjayahari]

@Sensenkawa Looking at common_params_fit_impl in fit.cpp — the margin estimation at lines 250-256 computes projected_free = dmd.free - projected_used. On 8GB cards, is there a chance dmd.free is underreported at startup (before drivers free up memory), making the fit think less VRAM is available than it actually is?

[Sensenkawa]

Real-world b9536 speed — rolling back confirms the regression

After the corrected data (cudart DLL fix), we rolled back to b9536 (commit 308f61c) with the exact same configuration and model. The difference is dramatic:

b9536 live server log (2026-07-23, 22:18 run, 5+ hours of continuous operation)

Metric	b9536 🏆	b10091 ❌	Change
Average generation speed	~46 t/s	~21 t/s	-54%
Prefill (large prompt)	170-242 t/s	28-37 t/s (or timeout)	-80%
GPU clock	2625 MHz / 47W	1275-1455 MHz / 18W	-45% freq
--fit + --no-mmap	Works	Aborts	❌
Session stability	5+ hrs stable	N/A	✅

Sample timings from the live log (b9536)

Prefill comparison

Scenario	b9536	b10091
18K prompt prefill	242 t/s	28.5 t/s (-88%)
11K prompt prefill	221 t/s	(timeout)
7K prompt prefill	181 t/s	~37 t/s (-80%)
5.5K prompt prefill	169 t/s	—

The gap is wider than our benchmark suggests

The 21.1 t/s average from our V2 benchmark (corrected) was already bad. But the live server on b9536 is averaging ~46 t/s across hundreds of tasks — more than double. The real regression is closer to 50-55% for generation and 80%+ for prefill.

Summary

b9536 works correctly — GPU boosts to 2625 MHz, fit offloads properly, prefill is fast, generation is stable at 40-75 t/s. The three root causes identified earlier (fit+no-mmap incompatibility, GPU not boosting, CPU MoE regression) are all confirmed by this rollback test.

The most impactful fix would be restoring --fit compatibility with --no-mmap (root cause 1). Without it, 8GB VRAM users with MoE models lose the ability to dynamically manage GPU/CPU layer allocation, which is critical for fitting large models into limited VRAM.

[Sensenkawa]

Correction: the ~46 t/s figure was wrong

I made a mistake in the previous comment — the calculation includes prompt tokens (prefill), not just generated tokens. The real generation-only speed () from the live b9536 server log is:

• Average: 26.7 t/s (151 samples from a 5-hour session)
• Median: 25.8 t/s
• Range: 17.4 - 38.0 t/s

The benchmark data remains the cleanest comparison:

Metric	b9536 🏆	b10091 ❌	Change
V2 benchmark Hot avg	32.4 t/s	21.1 t/s	-35%
Live server avg tg	26.7 t/s	N/A (no live data)	—

The ~35% regression from the benchmark is the correct figure. The three root causes (fit+no-mmap incompatibility, GPU not boosting, CPU MoE regression) are unchanged.