Skip to content

README.md

Latest commit

 

History

History
94 lines (69 loc) · 7.59 KB

README.md

File metadata and controls

94 lines (69 loc) · 7.59 KB
Raw

Research — Model Optimization Experiments

Quantization, pruning, and fine-tuning experiments for Gemma 4 E2B (production model for plumber-bot).

TL;DR

# Approach Score Size Speed Verdict
12+15 QAT IQ3_XXS + imatrix + spec decoding (PROD) 3939/3940 (99.97%) 2.5 GB + 380 MB draft 93 tok/s ✅ Current prod
12 QAT IQ3_XXS + imatrix (standalone) 3938/3940 (99.95%) 2.5 GB 65 tok/s Prev prod (rollback)
01 Unsloth UD-Q4_K_XL (rollback option) 3933/3940 (99.82%) 2.5 GB 74 tok/s ✅ Faster single-stream
15 Spec decoding (domain-distilled 0.5B draft) +1 test, 1.43× faster +380 MB 93 tok/s ✅ Zero-risk speedup
09 Cascade router (distill fast-tier) 3929/3940 (99.7%) 380 MB fast-tier ~1.6× real traffic ✅ Validated, ceiling reached
02 Imatrix Q4_K_M 3934/3940 (99.85%) 3.2 GB 67 tok/s ✅ Best reproducible
— Q3_K_M + imatrix (non-QAT) 3933/3940 (99.80%) 3.0 GB 67 tok/s ❌ Bigger than prod
— Q2_K_XL (QAT) 3892/3940 (98.78%) 2.1 GB 35.6 tok/s ❌ Slower (kernel)
07 Distillation (Qwen 0.5B) 3810/3940 (96.7%) 380 MB 253 tok/s ✅ Cascade fast-tier
08 LoRA fine-tune (Unsloth) 3305/3940 (83.9%) 3.2 GB 63 tok/s ❌ Catastrophic
— Activation zeroing 10% 3909/3940 (99.21%) 3.18 GB 64 tok/s ❌
— Wanda zeroing 10% 3900/3940 (98.98%) 3.18 GB 69 tok/s ❌

Current prod: QAT IQ3_XXS + imatrix + spec decoding — 3939/3940 (99.97%), 93 tok/s (1.43× over baseline), zero quality risk. See 12-qat-q3/ + 15-spec-decoding/. Rollback: Q4_K_XL (Unsloth) — 99.82%, 74 tok/s single-stream. See 01-unsloth-qat/. Avoid: LoRA fine-tuning on QAT weights (catastrophic forgetting). Pruning on this HW (memory-bound, 0% speedup for unstructured; code-level blocker for structured). Key wins (2026-06-21 to 06-24): IQ3_XXS beats Q4 on quality (+5 tests). Spec decoding adds 1.43× speed via domain-distilled draft. --parallel 16 gives ~2-3× aggregate scaling under concurrent load. Prefix caching gives 9× warm TTFT.

Folders

Quantization

Folder Description
01-unsloth-qat/ Unsloth Dynamic 2.0 QAT — rollback model (reproduce.sh SHA-pinned)
02-imatrix-quant/ Domain-specific imatrix quantization (4 imatrix files + calibrations)

Pruning

Folder Description
03-pruning-analysis/ Architecture, layer ablation, 3 zeroing methods (Method A/B/C) + all data & tools

Fine-Tuning

Folder Description
07-distillation/ Knowledge distillation to Qwen2.5-0.5B (380 MB, 253 tok/s) — spec decoding draft source
08-unsloth-lora/ Unsloth BF16 LoRA + full fine-tune — both failed catastrophically

Inference Optimization (2026-06 sessions)

Folder Description
09-cascade-router/ Cascade router Phase 1+2: distill fast-tier + E2B safety-tier (99.7%, ceiling reached)
09-quant-sweep/ Q3/IQ4/Q2 quant sweep + concurrent throughput analysis
10-npu-offload/ NPU offload investigation (hardware ready, tooling pending)
11-semantic-cache/ Semantic cache investigation (not worth it — no real repetition)
12-qat-q3/ QAT IQ3_XXS production model (99.95% standalone, reproduce.sh)
13-prefix-cache/ Prefix caching (already optimal — 9× warm TTFT)
14-gbnf-grammar/ GBNF grammar (not worth it — outputs already 100%)
15-spec-decoding/ Spec decoding — current prod (99.97%, 1.43× faster, zero risk)

Lab & Infrastructure

Folder Description
quantization-lab/ Shared PyTorch infra: model loader, bench runner, calibration
pruned-execution/ Pruning analysis + 13 inference acceleration approaches
vision-triage-design/ Vision triage design test set — informed bench/tests/*/vision_*

Key Finding

The Google QAT weights are near-optimal for 4-bit quantization. The improvements came from:

  1. Importance-aware quantization — IQ3_XXS + imatrix beats Q4_K_XL on quality (3938 vs 3933) at the same size
  2. Speculative decoding — domain-distilled 0.5B as draft model gives 1.43× speedup with zero quality risk
  3. Concurrent batching — --parallel 16 gives ~2-3× aggregate throughput under load
  4. Prefix caching — system prompt (538 tokens) cached at 9× warm speedup

Pruning (all 4 approaches) and LoRA fine-tuning on QAT weights are dead ends. The cascade router reaches a ceiling at 99.7%.

Hardware

  • AMD Ryzen AI MAX+ 395 (Strix Halo), 128 GB unified memory
  • PyTorch 2.12 ROCm nightly (gfx1151)
  • llama.cpp (CPU build for imatrix/quantize, ROCm container for inference)

Benchmark

All results use the full 3940-test suite:

  • 1830 good-path tests (plumbing conversations)
  • 2110 attack tests (prompt injection, social engineering, encoding tricks, vision attacks)

Production baseline: python3 bench/run_tests.py --backend llama-gemma4-e2b-qat