Hands-on Discussion: Chapter 6 - Knowledge recovery through distillation

[peremartra]

Discussion

This is the space to share your findings from the experiments in Chapter 6. This lab is different from the previous ones — there's a concrete target to beat.

We've published a baseline model at oopere/gemma-3-270m-14L-distilled that achieved 88.75% capability retention and 11.06 perplexity using 40K samples and soft & hard labels distillation only. Your mission: surpass it using the techniques from this chapter.

No A100? No problem. The baseline training took ~68 minutes on an A100. If you're on a T4 or similar, you can still contribute: experiment with loss weights, mapping strategies, and block selection, then share your configuration ideas and partial results here. The discussion is as valuable as the final numbers.

🎯 The Challenges

Pick one (or more) and share what you find:

1. Beat the Baseline: Using CH06_NB03_Hands_on.ipynb, build a 14-block student that outperforms the baseline in at least 1 of the 5 benchmarks (arc_easy, winogrande, hellaswag, lambada_openai, piqa) and in overall capability retention. You can change anything: block selection, loss weights (alpha/beta/gamma), mapping strategy, dataset composition, or training hyperparameters. The notebook generates a model card automatically — once you're happy with your results, publish it to the Hugging Face Hub.

2. The Data Efficiency Frontier: The baseline used 40K samples. Can you match it (within 2% capability retention) using only 15K? Activate FDD (delta=0.1-0.2) and Skew KLD (skew_alpha=0.1) to squeeze more learning from each sample. What's the minimum sample count where you can still recover >85% of capabilities?

3. Combining Width and Depth Pruning (Advanced): Open CH06_NB02_Width_Pruned_Model_Recovery.ipynb and add depth pruning on top: remove two additional blocks from the already width-pruned model using optipfair.prune_model. Use the layer importance analysis from section 6.1 to decide which blocks to cut, then apply compound loss distillation to this doubly-pruned architecture.

4. Community Challenge: Once you've beaten the baseline, publish your model to the Hugging Face Hub using the auto-generated model card and drop the link here.

📊 Share Your Results

Model analyzed:           [e.g., gemma-3-270m, 14 blocks]
Challenge attempted:      [e.g., Beat the Baseline / Data Efficiency]
Key config changes:       [e.g., alpha=0.35, beta=0.45, gamma=0.2, FDD activated]
Benchmark results:        [e.g., hellaswag: 0.52 → 0.57 vs baseline 0.54]
Capability retention:     [e.g., 89.3% vs baseline 88.75%]
Training time / hardware: [e.g., 55 min on A100 / 3h on T4 with 15K samples]
HF Hub link:              [if published]
What had the biggest impact: [e.g., switching to selected mapping / activating Skew KLD]

Questions, failed experiments, and OOM errors are just as welcome as polished results. If you tried something that didn't work, share it — knowing what doesn't help is genuinely useful for everyone building on these techniques.