Hands-on Discussion: Chapter 3 - A blueprint to modern transformers

[peremartra]

This is the space to share your findings from the experiments in Chapter 3. In this lab, we went beyond the surface of Transformer models to examine what's actually inside: how MLP blocks are structured, why actual models use GLU variants, and how architectural decisions like expansion ratios shape a model's character.

🎯 The Challenges

Pick one (or more) and share what you find:

1. GLU Hunters: Load Qwen-3-0.5B and Gemma-3-270M and compare their MLP structure with Llama-3.2-1B. What do Qwen and Gemma share that Llama doesn't?
2. Expansion Ratio Analysis: Calculate the expansion ratios across at least two models. Would you classify Qwen-3 as a "wide" or "deep" model? What trade-offs does that imply?
3. Attention Archaeology: Examine Phi-4's qkv_proj structure. What attention mechanism does it implement, and what makes it different from standard multi-head attention?
4. Multimodal Expedition: Load a multimodal model like gemma-3n-E4B-it and open its config.json. What sections or keys clearly don't belong to a standard text Transformer?

📊 Share Your Results

Model(s) analyzed: [e.g., Qwen-3-0.5B vs Gemma-3-270M]
Challenge attempted: [e.g., GLU Hunters]
Key finding: [What did you discover?]
What surprised you: [Anything unexpected in the architecture?]

No need to have all the answers. A partial finding or a good question is just as valuable.

[peremartra]

I'll start with something that caught my attention while preparing the next chapters.

I loaded Qwen3.5-0.6B and the structure is genuinely different from everything we've analyzed so far. Instead of uniform Transformer blocks, it alternates between two types of layers: a Qwen3_5GatedDeltaNet (a linear attention mechanism) and a standard Qwen3_5Attention — roughly one standard attention block every 4 layers.

It also uses a vocabulary of 248,320 tokens, which is a strong hint that this model is designed with multimodal input in mind.

What's interesting for us is that despite this hybrid architecture, the MLP module remains structurally identical to what we've seen in other models: gate_proj, up_proj, down_proj, SiLU activation. The depth pruning and distillation techniques from Chapter 2 apply without modification.

This is not a coincidence, the techniques in this book have been designed from the ground up to be as model-agnostic as possible. The goal is that you can take what you learn here and apply it to whatever architecture comes next, not just the ones we use as examples.

We'll revisit this model in Chapter 6. Wanted to share it here as a preview of how varied real-world architectures can get — and why understanding the internals matters.

[entzyeung]

Hi @peremartra,

I would really love to join — this looks really interesting! It's just that I'm currently caught up with some other projects unfortunately. Like this one: https://uk-property-app.entzai.com (a multi-agent RAG system over HM Land Registry data).

I'll definitely be following the discussion closely though, and if I manage to carve out some time I'll try to run a couple of the GLU/expansion-ratio experiments and share what I find. Really appreciate you putting these hands-on threads together — super valuable for the community!

Best,
Lorentz

[peremartra]

Hi Lorentz!

Thanks for stopping by. Your project looks genuinely interesting, teh site is amazing! and multi-agent RAG over Land Registry data is no small thing. Good luck with it!

he discussion will be here whenever you find a gap.

You'll always be welcome.

Pere.