You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The curriculum is organized as five parts and 23 self-contained notebooks.
236
+
The curriculum is organized as five parts and 26 self-contained notebooks.
237
237
238
238
```text
239
239
Modern LLM Notebook
@@ -248,6 +248,7 @@ Modern LLM Notebook
248
248
│
249
249
├── Part 2: Training
250
250
│ ├── From GPT-2 to modern models
251
+
│ ├── Model config
251
252
│ ├── Mixture of Experts
252
253
│ ├── Training and loss
253
254
│ ├── Scaling laws
@@ -269,7 +270,8 @@ Modern LLM Notebook
269
270
└── Part 5: Production
270
271
├── Evaluation
271
272
├── Distillation
272
-
└── On-policy distillation
273
+
├── On-policy distillation
274
+
└── RAG
273
275
```
274
276
275
277
Each notebook is designed to be runnable on its own. You can follow the full sequence or jump to a
@@ -287,13 +289,14 @@ topic without depending on hidden runtime state from earlier notebooks.
287
289
| 04 |[Position Encoding](notebooks-en/part1-foundation/04-position-encoding.ipynb)| How does the model know word order? | Sinusoidal encoding, input assembly |
288
290
| 05 |[Attention & Transformer Block](notebooks-en/part1-foundation/05-transformer-block.ipynb)| How does attention move information? | MHA, residuals, normalization |
289
291
| 06 |[Mini-GPT](notebooks-en/part1-foundation/06-mini-gpt.ipynb)| How does a GPT-style model fit together? | Decoder-only model, LM head |
290
-
|08|[BERT Encoder](notebooks-en/part1-foundation/08-bert-encoder.ipynb)| Why can encoder-only models read bidirectionally? | MiniBERT, MLM head |
292
+
|07|[BERT Encoder](notebooks-en/part1-foundation/07-bert-encoder.ipynb)| Why can encoder-only models read bidirectionally? | MiniBERT, MLM head |
| 09 |[From GPT-2 to Modern Models](notebooks-en/part2-training/09-gpt2-to-modern-models.ipynb)| What changed architecturally after GPT-2? | RMSNorm, SwiGLU, RoPE, GQA, QK-Norm, MLA |
298
+
| 08 |[From GPT-2 to Modern Models](notebooks-en/part2-training/08-gpt2-to-modern-models.ipynb)| What changed architecturally after GPT-2? | RMSNorm, SwiGLU, RoPE, GQA, QK-Norm, MLA |
299
+
| 09 |[Model Config](notebooks-en/part2-training/09-model-config.ipynb)| What does each field in a real config.json mean? | vocab_size, hidden_size, layers, heads |
297
300
| 10 |[Mixture of Experts](notebooks-en/part2-training/10-moe.ipynb)| How does sparse expert routing work? | Router gate, top-k experts, aux-free load balancing |
298
301
| 11 |[Training & Loss](notebooks-en/part2-training/11-training-loss.ipynb)| How does a language model learn from prediction errors? | Training loop, loss, gradients, Multi-Token Prediction |
299
302
| 12 |[Scaling Laws](notebooks-en/part2-training/12-scaling-laws.ipynb)| How do model size, data, and compute trade off? | FLOPs estimates, Chinchilla intuition |
@@ -325,6 +328,7 @@ topic without depending on hidden runtime state from earlier notebooks.
325
328
| 23 |[Evaluation](notebooks-en/part5-production/23-evaluation.ipynb)| How do we tell whether a model is better? | Win-rate matrices, RAGAS, judge metrics |
326
329
| 24 |[Distillation](notebooks-en/part5-production/24-distillation.ipynb)| How does a small model learn from a large one? | Soft labels, temperature, logit distillation |
327
330
| 25 |[On-Policy Distillation](notebooks-en/part5-production/25-opd.ipynb)| How can distillation reduce exposure bias? | OPSD, KL estimator taxonomy |
331
+
| 26 |[RAG](notebooks-en/part5-production/26-rag.ipynb)| How does retrieval-augmented generation connect external knowledge to a model? | Chunking, embedding retrieval, reranking, generation |
Copy file name to clipboardExpand all lines: notebooks-en/part1-foundation/03-embedding.ipynb
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -376,7 +376,7 @@
376
376
"\n",
377
377
"Weight decay is a constraint the optimizer applies during each parameter update: first shrink the parameter proportionally (w <- w - lr x lambda x w), then subtract the current gradient direction. This is equivalent to adding a lambda*||w||^2/2 penalty to the loss, applying a stronger pull-back force on large weights to prevent parameter values from diverging.\n",
378
378
"\n",
379
-
"A common intuition is: each row of the Embedding matrix is a token's semantic vector. If we pull the row vector toward the origin (i.e., decay each component's absolute value), the semantic vector's \"length\" is artificially shortened -- and semantics shouldn't have强弱之分. Following this logic, Embedding weights should be excluded from weight decay.\n",
379
+
"A common intuition is: each row of the Embedding matrix is a token's semantic vector. If we pull the row vector toward the origin (i.e., decay each component's absolute value), the semantic vector's \"length\" is artificially shortened -- and semantics shouldn't have an artificial strong/weak ranking. Following this logic, Embedding weights should be excluded from weight decay.\n",
380
380
"\n",
381
381
"But this intuition lacks experimental support. Actual open-source code does the opposite.\n",
Copy file name to clipboardExpand all lines: notebooks-en/part2-training/08-gpt2-to-modern-models.ipynb
+14-4Lines changed: 14 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -34,7 +34,7 @@
34
34
"\n",
35
35
"Take Post-Norm as an example. Post-Norm places normalization after the residual addition: after computing Attention, add the residual, then apply Norm. With few layers, this is easy to understand and works fine. But as layers get deeper, every time the gradient passes through a layer it also passes through normalization, and its scale gets repeatedly altered. Pre-Norm moves normalization before the sublayer, making the residual path more like a direct highway, and deep training is typically more stable.\n",
36
36
"\n",
37
-
"Activation functions follow a similar pattern. ReLU's rule is simple: positive numbers pass through, negative numbers become 0. It's easy to understand and implement, but the gradient is 0 in the negative region, which can lead to \"neurons that don't learn for a long time.\" Modern LLMs more commonly use GELU or SwiGLU - smoother or gated activations that don't simply cut information with a single一刀 (single一刀).\n",
37
+
"Activation functions follow a similar pattern. ReLU's rule is simple: positive numbers pass through, negative numbers become 0. It's easy to understand and implement, but the gradient is 0 in the negative region, which can lead to \"neurons that don't learn for a long time.\" Modern LLMs more commonly use GELU or SwiGLU - smoother or gated activations that don't simply cut information with a single hard cutoff.\n",
38
38
"\n",
39
39
"LLaMA (2023) combines these modern components systematically: Pre-Norm + RMSNorm, SwiGLU, RoPE. Later models like Mistral, LLaMA 3, Qwen, and DeepSeek continue to refine around inference efficiency and long context, with improvements like GQA, MLA, QK-Norm, and more.\n",
40
40
"\n",
@@ -250,7 +250,7 @@
250
250
"rms_values = [f\"{v:.4f}\" for v in x_rmsnorm.tolist()]\n",
"print(f\"\\n-> Different values, but both achieve \"standardization\"\")\n",
253
+
"print('\\n-> Different values, but both achieve \"standardization\"')\n",
254
254
"print(f\"-> LayerNorm forces mean=0, RMSNorm doesn't (but in practice it's often good enough)\")"
255
255
]
256
256
},
@@ -1852,7 +1852,17 @@
1852
1852
"source": [
1853
1853
"## Exercises\n",
1854
1854
"\n",
1855
-
"To be added"
1855
+
"**Exercise 1: RMSNorm vs LayerNorm edge cases**\n",
1856
+
"\n",
1857
+
"Change the RMSNorm hand-calculation input to `[1, -1, 1, -1]` and `[10, 10, 10, 10]`. Observe the LayerNorm and RMSNorm outputs. Write two conclusions: which one forces mean 0, and which one only controls RMS?\n",
"Assume `hidden_size=4096` and a regular FFN uses `intermediate_size=11008`. Estimate the linear-layer parameter count for a ReLU/GELU FFN and a SwiGLU FFN. Explain why SwiGLU often adjusts the intermediate dimension to control parameter size.\n",
1862
+
"\n",
1863
+
"**Exercise 3: KV Cache savings from GQA**\n",
1864
+
"\n",
1865
+
"Assume `num_attention_heads=32`, `num_key_value_heads=8`, `head_dim=128`, and sequence length 4096. Calculate how many K/V scalars a standard MHA layer and a GQA layer cache, then report the savings ratio."
Copy file name to clipboardExpand all lines: notebooks-en/part2-training/10-moe.ipynb
+13-3Lines changed: 13 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -742,7 +742,7 @@
742
742
"\n",
743
743
"| Metric | What to check | Warning sign |\n",
744
744
"|:---|:---|:---|\n",
745
-
"| `task_loss` | Is the main task being learned? | Loss not decreasing or剧烈 jittering |\n",
745
+
"| `task_loss` | Is the main task being learned? | Loss not decreasing or severe jittering |\n",
746
746
"| `expert_usage` | Selection ratio per expert | One expert consistently at 80%+ |\n",
747
747
"| `imbalance` | Gap between busiest and idlest expert | Gap persistently large |\n",
748
748
"| `router_entropy` | Is the router becoming too rigid too early? | Low entropy, meaning selection is too peaked |\n",
@@ -937,7 +937,17 @@
937
937
"source": [
938
938
"## Exercises\n",
939
939
"\n",
940
-
"To be added."
940
+
"**Exercise 1: Observe router collapse**\n",
941
+
"\n",
942
+
"Change router top-k from 2 to 1, then try increasing or decreasing the softmax temperature. Record each expert's usage count and decide whether a few experts are being overused.\n",
943
+
"\n",
944
+
"**Exercise 2: Implement a simple capacity limit**\n",
945
+
"\n",
946
+
"Give each expert a maximum number of accepted tokens, for example `capacity = ceil(num_tokens / num_experts * 1.25)`. Tokens beyond capacity can be skipped or routed to their second-choice expert. Compare expert usage with and without the capacity limit.\n",
947
+
"\n",
948
+
"**Exercise 3: Calculate active parameters**\n",
949
+
"\n",
950
+
"Assume 8 experts, each with 100M parameters, plus 300M parameters in the router and shared layers. Each token activates only 2 experts. Compute total parameters and per-token active parameters, then explain why MoE can have many parameters but relatively low per-token compute."
0 commit comments