-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathThe Mathematics of Learning.html
More file actions
680 lines (586 loc) · 45.4 KB
/
Copy pathThe Mathematics of Learning.html
File metadata and controls
680 lines (586 loc) · 45.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>The Mathematics of Learning — Gradient Descent & Backpropagation</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Playfair+Display:wght@400;500;600&family=JetBrains+Mono:wght@400;500;600&family=Source+Serif+4:wght@400;500;600&display=swap" rel="stylesheet">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.21/dist/katex.min.css" crossorigin="anonymous">
<script src="https://cdn.jsdelivr.net/npm/katex@0.16.21/dist/katex.min.js" crossorigin="anonymous"></script>
<link rel="stylesheet" href="styles.css" />
</head>
<body>
<div id="scroll-progress"></div>
<nav id="side-nav">
<a href="#hero" data-label="Hero"></a>
<a href="#s1" data-label="What is GD?"></a>
<a href="#s2" data-label="Loss Landscape"></a>
<a href="#s3" data-label="Variants"></a>
<a href="#s4" data-label="Learning Rate"></a>
<a href="#s5" data-label="Optimizers"></a>
<a href="#s6" data-label="Network"></a>
<a href="#s7" data-label="Forward Pass"></a>
<a href="#s8" data-label="Backprop"></a>
<a href="#s9" data-label="Training Loop"></a>
<a href="#s10" data-label="Playground"></a>
</nav>
<!-- ============ SECTION 0: HERO ============ -->
<section id="hero" class="full">
<canvas id="hero-canvas"></canvas>
<div class="hero-content">
<div class="sub">An Interactive Essay · 10 Chapters</div>
<h1>The Mathematics<br/>of Learning</h1>
<div class="meta">Gradient Descent · Variants · Backpropagation</div>
</div>
<div class="scroll-cue">
<span>SCROLL</span>
<span class="line"></span>
</div>
</section>
<div class="divider"></div>
<!-- ============ ABSTRACT ============ -->
<section id="abstract" style="padding-top:96px;padding-bottom:24px">
<div class="abstract reveal">
<span class="lab">An interactive essay · ~15 minute read</span>
<p>Modern AI does one thing, repeatedly: it changes its own numbers — billions of them — to make its mistakes smaller. The procedure that picks <em>which numbers to change and by how much</em> is called <strong style="color:var(--text)">gradient descent</strong>. The trick that lets it work efficiently across a deep network is called <strong style="color:var(--text)">backpropagation</strong>. Together, these two ideas power every neural network ever trained.</p>
<p style="margin-top:14px">This essay assumes you can read basic notation (sums, derivatives) but does not assume you've trained a model. We start with a marble rolling down a curve and end with a working classifier you can train in your browser.</p>
<ul>
<li><strong>Chapters 1–2</strong> the idea & the landscape</li>
<li><strong>Chapters 3–5</strong> the algorithm variants</li>
<li><strong>Chapters 6–8</strong> applying it to neural networks</li>
<li><strong>Chapters 9–10</strong> the full training loop, playground</li>
</ul>
<div class="color-key">
<span class="ck-label">Color key</span>
<span class="ck-item"><span class="ck-dot cyan"></span>Definitions & forward direction</span>
<span class="ck-item"><span class="ck-dot amber"></span>Parameters & learning rate</span>
<span class="ck-item"><span class="ck-dot rose"></span>Gradients & error signals</span>
<span class="ck-item"><span class="ck-dot green"></span>Optimal states & convergence</span>
</div>
</div>
</section>
<div class="divider"></div>
<!-- ============ SECTION 1: WHAT IS GD ============ -->
<section id="s1">
<div class="section-num">Chapter 01</div>
<h2 class="reveal">Finding the bottom of a curve.</h2>
<p class="lede reveal">Every learning algorithm reduces to one question: <em class="term">which way is down?</em> Gradient descent answers it by repeatedly taking a small step in the direction opposite to the gradient.</p>
<div class="defbox reveal">
<span class="label">Definition</span>
<h4>Gradient Descent</h4>
<p>A first-order iterative optimization algorithm. To minimize a differentiable function <span class="mono">L(θ)</span>, start at some <span class="mono">θ</span>, compute the gradient <span class="mono">∇L(θ)</span>, take a small step in the direction <em>opposite</em> to it, and repeat. The function gets smaller every step, until it cannot get any smaller.</p>
</div>
<p class="wide reveal">Most things we want a computer to learn — recognize a face, translate a sentence, predict tomorrow's price — can be framed as a single number we want to be small. That number is called the <span class="gloss" data-label="loss / cost function" data-def="A scalar measure of how badly the model is doing on the training data. Lower is better. Common examples: mean-squared error for regression, cross-entropy for classification.">loss</span>. It depends on every adjustable parameter in the model. Make the parameters bigger or smaller, and the loss changes. <em>Find the parameter values that make the loss as small as possible</em>, and you have a model that works.</p>
<p class="wide reveal">The problem is that a modern model has millions or billions of parameters. We cannot try every combination. We cannot even visualize them. But we <em>can</em> ask a local question, fast: <em>if I were to change each parameter just a little, how would the loss respond?</em> That answer is the gradient — and it tells us, exactly, which direction makes the loss go up. Go the other way.</p>
<div class="two-col reveal">
<div>
<p>For a smooth scalar loss <span class="mono">L(θ)</span> the <span class="gloss" data-label="gradient" data-def="The vector of partial derivatives. In one dimension it's just the slope. In many dimensions it's the direction of steepest ascent and a magnitude equal to that steepness.">gradient</span> <span class="mono">∇L</span> points in the direction of steepest <em>increase</em>. To minimize, we negate it. Each iteration nudges <span class="mono">θ</span> a small distance <span class="mono">α</span> (the <span class="gloss" data-label="learning rate" data-def="A small positive number — typically between 1e-5 and 1.0 — that scales the gradient before subtracting. The single most important hyperparameter to tune.">learning rate</span>) in the descent direction, evaluates the new loss, and repeats.</p>
<p>Watch the dot. At every step the tangent line is recomputed, an arrow projects the next move, and <span class="mono">θ</span> slides further into the valley. As the slope flattens near the minimum, steps get smaller automatically — the gradient itself is shrinking.</p>
<div class="eq" id="eq-gd">
<span class="corner-label">Update Rule · Hover Any Symbol</span>
</div>
<div class="callout cyan reveal">
<span class="tag">Read it as</span>
<div><p><em>"new θ equals old θ minus alpha times the gradient of L evaluated at θ."</em></p>
<p style="color:#888888">Subtraction is the only thing that makes this "descent" and not "ascent." Flip the sign and you'd be climbing the loss surface instead — exactly what you don't want.</p></div>
</div>
<div class="intuition">
You stand on a foggy mountain. You cannot see — only feel the ground beneath you. Gradient descent says: feel the slope under your feet, then take one careful step downhill. Repeat until the ground is flat.
</div>
</div>
<div>
<div class="canvas-frame">
<span class="label">FIG. 1.1 · LOSS CURVE WITH DESCENT</span>
<canvas id="s1-canvas"></canvas>
<div class="controls">
<button class="btn" id="s1-restart">↺ Restart</button>
<span style="color:var(--muted);font-size:11px">click the curve to drop the dot</span>
</div>
</div>
<div class="callout reveal" style="margin-top:24px">
<span class="tag">Why subtract?</span>
<p>The gradient is a vector that points "uphill." If the loss looks like a valley and you stand on its left wall, the gradient there points <em>up and to the left</em>. To go <em>down</em>, you have to walk in the opposite direction — down and to the right. Subtracting the gradient is exactly that.</p>
</div>
<div class="callout cyan reveal">
<span class="tag">In practice</span>
<p>Gradient descent is everywhere: linear regression, logistic regression, support vector machines, deep neural networks, large language models. The thing that varies between them is <em>what</em> we are computing the gradient of, and <em>how</em> we compute it efficiently — not the descent step itself.</p>
</div>
</div>
</div>
</section>
<div class="divider"></div>
<!-- ============ SECTION 2: LOSS LANDSCAPE ============ -->
<section id="s2">
<div class="section-num">Chapter 02</div>
<h2 class="reveal">The landscape is not always kind.</h2>
<p class="lede reveal">A real loss surface has valleys, ridges, plateaus and ravines. Where you start determines where you end.</p>
<p class="wide reveal">In our first example we used a single parameter. Real models have millions. The loss surface is a function in a million-dimensional space — impossible to draw. But the <em>geometry</em> generalizes from two dimensions: there are still hills you can get stuck on top of, valleys you fall into, ridges that funnel you the wrong way, and flat regions where the gradient is nearly zero and progress stalls. Below we visualize a 2-parameter version. The lessons are the same.</p>
<div class="canvas-frame reveal" id="landscape-wrap" style="aspect-ratio:16/9">
<span class="label">FIG. 2.1 · LOSS LANDSCAPE · 3 INITIALIZATIONS</span>
<canvas id="s2-canvas"></canvas>
<div class="legend">
<div><span class="dot" style="background:#06b6d4;box-shadow:0 0 6px #06b6d4"></span> Global minimum</div>
<div><span class="dot" style="background:#f59e0b"></span> Local minimum</div>
<div><span class="dot" style="background:#f43f5e"></span> Saddle point</div>
<div><span class="dot" style="background:#888888"></span> Plateau</div>
</div>
<div class="controls">
<button class="btn" id="s2-reset">↺ Reset Balls</button>
</div>
</div>
<div class="text-two reveal" style="margin-top:36px">
<div>
<span class="lab">Global minimum</span>
<h4>The lowest point.</h4>
<p>The value of the parameters that produces the smallest possible loss anywhere on the surface. In a non-convex landscape — which most real ones are — gradient descent is <em>not</em> guaranteed to find this. It finds whatever valley you happen to roll into.</p>
</div>
<div>
<span class="lab">Local minimum</span>
<h4>A trap.</h4>
<p>A point where every nearby direction goes <em>up</em>, so the gradient is zero and descent stops. But the loss elsewhere could be much lower. Modern deep networks turn out to have surprisingly few bad local minima — most flat-bottomed valleys are roughly as good as the best one — but in lower-dimensional problems, they bite.</p>
</div>
<div>
<span class="lab">Saddle point</span>
<h4>Up in some directions, down in others.</h4>
<p>The gradient is zero but the point is not a minimum: it's a min in one direction and a max in another. In high dimensions these vastly outnumber local minima. Vanilla gradient descent slows down enormously here. Optimizers with momentum (Chapter 5) cut through them.</p>
</div>
<div>
<span class="lab">Plateau</span>
<h4>The slow flatland.</h4>
<p>A wide region where the gradient is tiny but nonzero. Descent technically works but each step makes negligible progress. You can sit on one of these for thousands of steps watching the loss not move. Adaptive optimizers help by boosting the effective step size in flat directions.</p>
</div>
</div>
<div class="insight reveal">Initialization is not a detail. With non-convex loss surfaces, where you start determines which basin you find — and a hundred runs of the same algorithm from different starts can produce a hundred different "trained" models.</div>
</section>
<div class="divider"></div>
<!-- ============ SECTION 3: TYPES ============ -->
<section id="s3">
<div class="section-num">Chapter 03</div>
<h2 class="reveal">Three flavors of descent.</h2>
<p class="lede reveal">The gradient is an average over your training data. <em class="term">How much</em> of that data you average is a choice — and the choice has a name.</p>
<p class="wide reveal">Recall that the loss <span class="mono">L</span> is summed (or averaged) over <em>all</em> the training examples. To compute the <em>true</em> gradient, you'd need to evaluate every single example, every single step. For a dataset with a million images, that means a million forward passes per parameter update — far too slow. So in practice, we approximate.</p>
<p class="wide reveal">There are three standard ways to do this, and they trade off speed against stability. Below, each demo shows the same toy problem from the same starting point — but uses a different subset of the data per update.</p>
<div class="gd-grid">
<div class="gd-card reveal">
<h3>3A · Batch</h3>
<div class="tag">USES ENTIRE DATASET</div>
<canvas id="s3-batch"></canvas>
<div class="desc">Every step uses every sample. The gradient is the true expected gradient — the path is smooth. The price is one full pass through the data per single update.</div>
<div style="margin-top:14px"><button class="btn" data-restart-for="s3-batch">↺ Restart</button> <span style="color:var(--muted);font-size:11px;margin-left:8px">click canvas to reposition θ</span></div>
</div>
<div class="gd-card reveal">
<h3>3B · Stochastic</h3>
<div class="tag">ONE SAMPLE PER STEP</div>
<canvas id="s3-sgd"></canvas>
<div class="desc">Each step uses a single random sample. Updates are cheap, frequent, and noisy. The noise can actually help — it shakes the parameters out of shallow local minima.</div>
<div style="margin-top:14px"><button class="btn" data-restart-for="s3-sgd">↺ Restart</button> <span style="color:var(--muted);font-size:11px;margin-left:8px">click canvas to reposition θ</span></div>
</div>
<div class="gd-card reveal">
<h3>3C · Mini-Batch</h3>
<div class="tag">A HANDFUL PER STEP</div>
<canvas id="s3-mini"></canvas>
<div class="desc">Average over 8, 32, 256 samples. Stable enough to be reliable, small enough to update often. This is what virtually all modern deep learning uses.</div>
<div style="margin-top:14px"><button class="btn" data-restart-for="s3-mini">↺ Restart</button> <span style="color:var(--muted);font-size:11px;margin-left:8px">click canvas to reposition θ</span></div>
</div>
</div>
<div class="callout cyan reveal">
<span class="tag">In practice</span>
<p>"SGD" in modern code almost always means <em>mini-batch SGD</em>. Pure one-sample stochastic descent is rarely used today — it doesn't take advantage of GPU parallelism. Typical batch sizes range from 32 (small models, limited memory) to thousands (large language models trained on clusters).</p>
</div>
<table class="compare reveal" id="s3-compare">
<thead>
<tr><th>Property</th><th>Batch</th><th>Stochastic</th><th>Mini-Batch</th></tr>
</thead>
<tbody>
<tr>
<td>Update speed</td>
<td><span class="bar" data-w="20"></span> slow</td>
<td><span class="bar rose" data-w="95"></span> very fast</td>
<td><span class="bar amber" data-w="70"></span> fast</td>
</tr>
<tr>
<td>Gradient noise</td>
<td><span class="bar" data-w="8"></span> very low</td>
<td><span class="bar rose" data-w="92"></span> very high</td>
<td><span class="bar amber" data-w="40"></span> moderate</td>
</tr>
<tr>
<td>Memory cost</td>
<td><span class="bar rose" data-w="90"></span> high</td>
<td><span class="bar" data-w="10"></span> minimal</td>
<td><span class="bar amber" data-w="35"></span> low</td>
</tr>
<tr>
<td>Hardware fit (GPU)</td>
<td><span class="bar amber" data-w="60"></span> ok</td>
<td><span class="bar rose" data-w="20"></span> poor</td>
<td><span class="bar" data-w="95"></span> excellent</td>
</tr>
<tr>
<td>Convergence quality</td>
<td><span class="bar" data-w="85"></span> stable</td>
<td><span class="bar amber" data-w="55"></span> oscillates</td>
<td><span class="bar" data-w="80"></span> stable</td>
</tr>
</tbody>
</table>
<div class="insight reveal">Stochastic gradient descent works not despite its noise but partly <em>because</em> of it. The noise smooths the loss landscape implicitly, helping the optimizer escape narrow traps that exact gradient descent would settle into.</div>
</section>
<div class="divider"></div>
<!-- ============ SECTION 4: LR ============ -->
<section id="s4">
<div class="section-num">Chapter 04</div>
<h2 class="reveal">The most important hyperparameter.</h2>
<p class="lede reveal">The learning rate <em class="term">α</em> is a single number that decides whether your model trains in five minutes or never trains at all.</p>
<div class="defbox reveal">
<span class="label">Definition</span>
<h4>Learning rate (α)</h4>
<p>A positive scalar that multiplies the gradient before subtracting. It controls how big each step is. Too small and training never finishes. Too large and the parameters fly off to infinity. The sweet spot depends on the curvature of the loss and changes as training progresses.</p>
</div>
<p class="wide reveal">If the gradient is "which way is down," the learning rate is "how far do I commit to going that way before re-checking?" A small <span class="mono">α</span> is timid — it confirms the slope at every centimeter. A large <span class="mono">α</span> is brave — it takes a kilometer-long stride and hopes the ground beneath it hasn't curved upward. Drag the slider below to feel both failure modes.</p>
<div class="lr-wrap reveal">
<div class="canvas-frame">
<span class="label">FIG. 4.1 · DESCENT VS. LEARNING RATE</span>
<canvas id="s4-canvas" class="lr-canvas"></canvas>
</div>
<div class="slider-block">
<div class="lr-label">Learning rate · drag to adjust</div>
<div class="lr-value" id="s4-value">α = 0.300</div>
<div class="lr-state ok" id="s4-state">OPTIMAL · CONVERGES SMOOTHLY</div>
<input type="range" class="slider" id="s4-slider" min="0.005" max="2.05" step="0.005" value="0.3" />
<div class="slider-ticks">
<span>0.005</span>
<span>0.5</span>
<span>1.0</span>
<span>1.5</span>
<span>2.0</span>
</div>
<p style="margin-top:28px;font-size:15px;color:#888888">Too small: each step makes negligible progress and training never finishes. Too large: every step overshoots the minimum, accumulating energy until the parameters fly off to infinity. The "right" value depends on the curvature of the loss — and good optimizers adapt it automatically.</p>
</div>
</div>
<div class="text-two reveal" style="margin-top:48px">
<div>
<span class="lab">Symptom 1 · loss is NaN</span>
<h4>You diverged.</h4>
<p>Your learning rate is too large. The first sign is usually a loss that explodes after a few steps, then becomes <span class="mono">NaN</span> (not-a-number). The fix is mechanical: cut the learning rate in half and retry.</p>
</div>
<div>
<span class="lab">Symptom 2 · loss won't move</span>
<h4>You're too cautious.</h4>
<p>If the loss plateaus immediately and never falls, your learning rate may be too small for the scale of your gradients — or you're trapped on a saddle/plateau. Try 10× larger and watch what happens.</p>
</div>
</div>
<div class="callout cyan reveal">
<span class="tag">In practice · LR schedules</span>
<p>Modern training rarely uses a single fixed learning rate. Instead, an <em>LR schedule</em> starts large and gradually decreases — taking big confident strides early when far from the optimum, then small careful ones near the end. Common schedules: <em class="term">cosine annealing</em>, <em class="term">step decay</em>, <em class="term">warmup followed by decay</em>.</p>
</div>
</section>
<div class="divider"></div>
<!-- ============ SECTION 5: OPTIMIZERS ============ -->
<section id="s5">
<div class="section-num">Chapter 05</div>
<h2 class="reveal">Smarter steps.</h2>
<p class="lede reveal">Plain gradient descent has one knob. Modern optimizers add memory: they remember the velocity of past steps and adapt the learning rate per parameter.</p>
<p class="wide reveal">Vanilla SGD treats every step independently — it knows the slope right here, right now, and that's it. But the past is full of useful information. If the gradient has pointed in roughly the same direction for the last fifty steps, it probably will again. If a particular parameter has had wildly varying gradients, maybe we should be more cautious with it. The three optimizers below — each a small modification to the basic update rule — embody these ideas.</p>
<div class="opt-wrap reveal">
<div class="canvas-frame opt-race">
<span class="label">FIG. 5.1 · OPTIMIZER RACE</span>
<div class="opt-legend">
<div style="color:#888888"><span class="dot" style="background:#888888;color:#888888"></span>SGD + Momentum</div>
<div style="color:#f59e0b"><span class="dot" style="background:#f59e0b;color:#f59e0b"></span>RMSProp</div>
<div style="color:#06b6d4"><span class="dot" style="background:#06b6d4;color:#06b6d4"></span>Adam</div>
</div>
<canvas id="s5-race"></canvas>
<div class="controls">
<button class="btn" id="s5-restart">↺ Restart Race</button>
</div>
</div>
<div class="canvas-frame opt-chart">
<span class="label">FIG. 5.2 · LOSS CURVE</span>
<canvas id="s5-chart"></canvas>
</div>
</div>
<p class="wide reveal" style="margin-top:24px">The track above is a <em class="term">ravine</em>: steep in one direction, gently sloped in the other. Vanilla SGD bounces back and forth across the steep walls while making slow progress down the gentle valley. Every modern optimizer is, in part, a fix for this scenario.</p>
<div class="opt-eqs">
<div class="opt-eq-card reveal">
<div class="name" style="color:#888888">SGD + Momentum</div>
<div id="eq-sgd-mom"></div>
<p style="font-size:13px;color:#888888;margin-top:12px;line-height:1.5">Velocity accumulates — like a ball rolling, it picks up speed in consistent directions and dampens noise. Typical β ≈ 0.9.</p>
</div>
<div class="opt-eq-card reveal">
<div class="name" style="color:#f59e0b">RMSProp</div>
<div id="eq-rmsprop"></div>
<p style="font-size:13px;color:#888888;margin-top:12px;line-height:1.5">Per-parameter step size: divide by recent squared gradient. Steep dimensions slow down, flat ones speed up. Solves the ravine problem.</p>
</div>
<div class="opt-eq-card reveal">
<div class="name" style="color:#06b6d4">Adam</div>
<div id="eq-adam"></div>
<p style="font-size:13px;color:#888888;margin-top:12px;line-height:1.5">Momentum plus RMSProp, plus bias correction. The default optimizer of modern deep learning since ~2015.</p>
</div>
</div>
<div class="callout cyan reveal">
<span class="tag">When to use what</span>
<p><strong style="color:var(--text)">Adam</strong> is the safe default — start here for almost any problem. <strong style="color:var(--text)">SGD + Momentum</strong> with a tuned learning rate schedule can sometimes generalize better, especially for image classification (ResNets are still often trained this way). <strong style="color:var(--text)">RMSProp</strong> shows up in reinforcement learning and recurrent networks. <strong style="color:var(--text)">AdamW</strong>, a variant of Adam with corrected weight decay, is now standard for training large language models.</p>
</div>
</section>
<div class="divider"></div>
<!-- ============ SECTION 6: NN ARCHITECTURE ============ -->
<section id="s6">
<div class="section-num">Chapter 06</div>
<h2 class="reveal">A network is a graph of multiplications.</h2>
<p class="lede reveal">Before we can compute a gradient, we need a thing to take the gradient of. A neural network is a chain of weighted sums and nonlinearities — and crucially, it is differentiable end-to-end.</p>
<div class="defbox reveal">
<span class="label">Definition</span>
<h4>Neural network</h4>
<p>A parameterized function built by composing many simple operations: multiply by a matrix of <em>weights</em>, add a <em>bias</em>, apply a nonlinear <em>activation function</em>, repeat. The parameters are tuned by gradient descent on a loss measured against training data.</p>
</div>
<p class="wide reveal">Think of every node as a small calculator. Each one takes a bunch of numbers in, multiplies each by a learned weight, adds them up, adds a learned bias, and runs the result through a squishing function. Then it ships that single number to the next layer. Repeat this through three or four or three hundred layers, and you have a function flexible enough to recognize cats, translate French, or predict protein structure.</p>
<div class="nn-wrap reveal">
<svg id="nn-svg"></svg>
</div>
<div class="nn-controls reveal">
<button class="btn primary" id="nn-forward">▶ Forward Pass</button>
<button class="btn" id="nn-backward">◀ Backpropagate</button>
<button class="btn" id="nn-update">↻ Update Weights</button>
<button class="btn" id="nn-reset">Reset</button>
<span class="status" id="nn-status">STATUS: <span>READY</span></span>
</div>
<div class="text-two reveal" style="margin-top:36px">
<div>
<span class="lab">Nodes</span>
<h4>Neurons (a.k.a. units)</h4>
<p>Each circle holds a single number, called an <em class="term">activation</em>. Input nodes hold values from the data — pixels, words, sensor readings. Every other node's activation is computed from the layer behind it.</p>
</div>
<div>
<span class="lab">Edges</span>
<h4>Weights</h4>
<p>Each line is a single learnable number. Cyan = positive (the source amplifies the target), rose = negative (the source suppresses it). Thickness shows magnitude. These are the parameters gradient descent is updating.</p>
</div>
<div>
<span class="lab">Layers</span>
<h4>Stacks of neurons</h4>
<p>Input → hidden → hidden → output. Each layer computes a new representation of the data. Early layers tend to capture simple features (edges, syllables); deep layers capture abstract ones (faces, sentiment).</p>
</div>
<div>
<span class="lab">Depth</span>
<h4>Why "deep" learning?</h4>
<p>"Deep" just means many layers. The depth matters: each additional nonlinear layer dramatically increases the family of functions the network can represent. A two-layer network can in theory approximate anything, but a ten-layer network does it efficiently.</p>
</div>
</div>
<p class="reveal wide" style="margin-top:36px">Use the buttons above the diagram to walk through the three phases of a training step: a <em class="term">forward pass</em> (Chapter 7), <em class="term">backpropagation</em> (Chapter 8), and a <em class="term">weight update</em>. Then come back to this network as you read the next two chapters.</p>
</section>
<div class="divider"></div>
<!-- ============ SECTION 7: FORWARD PASS ============ -->
<section id="s7">
<div class="section-num">Chapter 07</div>
<h2 class="reveal">Forward.</h2>
<p class="lede reveal">For each layer: multiply, sum, squash. Repeat until you reach the output. The final number is your prediction; the difference from the true label is your loss.</p>
<p class="wide reveal">A <span class="gloss" data-label="forward pass" data-def="One left-to-right evaluation of the network: input data is transformed layer by layer into a prediction at the output, and then compared against the true label to compute the loss.">forward pass</span> is just the network "running" — turning input into prediction. Everything you've heard about neural networks "doing inference" or "making predictions" refers to a forward pass. There is no training involved; the weights are frozen and only the activations change as data flows through.</p>
<div class="two-col reveal">
<div>
<p>At every neuron <span class="mono">j</span> in layer <span class="mono">ℓ</span> we combine the previous activations with incoming weights, then apply a nonlinearity. Hover any symbol — instances across all equations on the page light up together.</p>
<div class="eq" id="eq-forward-z"><span class="corner-label">Pre-activation</span></div>
<div class="eq" id="eq-forward-a"><span class="corner-label">Activation</span></div>
<p>Two equations cover the entire forward pass. Without the nonlinearity <em class="term">σ</em>, stacking layers would collapse into a single matrix multiplication. <em class="term">σ</em> is what makes deep networks deep.</p>
<div class="eq" id="eq-loss"><span class="corner-label">Loss · Mean-Squared Error</span></div>
<p style="font-size:14px;color:#888888;margin-top:10px">For classification, swap mean-squared error for <em class="term">cross-entropy</em>, which compares probability distributions instead of raw numbers. The mechanics of backprop are identical.</p>
</div>
<div>
<div class="card">
<span class="corner">FIG. 7.1 · ACTIVATIONS</span>
<h3 style="font-size:18px;margin-top:14px">σ(z) — three common choices</h3>
<svg viewBox="0 0 360 220" style="width:100%;height:auto;margin-top:16px">
<line x1="30" y1="180" x2="340" y2="180" stroke="#3a3a3a" />
<line x1="180" y1="20" x2="180" y2="200" stroke="#3a3a3a" />
<text x="335" y="195" font-family="JetBrains Mono" font-size="9" fill="#737373">z</text>
<text x="186" y="20" font-family="JetBrains Mono" font-size="9" fill="#737373">σ</text>
<path d="M 30 175 C 100 175 120 110 180 100 C 240 90 260 25 330 25" fill="none" stroke="#06b6d4" stroke-width="1.8" />
<text x="40" y="40" font-family="JetBrains Mono" font-size="10" fill="#06b6d4">sigmoid</text>
<path d="M 30 180 L 180 180 L 330 30" fill="none" stroke="#f59e0b" stroke-width="1.8" />
<text x="40" y="55" font-family="JetBrains Mono" font-size="10" fill="#f59e0b">ReLU</text>
<path d="M 30 170 C 100 170 130 168 180 100 C 230 32 260 30 330 30" fill="none" stroke="#f43f5e" stroke-width="1.8" />
<text x="40" y="70" font-family="JetBrains Mono" font-size="10" fill="#f43f5e">tanh</text>
</svg>
<p style="font-size:13px;color:#888888;margin-top:14px;line-height:1.5"><strong style="color:#06b6d4">Sigmoid</strong> squashes to (0, 1) — historical, now mostly used at outputs of binary classifiers. <strong style="color:#f59e0b">ReLU</strong> passes positives through, zeros negatives — fast, simple, and the workhorse of modern deep nets. <strong style="color:#f43f5e">Tanh</strong> is sigmoid centered at zero, ranges (−1, 1).</p>
</div>
<div class="card" style="margin-top:20px">
<span class="corner">FIG. 7.2 · TRY IT</span>
<h3 style="font-size:18px;margin-top:14px">Propagate values</h3>
<p style="margin-top:10px;font-size:14px;color:#888888">Hit <em class="term">Forward Pass</em> on the network above and watch values flow left to right through each layer.</p>
<p style="font-size:14px;color:#888888">Activations are written inside the nodes. Edge color shows the sign of the weight; thickness shows magnitude.</p>
</div>
</div>
</div>
<div class="insight reveal">A forward pass is the network <em>guessing</em>. Until we measure the guess against the truth, the network has no idea whether it was right or wrong — and no way to improve.</div>
</section>
<div class="divider"></div>
<!-- ============ SECTION 8: BACKPROP ============ -->
<section id="s8">
<div class="section-num">Chapter 08</div>
<h2 class="reveal">Backward.</h2>
<p class="lede reveal">Backpropagation is the chain rule, applied repeatedly. It tells us — for every weight in the network, no matter how deep — how a tiny change in that weight would change the final loss.</p>
<div class="defbox reveal">
<span class="label">Definition</span>
<h4>Backpropagation</h4>
<p>An efficient algorithm for computing <span class="mono">∂L/∂w</span> for every weight <span class="mono">w</span> in the network. Works by applying the chain rule layer by layer, starting from the loss and propagating an error signal backward. Time complexity: roughly the same as a forward pass.</p>
</div>
<p class="wide reveal">Forward pass tells the network <em>what</em> it predicted. Backprop tells it <em>who is responsible</em> for the prediction being wrong. A weight in layer 1 didn't talk to the output directly — it influenced an activation in layer 2, which influenced layer 3, which eventually influenced the loss. Backprop walks that path backward and computes the contribution of every weight along the way.</p>
<p class="wide reveal">Without backprop, training a network of more than a couple of layers is intractable. With it, training a network of <em>hundreds</em> of layers is routine. The algorithm was popularized in the 1980s and is the single most important reason deep learning works at all.</p>
<h3 class="reveal" style="margin-top:48px;color:var(--cyan);font-family:var(--mono);font-size:13px;letter-spacing:0.2em;text-transform:uppercase">Phase 1 · The chain rule</h3>
<p class="wide reveal">The loss depends on <span class="mono">w¹</span> only indirectly — through a series of intermediate quantities. To compute <span class="mono">∂L/∂w¹</span> we multiply the local derivatives along the path. Click <em class="term">Next</em> to build it term by term.</p>
<div class="chain-builder reveal" id="chain-builder">
<span class="corner-label">∂L / ∂w¹ · Step Builder</span>
<span class="cb-step-counter">STEP 0 / 6</span>
<div class="cb-eq"></div>
<div class="cb-hint">Click NEXT to build the chain rule term by term.</div>
<div class="cb-controls">
<button class="btn" data-cb="prev">◀ Prev</button>
<button class="btn primary" data-cb="next">Next ▶</button>
<button class="btn" data-cb="auto">▶ Auto</button>
<button class="btn" data-cb="reset">Reset</button>
</div>
</div>
<div class="callout cyan reveal">
<span class="tag">Read the chain rule</span>
<p>Each fraction asks: "if this thing changes by a tiny amount, by how much does the next thing change?" Multiply them all and you get the full effect of <span class="mono">w¹</span> on the loss. The whole of backprop is just bookkeeping for this product, done efficiently across millions of weights.</p>
</div>
<h3 class="reveal" style="margin-top:64px;color:var(--rose);font-family:var(--mono);font-size:13px;letter-spacing:0.2em;text-transform:uppercase">Phase 2 · Error signal propagation</h3>
<p class="wide reveal">Click <em class="term">Backpropagate</em> on the network above. Pulses flow <em>right to left</em>, in rose this time. At every node a delta <span class="mono">δ = ∂L/∂z</span> is computed by combining the deltas of the next layer with the connecting weights.</p>
<div class="eq reveal" id="eq-delta"><span class="corner-label">Backpropagated Error Signal</span></div>
<p class="wide reveal" style="font-size:15px;color:#b0b0b0">The recurrence above is the heart of backprop. Read it as: <em>"my error is the weighted sum of my downstream neighbors' errors, scaled by the slope of my activation function."</em> If σ′ is zero — for example, in a ReLU that has output 0 — the signal can't flow through, and any weights upstream of this neuron get no gradient this step. (This is the source of the famous "dying ReLU" problem.)</p>
<h3 class="reveal" style="margin-top:64px;color:var(--amber);font-family:var(--mono);font-size:13px;letter-spacing:0.2em;text-transform:uppercase">Phase 3 · Weight update</h3>
<p class="wide reveal">Once every node knows its <span class="mono">δ</span>, the gradient for any weight is a single product: the upstream activation times the downstream error. Click <em class="term">Update Weights</em> on the network to see the cyan ↑ / rose ↓ arrows — each edge nudges by an amount proportional to that product, and the picture of the network itself shifts.</p>
<div class="eq reveal" id="eq-weight-grad"><span class="corner-label">Per-Weight Gradient</span></div>
<div class="eq reveal" id="eq-weight-update"><span class="corner-label">Weight Update Rule</span></div>
<p class="wide reveal" style="font-size:15px;color:#b0b0b0">This is where the chapter loops back to the start. The gradient for a weight is <span class="mono" style="color:#ffffff">a</span> · <span class="mono" style="color:var(--rose)">δ</span>. The weight is updated by subtracting <span class="mono">α</span> times that gradient. The form of the rule is identical to the very first equation in Chapter 1: <span class="mono">θ ← θ − α · ∇L</span>. Backprop is the procedure that gives us <span class="mono">∇L</span>; gradient descent is what does the actual stepping.</p>
<div class="intuition reveal" style="margin-top:48px">
A forward pass asks: <em>given these weights, what does the network predict?</em> A backward pass asks: <em>given the error, who is to blame, and by how much?</em> Together they form the entire engine of modern deep learning.
</div>
</section>
<div class="divider"></div>
<!-- ============ SECTION 9: TRAINING LOOP ============ -->
<section id="s9">
<div class="section-num">Chapter 09</div>
<h2 class="reveal">The loop.</h2>
<p class="lede reveal">Now put it together. Training is just this cycle, run thousands of times, with the loss falling a little each pass.</p>
<p class="wide reveal">Every piece we've assembled — forward, backward, update — happens once per <em class="term">training step</em>. One full pass through the training dataset is called an <em class="term">epoch</em>. Most models train for dozens or hundreds of epochs. The "training" of a model is literally just this loop, running for hours or weeks, while the loss curve descends.</p>
<div class="loop-wrap reveal">
<div class="loop-diagram">
<span class="loop-readout" style="position:absolute;top:14px;left:16px">
EPOCH<span class="val" id="s9-epoch">0</span>
</span>
<span class="loop-readout" style="position:absolute;top:14px;right:16px;text-align:right">
LOSS<span class="val" id="s9-loss" style="color:#f59e0b">1.8000</span>
</span>
<canvas id="s9-loop"></canvas>
</div>
<div class="loop-chart">
<span class="loop-readout" style="position:absolute;top:14px;left:16px">LIVE TRAINING</span>
<canvas id="s9-chart"></canvas>
</div>
</div>
<p class="reveal wide" style="margin-top:32px">A pulse circles the loop: <em class="term">DATA</em> feeds the network, <em class="term">FORWARD</em> generates a prediction, <em class="term">LOSS</em> measures how wrong it was, <em class="term">BACKPROP</em> attributes blame, <em class="term">UPDATE</em> nudges weights. The loss chart on the right is the chart you stare at for hours when you train a real model.</p>
<div class="reveal" style="display:flex;gap:14px;align-items:center;flex-wrap:wrap;margin-top:24px">
<button class="btn primary" id="s9-pause">⏸ Pause</button>
<button class="btn" id="s9-restart">↺ Restart</button>
<div style="display:flex;align-items:center;gap:12px;margin-left:8px">
<span style="font-family:var(--mono);font-size:10px;letter-spacing:0.2em;text-transform:uppercase;color:var(--muted)">Speed</span>
<input type="range" class="slider" id="s9-speed" min="0.1" max="3" step="0.05" value="1" style="width:180px" />
</div>
</div>
<div class="text-two reveal" style="margin-top:48px">
<div>
<span class="lab">Step vs. epoch</span>
<h4>Two different counters.</h4>
<p>A <em class="term">step</em> is one weight update — one trip around the loop. An <em class="term">epoch</em> is one full pass through the training data. If your dataset has 50,000 samples and your batch size is 32, one epoch is ~1,560 steps.</p>
</div>
<div>
<span class="lab">Training vs. validation loss</span>
<h4>Two curves, not one.</h4>
<p>In practice you also track loss on a held-out <em class="term">validation set</em>. If training loss keeps falling but validation loss starts rising, the model is <em class="term">overfitting</em> — memorizing the training set instead of learning the underlying pattern. Time to stop, or to regularize.</p>
</div>
<div>
<span class="lab">When to stop</span>
<h4>Convergence and patience.</h4>
<p>Sometimes you train for a fixed number of epochs. More often you use <em class="term">early stopping</em>: keep training while validation loss improves; stop when it plateaus or worsens for a few epochs in a row.</p>
</div>
<div>
<span class="lab">What "training" actually is</span>
<h4>Just this loop, longer.</h4>
<p>GPT-class models train this same loop for trillions of steps on thousands of GPUs. The algorithm doesn't change; only the scale of data, model, and compute. The fundamental operation is still: forward, backward, update.</p>
</div>
</div>
</section>
<div class="divider"></div>
<!-- ============ SECTION 10: PLAYGROUND ============ -->
<section id="s10">
<div class="section-num">Chapter 10</div>
<h2 class="reveal">Your turn.</h2>
<p class="lede reveal">Two clusters of points, a single line separating them, and a loss to minimize. Pick an optimizer, set a learning rate, and watch the line learn.</p>
<p class="wide reveal">This is the simplest non-trivial machine learning problem: take a bunch of 2D points belonging to two classes, and find a line that separates them. The model has just three numbers — two weights and a bias — and the loss is binary cross-entropy. Train it with the optimizer of your choice and watch the decision boundary swing into place in real time.</p>
<div class="play-wrap reveal">
<div class="canvas-frame play-canvas">
<span class="label">FIG. 10.1 · 2D BINARY CLASSIFICATION</span>
<canvas id="s10-canvas"></canvas>
</div>
<div class="play-controls">
<div class="ctl-group">
<label>Optimizer</label>
<select id="s10-opt">
<option value="sgd">SGD</option>
<option value="mom">SGD + Momentum</option>
<option value="adam" selected>Adam</option>
</select>
</div>
<div class="ctl-group">
<label id="s10-lr-label">α = 0.300</label>
<input type="range" class="slider" id="s10-lr" min="0.005" max="1.5" step="0.005" value="0.3" />
</div>
<div style="display:flex;gap:10px;flex-wrap:wrap">
<button class="btn primary" id="s10-step">Train Step</button>
<button class="btn" id="s10-auto">Auto Train</button>
<button class="btn" id="s10-reset">Reset</button>
</div>
<div class="readout">
<div><span>Steps</span><strong id="s10-step-count">0</strong></div>
<div><span>Loss</span><strong id="s10-loss-val">–</strong></div>
</div>
<div class="canvas-frame" style="aspect-ratio:auto">
<canvas id="s10-loss" class="play-loss"></canvas>
</div>
</div>
</div>
<div class="text-two reveal" style="margin-top:48px">
<div>
<span class="lab">Try this</span>
<h4>Experiments</h4>
<p>1. Adam at α = 0.3 — should snap to a clean separation in seconds.<br/>2. Plain SGD at α = 0.01 — feel how slowly first-order methods can crawl.<br/>3. SGD at α = 1.4 — bordering on unstable; watch the line jitter.<br/>4. Click Reset to regenerate the clusters and try again.</p>
</div>
<div>
<span class="lab">What you're seeing</span>
<h4>Logistic regression, alive.</h4>
<p>The colored background shows the model's predicted probability across the plane: full cyan = "definitely class 1," full rose = "definitely class 0," in between = uncertain. The dashed white line is the decision boundary — where the model thinks p = 0.5. A deep network is just this kind of classifier, with many more parameters and a curvy boundary instead of a straight one.</p>
</div>
</div>
<p class="reveal wide" style="margin-top:48px;color:#888888;font-size:15px">The line you're watching is exactly what a single perceptron learns; a deep network is just hundreds of these stacked, with backprop attributing error across every layer. Once you understand this picture, everything else in deep learning is a generalization of it.</p>
<div class="insight reveal">From rolling a marble down a hill to training a billion-parameter language model is a continuous chain of ideas. None of them is hard on its own. The art is putting them together at scale.</div>
</section>
<footer>
END · BUILT WITH CANVAS, SVG, MATH · THE MATHEMATICS OF LEARNING
</footer>
<!-- Scripts -->
<script src="js/utils.js"></script>
<script src="js/equations.js"></script>
<script src="js/hero.js"></script>
<script src="js/s1-gd-intro.js"></script>
<script src="js/s2-landscape.js"></script>
<script src="js/s3-types.js"></script>
<script src="js/s4-lr.js"></script>
<script src="js/s5-optimizers.js"></script>
<script src="js/s6-nn.js"></script>
<script src="js/s9-trainingloop.js"></script>
<script src="js/s10-playground.js"></script>
</body>
</html>