Personal study of Andrej Karpathy's micrograd — a tiny autograd engine built from scratch.Personal notes and understanding written inline as comments.
Micrograd is a scalar-valued autograd engine that implements backpropagation over a dynamically built computational graph. It's the core mechanism behind every deep learning framework (PyTorch, TensorFlow) — just stripped down to ~100 lines of Python.
- Value class — how every scalar operation builds a computation graph
- Forward pass — how values flow through operations (+, *, tanh, relu)
- Backward pass — how gradients flow back through the graph via chain rule
- Backpropagation — manual derivation of gradients for each operation
- MLP architecture — Neuron → Layer → MLP built on top of the autograd engine
- Training loop — forward pass, loss computation, zero_grad, backward, update
Every operation (add, multiply, tanh) not only computes the output but also stores how to compute its own gradient. When .backward() is called, it walks the graph in reverse topological order and applies chain rule automatically. This is exactly how PyTorch works internally.
The main notebook (micrograd_walkthrough.ipynb) contains:
- Full implementation walkthrough
- Inline personal notes and understanding at each step
- Visualization of computational graphs
- MLP training example
- Python
- Jupyter Notebook
- Following: Karpathy's Zero to Hero — Lecture 1
This repo is part of my study series following Andrej Karpathy's Neural Networks: Zero to Hero course.
| Repo | Topic | Status |
|---|---|---|
| micrograd-study | Autograd engine, backprop | ✅ Done |
| makemore-study | Character-level LM, bigram model | 🔄 In progress |