Who this is for. You can program. You remember that a derivative is a slope. That is the entire prerequisite. No linear algebra, no probability, no optimization is assumed — not "assumed but rusty," not assumed. If you have met this material before, read the bold sentences and skip; nothing here is load-bearing twice.
Why this file exists. The main document is dense on purpose. A reference that unpacks every idea from first principles is unusable as a reference. So the unpacking happens here, once, and the main document is free to move fast. When a chapter there says "Prereq: primer §A.5", it means the mathematics is in this file and it will not be re-explained.
How every topic is structured. Plain words first, with no notation at all. Then the smallest concrete example that shows the idea, with real numbers worked through by hand. Then the notation, introduced as shorthand for the thing you already understand. Then one line saying where it shows up in the main document.
If you find yourself reading notation you cannot say out loud in English, stop and go back. That feeling is the file failing you, not you failing the file.
Sections in this file are lettered — §A.1 through §A.11 for the mathematics, §B.1 through §B.5 for the learning prerequisites — so that a reference to "primer §A.3" can never be confused with §3.x of the main document.
A vector is a displacement: a movement of a certain distance in a certain direction. "Twelve metres north-east, level" is a vector. It is a physical thing that exists whether or not anyone writes it down.
Three numbers are not a vector. Three numbers are what you get when you agree on three reference directions and ask "how far along each?" Change the reference directions and the same physical displacement gets different numbers. The displacement did not change. The description did.
This is not pedantry. It is the origin of an entire category of robotics bug, and it is worth being annoying about.
Stand at a crossroads. A traffic cone sits 3 m east and 4 m north of you.
Agree that direction one is east and direction two is north. Then the cone is at (3, 4).
Now turn 90° to your left and re-describe the world from where you are facing. Your new direction
one is north; your new direction two is west. The cone has not moved. But now you must ask: how far
north? 4 m. How far west? It is 3 m east, so it is −3 m west. The cone is at (4, -3).
Same cone. Same universe. (3,4) and (4,-3). If you hand someone the numbers without saying
which reference directions you used, you have handed them nothing. You have handed them something
worse than nothing, because it looks like information.
The reference directions are called a basis. The numbers are called coordinates with respect to that basis. A basis plus an origin is a frame.
Write a vector in bold:
which reads: the displacement is $p_1$ steps along the first reference direction, plus $p_2$ along
the second, plus $p_3$ along the third. The set of all three-coordinate vectors is written
The main document writes
Take two vectors and ask how much do they point the same way? Multiply matching coordinates and add.
For
Two facts fall out, and everything later leans on both:
-
A vector dotted with itself gives its squared length.
$(3,4)\cdot(3,4) = 9 + 16 = 25$ , so the length is 5. The familiar Pythagoras, written as a dot product. -
A dot product of zero means perpendicular.
$(3,4)\cdot(4,-3) = 12 - 12 = 0$ . Those two point at exactly 90° to each other. Any time you see a dot product set to zero, someone is saying "make these two things perpendicular," and it is usually the whole idea of the algorithm.
In symbols,
The standard confusion. Programmers meet vectors as float[3] and conclude a vector is an
array. That intuition survives right up until two subsystems disagree about axis conventions, and
then it costs a week. [1.2, 0.3, 0.0] is not a position. [1.2, 0.3, 0.0] in the rear-axle
frame, x-forward, in metres is a position.
Where this shows up. §1.1 (frames and the
A matrix is a machine that takes a vector in and gives a vector out, with one restriction: doubling the input doubles the output, and feeding it a sum gives you the sum of the outputs. That restriction is what "linear" means. Rotations, scalings, projections, and shears are all linear. Squaring something is not.
The most useful mental image: a matrix's columns tell you where the basis directions land. That is all a matrix is — a record of what happened to east, north, and up.
Take the machine that throws away the third coordinate:
Feed it the point
Sit with that. An entire line of input points collapsed to one output point. Every point on the
vertical line through
That collapsed set — every input the machine sends to zero — is the null space. Here it is the
third axis:
The set of outputs the machine can actually reach is the column space. Here it is all of the 2D plane; nothing is unreachable.
This example is a camera. A camera takes a 3D world and produces a 2D image, and depth is what lands in the null space. §0.1 opens the whole document with the observation that perception is inverting a many-to-one map. The null space is where the many-to-one lives, and it is why a single photograph cannot tell you how far away anything is.
The second column is exactly twice the first, so both columns point the same way. Everything this
machine produces lands on a single line — the line through
So
-
Column space
$\mathrm{col}(\mathbf{A})$ — everything reachable. -
Null space
$\mathrm{null}(\mathbf{A})$ — everything annihilated. Where lost information lives. -
Rank
$r$ — the dimension of the column space, i.e. how many genuinely independent output directions exist. For$\mathbf{B}$ above,$r = 1$ , not 2. - The rank–nullity bookkeeping:
$\dim\mathrm{null}(\mathbf{A}) = n - r$ . Inputs are conserved. Every dimension not surviving into the output went into the null space.
Where this shows up. §1.2, which states the perception reading directly: an unobservable direction is a null space direction of a measurement Jacobian. When monocular SLAM cannot recover scale, or LiDAR odometry slides along a featureless tunnel, that is a null space with a name. §38.6 turns this into a runtime check.
Most directions, fed into a matrix, come out pointing somewhere else. The machine turns them.
An eigenvector is a direction the map does not rotate — only scales. Feed it in, and what comes out points exactly the same way (or exactly backwards), just longer or shorter. The stretch factor is the eigenvalue.
That sentence is the whole concept. Everything else is consequence.
Try the diagonal direction
Out came the same direction, four times longer. So
Try the anti-diagonal
Same direction, doubled. Eigenvalue 2.
Now try something that is neither, say
Notice the two eigenvectors are perpendicular:
A matrix is symmetric if flipping it across its diagonal changes nothing — entry
Symmetric matrices have two properties that make them the well-behaved case, and essentially every matrix you care about in estimation is symmetric:
- All eigenvalues are real numbers (no complex arithmetic).
-
The eigenvectors are mutually perpendicular. So the matrix is nothing but "stretch by
$\lambda_1$ along this axis, by$\lambda_2$ along that perpendicular axis, and so on" — a set of independent stretches along a set of right-angled axes.
Covariance matrices are always symmetric, and here is why, concretely. The entry at row
This is the payoff, and it is the picture to keep.
Eigenvectors: the two axes. Eigenvalues: 4 and 1. Take square roots — 2 and 1 — and you have the standard deviations: this estimate is uncertain by 2 m east–west and 1 m north–south. Draw it and you get an ellipse, twice as wide as it is tall. A covariance matrix is an ellipse of uncertainty, with the eigenvectors giving the axis directions and the square-rooted eigenvalues giving the radii.
Now rotate that ellipse 45°:
Its eigenvectors are
For symmetric
Read right to left in the middle expression:
Where this shows up. §1.3, which uses exactly this to estimate LiDAR surface normals: take a
point's neighbours, build their
Eigenvectors only behave for square, well-mannered matrices. The singular value decomposition is the version that works for absolutely any matrix, of any shape, degenerate or not.
The claim it makes is remarkable and worth stating plainly: every linear map, no matter how strange it looks, is exactly three things in a row — a rotation, then a stretch along the axes, then another rotation. There is nothing else a matrix can do.
What does this do? Feed it
Written as rotate-scale-rotate:
Multiply it out and check — it comes back to
The stretch factors 3 and 2 are the singular values. They are the only part that carries magnitude; the two rotations just reorient.
This is why anyone cares.
- A zero singular value means a direction got crushed to nothing — the null space of §A.2, now detectable by looking at a number. For the degenerate $\mathbf{B} = \begin{bmatrix}1&2\2&4\end{bmatrix}$ from §A.2, the singular values are 5 and 0. The 0 is the collapse, stated numerically.
-
A nearly-zero singular value is worse than a zero one. It means a direction survived, but
barely, so anything measured along it is multiplied by an enormous factor on the way back out.
This is ill-conditioning, and it is how real degeneracy appears: not as exact zero but as
$10^{-9}$ sitting next to a 5. - The condition number
$\kappa = \sigma_1/\sigma_r$ — largest over smallest — is the error amplification factor. If$\kappa = 10^6$ and your data carries 6 significant digits, your answer carries none. Not "is less accurate." None.
Every
with
A symbol collision to note now.
Where this shows up. §1.4, which lists what SVD is actually used for in vision — solving
This section is the most important one in this file. The main document leans on the Jacobian in §3.3, §13.2, §28.2, §32.4, and §36.3, and treats it in six lines. Here is the long version.
There is a ladder of five rungs, and each rung is the same idea — how much does the output move when I nudge the input — applied to progressively bigger objects.
You know this one.
Check it numerically, because this is how you will always check derivatives:
Now
At
Stack the partials into one vector and you have the gradient: at
It is not merely bookkeeping. The gradient points in the direction of steepest increase, and its
length says how steep. Here the surface climbs twice as fast if you walk along
Written
Now the output is a vector too. And this is the rung that matters.
A Jacobian is a table. Row
Work one by hand. Take a function with two inputs and two outputs — converting a range and bearing
into
Four questions, four answers:
- Wiggle
$r$ , watch$x$ :$\partial x/\partial r = \cos\theta$ - Wiggle
$\theta$ , watch$x$ :$\partial x/\partial\theta = -r\sin\theta$ - Wiggle
$r$ , watch$y$ :$\partial y/\partial r = \sin\theta$ - Wiggle
$\theta$ , watch$y$ :$\partial y/\partial\theta = r\cos\theta$
Lay them out, outputs down the side, inputs across the top:
Put in numbers. A radar reports something at
Now read it out loud, because this is the skill:
- Top-left, 0.866: push the range out by 1 m and
$x$ grows by 0.87 m. - Top-right, −10: rotate the bearing by 1 radian and
$x$ drops by 10 m. Enormous — because at 20 m range, swinging the bearing sweeps the point a long way sideways. - The second column being large is the geometry telling you something real: at long range, bearing error dominates. One milliradian of bearing error at 20 m is 2 cm; at 200 m it is 20 cm. That fact lives in the second column of this Jacobian, and it is why a camera (precise bearing, no range) and a radar (precise range, coarse bearing) fuse so well — their Jacobians are large in perpendicular directions.
You did not need a fusion algorithm to see that. You needed to read a table.
Here is the second job the Jacobian does, and it is the one that makes the whole document work.
A nonlinear function is hard. A linear one is easy — you can solve linear systems exactly. So: near your current guess, replace the nonlinear function by the Jacobian. That substitution is what linearize means. Nothing more.
Concretely, near a point
In words: where I am now, plus the table times how far I move. It is the tangent-line approximation from first-year calculus, with vectors.
Check it on the radar example. At
The truth is
Take the gradient — which is itself a function of several inputs producing several outputs — and take its Jacobian. The result is the Hessian: the matrix of second derivatives, describing curvature. It answers "how fast is the slope itself changing?"
For a bowl-shaped cost function, the Hessian is the bowl's tightness in each direction. A tight direction is one you know well; a flat direction is one you barely constrain. That should sound like the covariance ellipse of §A.3, and the resemblance is not decorative — §2.7 shows the Hessian's inverse is the estimate's covariance.
| Rung | In | Out | Object | Shape |
|---|---|---|---|---|
| Derivative | scalar | scalar | 1 number | |
| Partial | vector | scalar | 1 number | |
| Gradient | vector | scalar | ||
| Jacobian | vector | vector | ||
| Hessian | vector | scalar |
For
The standard confusion. People treat "gradient" and "Jacobian" as loose synonyms. They are the
same idea at different rungs, and mixing them up gets you a shape mismatch at best and a silently
transposed update at worst. The gradient of a scalar cost is a column; the Jacobian of a residual
vector is a matrix. §3.2 combines both in one line —
Where this shows up. Everywhere. §3.2 and §3.3 (Gauss–Newton is literally "build
An optimization problem is: here is a knob, or several; here is a number that says how bad things currently are; turn the knobs to make that number small. The knobs are parameters, the badness is the cost or objective.
Perception has one specific shape of this problem, and it appears so relentlessly that recognising it is most of the skill.
You predicted something. You measured something. They disagree. The disagreement is a residual.
Square each residual, add them all up, and turn the knobs until that total is as small as possible. That is least squares. Squaring is what makes over- and under-shooting equally bad, and what makes the mathematics tractable.
Three range measurements of a static beacon: 10.1 m, 9.8 m, 10.3 m. One knob: the true range
The residuals are
Try
It is bottoming out near 10.067 — which is the mean, as it must be. Least squares with equal weights is averaging. That is not a coincidence to file away; it is the reason least squares is the right default, and §2.7 makes it precise: minimising squared residuals is exactly maximum likelihood estimation under Gaussian noise.
Now suppose the third reading came from a sensor you trust three times less. You want it to count less. Attach a weight to each residual — bigger weight, louder vote — and minimise the weighted sum. The right weight turns out to be one over the variance: a measurement you trust twice as precisely gets four times the weight, because variance is squared uncertainty.
This is the single most important identity in the main document (§1.5 says so in those words): set the weight matrix to the inverse of the noise covariance and weighted least squares becomes maximum likelihood.
If the prediction depends nonlinearly on the knobs — and it always does, because projection and rotation are nonlinear — you cannot solve it in one shot. So you do the only thing available:
- Guess.
- Linearize around the guess: build the Jacobian of the residuals (§A.5).
- Solve the resulting linear least-squares problem for a small correction.
- Apply the correction, and go back to step 2.
Repeat until the corrections stop mattering. That loop is called Gauss–Newton, and I want to be blunt about its significance: bundle adjustment, ICP, camera calibration, factor-graph SLAM, visual-inertial odometry, and the extended Kalman filter are all this loop. They differ in what the residual is and how the problem is stored, not in what is happening.
Notice step 1. A nonlinear problem has no guarantee of a single bottom — there can be several valleys, and this loop walks downhill into whichever one it started in. Initialization is part of the algorithm, not a detail (§3.8).
-
$\arg\min_\mathbf{x}$ — "the$\mathbf{x}$ that makes the following smallest," as opposed to$\min$ , which is the smallest value. You want the knob setting, not the badness. -
$\hat{\mathbf{x}}$ — the hat means estimate, a value inferred from data rather than known. -
$\mathbf{r}_i(\mathbf{x})$ — the$i$ -th residual, a function of the knobs. -
$|\mathbf{e}|^2_{\mathbf{\Sigma}} \equiv \mathbf{e}^\top\mathbf{\Sigma}^{-1}\mathbf{e}$ — the Mahalanobis norm: squared length measured in units of standard deviation, so that an error along a direction you know precisely counts for more. Plain squared length is the special case$\mathbf{\Sigma} = \mathbf{I}$ .
Where this shows up. §3.1 states this template and tells you — correctly — that recognising it lets you read the methods section of most SLAM and calibration papers directly. §3.3 is the Gauss–Newton loop, §3.4 is what to do when it diverges, §3.5 is what to do when some of your residuals are lies, and §2.7 is the probabilistic justification for the whole arrangement.
A random variable is a quantity whose value you do not know, described by how the possibilities are spread out.
For something continuous, like a range measurement, asking "what is the probability the range is exactly 10.000000 m?" gives zero — there are infinitely many values it could be. So you ask about intervals instead: what is the probability it lands between 9.9 and 10.1? A probability density is the function you integrate over an interval to get that answer. Its height at a point is not a probability. Only area is probability.
The expectation is the long-run average — where the mass balances.
The variance is the average squared distance from that centre: how spread out the possibilities are. Squared, so that errors either side count the same. Its square root, the standard deviation, has the same units as the thing itself, which is why you quote σ and not σ².
Work it. Five range readings: 9.8, 9.9, 10.0, 10.1, 10.2.
Mean:
So: "this sensor is good to about 14 cm, one sigma." That sentence is what the arithmetic was for.
With two quantities, you need a third number beyond the two variances: do they err together?
Take five 2D position errors, in metres:
Variance in
Now the cross term — average the product of the two errors:
Divide by 5: 0.02. Assemble:
Its eigenvalues are 0.04 and 0 (check with §A.3:
Report only the diagonal and you would say "14 cm in
This is why §2.2 says, in bold, that off-diagonal terms are where the information is, and why storing a covariance as an array of per-axis variances is a bug waiting for a lever arm. A GNSS/INS system's position and attitude errors are always correlated, and an implementation that ignores the cross-covariance discards the correlation that would have let it fix both.
Divide the covariance by the two standard deviations and you get correlation, a number from −1 to +1. Geometrically it is the tilt and pinch of the ellipse from §A.3:
- 0 — axis-aligned ellipse. Knowing one tells you nothing about the other.
-
+0.9 — a thin ellipse tilted up-and-right. Knowing
$x$ pins$y$ down hard. - −0.9 — the same, tilted down-and-right.
- ±1 — the ellipse has collapsed to a line. One quantity is a deterministic function of the other, and your covariance matrix is now singular.
Where this shows up. §2.2, and then permanently — every filter state in Chapter 32, every track
in Chapter 33, every optimizer's reported uncertainty in §2.7. When a visualizer draws an ellipse
around a tracked vehicle, it is drawing
The Gaussian, or normal distribution, is the bell curve. Most of the mass near the centre, falling off fast, symmetric.
Before the formula, the shape of the idea: the log of a Gaussian is a parabola. That is the whole reason it dominates. Taking the logarithm of a Gaussian density turns it into "minus one half times a squared distance from the mean." So maximising probability becomes minimising a squared distance — and §A.6 just spent a section on how to minimise sums of squares.
That is the bridge. Probability and least squares are the same subject viewed through a logarithm. Every time the main document says "MAP estimation with Gaussian noise is weighted nonlinear least squares" (§2.4 says exactly this), it is invoking this one fact.
In one dimension:
The exponent is
In several dimensions the squared distance becomes the Mahalanobis norm from §A.6:
The fraction out front only normalizes the area to 1;
You believed something. You made a measurement. What should you believe now?
New belief ∝ (how well the measurement is explained by each possibility) × (how plausible each possibility was to begin with).
That is Bayes' rule. Three named pieces:
- Prior — what you believed before.
- Likelihood — for each candidate world-state, how probable was the data you actually got.
- Posterior — what you believe now.
A radar reports an obstacle. Your detector fires on 90% of real obstacles, and also false-alarms on 10% of empty cells. This cell was empty 99% of the time historically.
Detector fires. Is there an obstacle?
- Real obstacle, and detected:
$0.01 \times 0.9 = 0.009$ - Empty, and false alarm:
$0.99 \times 0.10 = 0.099$
Normalize:
A 90%-accurate detector fired, and there is still a 92% chance nothing is there. Not because the detector is bad — because obstacles are rare, and rare things stay unlikely until the evidence is strong. This is the base-rate fallacy, it is the single most common misreading of a detector's confidence score, and it is why §34.2's occupancy grids accumulate evidence over many frames in log-odds instead of trusting one detection.
The standard confusion.
In practice you implement the logarithm of all this, for two reasons: products of small probabilities underflow to zero in floating point, and logs turn the Gaussians into the quadratics of §A.6. Multiplication becomes addition; belief fusion becomes summing quadratic forms.
Where this shows up. §2.3 (four concrete reasons the Gaussian dominates, and the three places it fails), §2.4 (Bayes as the engine, and the log form you actually write), §2.5 (Mahalanobis gating — the χ² thresholds that reject a multipath GNSS fix before it corrupts your filter), Chapter 32 (recursive estimation: yesterday's posterior is today's prior), and §34.2 (log-odds occupancy).
This section is intuition only. The machinery is Chapter 4 of the main document, which is the chapter that most repays careful study; the goal here is to make you want that chapter.
Do this physically, with a book.
Lay a book flat, cover up. Rotate it 90° about the vertical axis, then 90° about the axis pointing away from you. Note where the cover faces.
Start over. Same two rotations, opposite order.
The book ends up somewhere else.
Rotation is not commutative.
Three degrees of freedom, three numbers — why not store roll, pitch, yaw and add them?
Because the space is curved. An analogy that is exact enough to trust: positions on the Earth's surface. Latitude and longitude are two numbers for a 2D space, so surely you can add them?
Stand at the equator. Move 100 km north — longitude unchanged. Now stand at 89° latitude and move 100 km north. You pass the pole and your longitude changes by 180°. The same movement produced a wildly different coordinate change, because the coordinates are a flat description of a curved thing. At the pole they break entirely: every direction is south, and longitude is undefined.
Euler angles have exactly this pathology and it has a name — gimbal lock — occurring at pitch ±90°, where two of the three axes line up and one degree of freedom silently vanishes. §4.2 is titled "Why not Euler angles" and this is why.
Three facts to carry into Chapter 4:
- Rotations compose by multiplication, not addition. Do one then another, and the combined rotation is their product. Order matters, as the book showed.
- The set of rotations is a curved surface, not a flat space. Nine numbers in a rotation matrix with six constraints between them, leaving three genuine degrees of freedom. Add two rotation matrices entry-by-entry and the result is not a rotation at all.
-
Small rotations are nearly flat, and that is the loophole. Near any given rotation, a small
enough neighbourhood looks like ordinary flat space, so you can use a plain 3-vector for the
correction while keeping the rotation itself in a proper form. This is exactly the trick that
makes §A.6's linearize-solve-repeat loop work on rotations, and it is what the
$\boxplus$ operator in §4.7 means: "apply this small flat correction to this curved-space quantity, correctly."
You do not need Lie algebra yet. You need to stop expecting rotations to behave like arrows.
Where this shows up. Chapter 4 in full: §4.2 (why not Euler angles), §4.3 (quaternions), §4.5
(the Lie group picture that makes all of this systematic), §4.6 (left vs right perturbation — a
convention that must be stated and frequently is not), and §4.7 (
Here is the sentence the main document never writes, in either §5.2 or §20.2, and it is the whole concept:
Slide a small pattern across a signal. At each position, report how strongly the signal matches the pattern.
That is convolution. The small pattern is the kernel or filter. The output is a map of match strength — high where the signal locally looks like the pattern, low where it does not.
Signal — think of it as brightness along one row of pixels, dark then bright:
Kernel, three taps, an edge detector:
Read the kernel in English before computing: take the value on my right, subtract the value on my left, ignore the middle. If the signal is flat, right minus left is zero. If it is rising, positive. It measures local change.
Slide it. At each position, line up the three taps with three signal values, multiply pairwise, add.
Position 1, centred on index 1 (values 0, 0, 0):
Position 2, centred on index 2 (values 0, 0, 10):
Position 3, centred on index 3 (values 0, 10, 10):
Position 4, centred on index 4 (values 10, 10, 10):
Output:
Look at what happened. The output is zero in the flat dark region, zero in the flat bright region, and large exactly where the transition is. The filter found the edge. It found it without being told what an edge is — the pattern was the definition.
Two details you just met without being told:
- The output is shorter than the input. A 3-tap kernel cannot centre on the first or last sample. That is why real code pads the boundary, and why every convolution API has a padding argument.
- The kernel is flipped in true convolution and not flipped in cross-correlation. For symmetric kernels there is no difference. Deep learning libraries implement cross-correlation and call it convolution, which is harmless because the weights are learned, but it will confuse you the first time you compare against a signal-processing textbook. §5.2 uses the true definition.
Same idea, kernel now a small square. Take a
The kernel is the same "right minus left" idea, stacked three rows tall so it averages over height while differencing across width — less noise-sensitive than a single row.
Place it over the top-left 0,0,10 in each
row. Each row contributes
Slide right by one, covering columns 2–4, values 0,10,10 per row. Each row gives
Both interior positions light up strongly, and a horizontal-edge kernel — the same thing rotated — would return 0 everywhere on this image. The filter responds to the structure it was shaped like. That is the entire idea behind hand-designed image filters, and it is also the entire idea behind convolutional networks, with the single difference that the network learns the numbers in the kernel instead of you choosing them (§20.1).
Every symbol:
If that formula had come first, it would have taught you nothing. It is a compressed record of the sliding you already did by hand.
Where this shows up. §5.2 (convolution and filtering as signal processing), §5.1 and §A.11 (the frequency-domain view: convolution in space is multiplication in frequency, which is why blurring removes high frequencies), §15.2 — linear filtering, where these same kernels get names: box, Gaussian, Sobel — and §20.1–20.3 (convolutional networks — why weight sharing is the right prior for images, and how the receptive field grows).
You have seen a wheel in a film appear to spin backwards. The wheel is going forwards. The camera samples 24 times a second, and between frames a spoke moves almost a full spoke-spacing forward. Your eye takes the nearest interpretation — a small backwards step — and it is wrong.
Nothing is broken. The camera recorded the truth. But sampling discarded the information needed to distinguish "nearly one spoke forward" from "slightly one spoke back," and once discarded it cannot be recovered.
Aliasing is when a signal changing too fast for your sampling rate masquerades as a slower signal. Note masquerades: it does not appear as noise, which you would notice. It appears as a clean, plausible, wrong answer.
To capture a signal oscillating at
Concretely. A vibration at 100 Hz, an IMU logging at 150 Hz. Nyquist demands more than 200 Hz, so
this is undersampled. The 100 Hz vibration reappears at
The fix must happen before sampling: a physical low-pass filter that removes above-Nyquist content while it is still analogue. Filtering afterwards cannot work — by then the 100 Hz and the 50 Hz are the same numbers. This is why IMUs have analogue anti-alias filters ahead of the ADC, and why raising a logging rate sometimes makes a mysterious oscillation vanish.
The same rule, with distance instead of time. Pixels sample the image plane, so image detail finer than two pixels aliases. This is the moiré on a striped shirt, and the jagged staircase on a thin diagonal wire.
It is also why you must blur before you downsample. Halving an image by taking every second pixel folds all the fine detail into false coarse patterns. Blur first — removing the detail the smaller grid cannot represent — then discard. Every correct image pyramid does this, which is why §15.5 — pyramids and scale — blurs at every level rather than naively subsampling, and why the scale-space detectors of §16.3 are built on those pyramids.
Sampling at interval
Where this shows up. §5.1 (sampling and aliasing directly), §5.3 — which the main document calls the most underestimated topic in the whole book, and it is about time, timestamps, and what happens when different sensors sample the world at different instants — §5.4 and §8.7 (motion compensation and LiDAR deskewing: a spinning LiDAR samples each column at a different time, so a "scan" is not a snapshot), and §7.4 (rolling shutter, the same problem in a camera).
Part A was mathematics that has been true for two centuries. Part B is five ideas that make neural networks comprehensible. It is deliberately short: almost everything in deep learning is §A.5's Jacobian and §A.6's downhill walk, applied at scale with good engineering. If Part A landed, this part is mostly renaming.
You have inputs and the outputs you wish you got. You want a machine that turns one into the other.
You do not search over all possible machines — that is not a well-posed thing to do. Instead you pick a family of machines described by a set of adjustable numbers, and search over the numbers. The numbers are parameters (or weights). The family is the model. Choosing the family is where all your assumptions live.
Then you need a way to score a setting of the numbers: a loss, one number saying how badly the current machine is doing on your data. Low is good. Fitting is turning the knobs to make the loss small — which is §A.6 exactly, with more knobs.
Predict stopping distance from speed. Family: straight lines,
Three data points: at 10 m/s it took 20 m; at 20 m/s, 45 m; at 30 m/s, 65 m.
Try
Try
Better. That comparison — change the numbers, recompute the loss, keep what helps — is the entire activity. The rest is doing it efficiently, which is §B.2, and choosing richer families than straight lines, which is §B.3.
The one thing that is genuinely new. In §A.6 the model came from physics: projection, rotation, range. The residual meant something. Here the family is chosen for flexibility rather than correctness, and it will happily fit patterns that are accidents of your particular data. That failure has a name — overfitting — and it is why a held-out test set is not optional. §25.6 treats validation methodology as an engineering discipline, and it is right to.
Where this shows up. §19.1 (the problem being solved), §19.2 (from linear models to networks), §25.1 (the data engine), §25.6 (validation methodology).
The loss is a landscape. The parameters are your position. You want the lowest point, in the dark, knowing only the slope beneath your feet.
So feel which way is downhill, take a small step, repeat. That is gradient descent. The gradient (§A.5, rung 3) points steepest uphill, so you step against it. The step size has a name — learning rate — and it is the parameter that decides whether this works at all.
Minimise
The slope is
| Step | slope |
||
|---|---|---|---|
| 0 | 0 | −6 | 0.6 |
| 1 | 0.6 | −4.8 | 1.08 |
| 2 | 1.08 | −3.84 | 1.464 |
| 3 | 1.464 | −3.07 | 1.771 |
Each step closes 20% of the remaining gap. It converges — geometrically, never quite arriving, which is fine.
Learning rate 0.5. Slope at
Learning rate 1.0. Step is
Learning rate 1.1. From NaN within a few hundred iterations.
That is the entire tuning story: too small and you crawl, too large and you oscillate or explode, and the boundary depends on the curvature, which differs per direction and changes as you move. Real optimizers (§19.5) are all schemes for choosing the step per-parameter and adapting it, and now you know what they are protecting you from.
In practice you do not evaluate the gradient on all your data — millions of examples, one step. You use a random mini-batch of a few hundred and accept a noisy gradient estimate, which is SGD, stochastic gradient descent. The noise turns out to help, by shaking the walk out of shallow bad valleys.
The connection worth making. §A.6's Gauss–Newton also walks downhill, but it uses the Jacobian to
model the shape of the valley and jump to its bottom in one solve. Gradient descent uses only the
slope and takes a fixed small step. Gauss–Newton is far faster when you can afford to build and
factor
Where this shows up. §19.5 (optimization in practice: SGD, momentum, Adam, schedules), §3.4 (Levenberg–Marquardt, which interpolates between gradient descent and Gauss–Newton by exactly the trade-off above).
A layer that multiplies by a matrix is a linear map (§A.2). Stack two:
The product of two matrices is a matrix. Two layers are one layer. Stack fifty and it is still one layer, with a slower forward pass and a harder optimization problem. Depth bought exactly nothing.
Concretely:
Feed
Put a simple nonlinear function between the layers and the collapse stops. The standard choice is ReLU: pass positive numbers through unchanged, replace negatives with zero.
Trivial. And it is enough, because of what it does geometrically: it is a hinge. The function is flat, then it bends and rises. One bend.
Now watch bends accumulate. Take three ReLU units on the same input
Evaluate:
-
$x=0$ : all three arguments negative, all zero.$g = 0$ . -
$x=1.5$ : first unit gives 0.5, others 0.$g = 2(0.5) = 1.0$ . -
$x=3$ : first gives 2, second gives 1, third 0.$g = 2(2) - 3(1) = 1.0$ . -
$x=5$ : first gives 4, second 3, third 1.$g = 8 - 9 + 1 = 0$ .
Plot those and you get a curve that rises, flattens, and falls — three straight segments joined at the thresholds. Not a line. With enough hinges placed at enough thresholds you can trace any continuous shape as closely as you like, which is the honest content of the "universal approximation" result.
So the nonlinearity is not a detail bolted onto the linear algebra. It is the only reason depth exists. Without it there is nothing to be deep about.
The standard confusion. People assume the nonlinearity must be something sophisticated. ReLU is
max(0,x) and outperformed the elegant smooth sigmoids it replaced, largely because its gradient is
exactly 1 wherever it is active — no shrinkage as signals travel back through many layers, which is
the vanishing-gradient problem the sigmoids suffered from.
Where this shows up. §19.2 (from linear models to networks — the same collapse argument), §19.6 (normalization layers, and a genuine deployment trap), §20.1 (why convolution is the right prior for images: a convolution is a linear layer with weights shared across position, so the same reasoning applies with an extra structural constraint).
A classifier's final layer emits a few arbitrary real numbers — one per class, any sign, any magnitude. Call them scores, or logits. They are not probabilities: they can be negative, and they do not sum to anything in particular.
You want probabilities: all positive, summing to 1. So do two things. Make them positive by exponentiating. Make them sum to one by dividing by their total. That is softmax, complete.
Exponentiating has a side effect worth knowing about: it amplifies differences. A score 1 higher
becomes
Three classes — car, pedestrian, cyclist — with scores
Exponentiate:
Total:
Divide:
Now the loss. You want a score that is near zero when the model puts high probability on the right answer, and grows without limit as that probability approaches zero.
The function with that shape is the negative logarithm. That is cross-entropy: take the probability the model assigned to the correct class, and report minus its log.
Continuing the example. If the true class is car, the model gave it 0.659:
If the true class was actually cyclist, which the model gave 0.099:
Five times worse. And if the model had been confidently wrong — 0.001 on the true class —
One fact that makes the implementation clean. Differentiate cross-entropy composed with softmax and almost everything cancels:
Predicted minus target. For the example above with true class car, the gradient on the logits is
Where this shows up. §19.3 (loss functions and what they encode), §21.2 (scaled dot-product attention, whose weights are a softmax over similarity scores — the same function doing a different job), §23.4 — losses and imbalance, where a rare class breaks plain cross-entropy and Dice loss answers it — §25.7 (uncertainty and calibration: a softmax output is not a calibrated probability, and treating it as one is a real and common error).
A network has millions of parameters and one loss. Training needs, for every single parameter, the answer to: if I nudge you, how much does the loss move?
That is the Jacobian question from §A.5, asked ten million times.
The obvious approach is to nudge each parameter and re-run the network. With
The trick rests on one observation: there are many inputs and only one output. The loss is a single number. So instead of pushing perturbations forward from each parameter, start at the loss and propagate sensitivity backwards toward the parameters. One pass, and every parameter gets its answer.
The cost of a backward pass is about the same as a forward pass, regardless of parameter count. That is the whole reason training large networks is feasible. Backpropagation is not a learning algorithm — gradient descent is the learning algorithm (§B.2). Backpropagation is just how you get the gradient cheaply.
One weight, one data point.
Forward, recording each intermediate:
$u = wx = 2 \times 3 = 6$ $r = u - y = 6 - 5 = 1$ $L = r^2 = 1$
Backward, starting from the end with
$\dfrac{\partial L}{\partial r} = 2r = 2$ $\dfrac{\partial L}{\partial u} = \dfrac{\partial L}{\partial r}\cdot\dfrac{\partial r}{\partial u} = 2 \times 1 = 2$ $\dfrac{\partial L}{\partial w} = \dfrac{\partial L}{\partial u}\cdot\dfrac{\partial u}{\partial w} = 2 \times 3 = 6$
So nudging
Verify it numerically, the §A.5 habit. Set
Each step above multiplied by one local Jacobian — how that operation's output responds to its input. The chain rule says the total sensitivity is the product of the local ones, and going backwards means multiplying them right-to-left.
For a network of layers
with backward() is: one vector-Jacobian product per
operation.
Two consequences that explain a great deal of practical deep learning:
-
The forward pass's intermediates must be kept in memory, because the backward pass needs them
(
$\partial u/\partial w = x$ needed the input$x$ ). This is why training memory scales with depth and batch size while inference memory does not, and why gradient checkpointing — recompute instead of store — is a real technique. - Long chains multiply many factors together. If the factors are consistently below 1 the product decays to nothing (vanishing gradients, and early layers stop learning); above 1 and it explodes. Residual connections, normalization layers, and ReLU's gradient-of-exactly-1 are all responses to this single arithmetic fact.
Where this shows up. §19.4 (backpropagation), §19.6 (normalization layers, and the trap where train-time and deploy-time behaviour differ), §3.9 — whose advice to verify analytic Jacobians against finite differences is the same habit used above, and applies with equal force to a custom autograd operation as to an ICP residual.
You have the mathematics. Three routes into the main document:
Reading it linearly. Start at Part 0 and go. Part I will now read as the perception interpretation of what you just learned rather than as a wall of new material — that is precisely its redesigned job.
Coming back to look something up. Each chapter's Reference card is written for you. This file is not; you have already read it.
Hitting something unfamiliar mid-chapter. Chapters carry a > **Prereq:** block naming the
primer section they assume. Follow it, read that one section, come back.
Two things stayed out of this file on purpose. Rotations get only intuition here (§A.9), because doing them properly needs Lie groups and Chapter 4 does that better than a primer could. Anything sensor-specific — how a LiDAR actually measures range, what a rolling shutter does to a moving scene — is physics rather than mathematics, and lives in Part II.
One habit to carry across, above all others. When you meet a new equation in the main document, do not read it as a statement. Read it as an answer to a question, and find the question first. Most of them are asking either how much does the output move when I nudge the input (§A.5) or which setting of the knobs makes the disagreement smallest (§A.6). Those two questions cover a startling fraction of the book.