Subspaces, Hands On

← text notes · Foundations › Linear algebra. Four widgets for column space, null space and the residual space: what a matrix can reach, what it cannot see, and what is left over when you fit.

1. What a 2 × 2 matrix does to the plane

The matrix A has two columns, a₁ and a₂. Multiplying, Ax = x₁a₁ + x₂a₂, is "walk x₁ steps along a₁, then x₂ steps along a₂". So the set of outputs A can produce, the column space, is everything you can reach by combining the columns.

Drag the arrow tips of a₁ and a₂ on the right, and the input vector x on the left. While the columns point different ways, the grid is stretched but still covers the whole plane: the column space is all of ℝ² and only x = 0 maps to 0. Drag one column until it lines up with the other (it snaps). The grid flattens onto a line. That line is the column space, and a whole line of inputs, the null space, now lands on 0. Drag x along the orange line and watch Ax not move.

input space: where x lives

output space: where Ax lives

column / row space null / left null space x and Ax
Things to try. Make it singular, then drag x across the orange line: Ax slides along the blue line. Drag it along the orange line: Ax stays put. Every input splits into a part A sees (row space) and a part it cannot (null space), and those two lines are always at right angles.

2. Least squares is a right angle

Fit a straight line y = c₀ + c₁t through three points at t = 0, 1, 2. Stack the three heights into one vector b ∈ ℝ³. The design matrix has columns a₁ = (1, 1, 1) (the intercept) and a₂ = (0, 1, 2) (the slope), so every straight line you could draw is a point Ax on the plane they span. The best fit p is the point of that plane closest to b, and the miss r = b − p meets the plane at a right angle.

Drag the points on the right (or use the sliders) and the 3-D picture on the left follows. Drag on the left to rotate. Notice that r only ever points one way, along ±(1, −2, 1): with three points and two columns the residual space is one-dimensional. Switch to fit a constant and the column space shrinks to the line through (1, 1, 1) while the residual space grows to a plane. The residual is then just the data with its mean subtracted: centring is projecting onto a residual space.

output space ℝ³ (drag to rotate)

the same numbers as a fit (drag the points)

column space and its columns data b fit p = Ax̂ residual r = b − p
The normal equations AᵀA x̂ = Aᵀb are the right angle written in algebra: they say Aᵀ(b − Ax̂) = 0, i.e. r is perpendicular to every column. The readout checks that, and checks Pythagoras, ‖b‖² = ‖p‖² + ‖r‖², for whatever b you set.

3. Many solutions, and the one gradient descent picks

One equation in two unknowns, a·x = β, has a whole line of solutions: any one solution plus anything from the null space (the dashed orange line through 0, all x with a·x = 0). Gradient descent on ½(a·x − β)² takes steps along a only, because the gradient is (a·x − β)·a. So it moves perpendicular to the solution line and the null-space coordinate of x never changes.

Drag the start point, then press Run. From 0 it lands on the blue dot, the minimum-norm solution A⁺β. From anywhere else it lands exactly one null-space offset away from it, the offset it started with.

solution line and null space row space; minimum-norm solution gradient-descent path
This is the mechanism behind two statements in the paper notes. In the NTK notes, "Δ⁰ never moves": the part of the error in the kernel's null space is never trained. And in overparameterised regression, gradient descent from zero initialisation returns the minimum-norm interpolant, not just an interpolant.

4. Principal subspace vs. residual subspace

Training features rarely fill their space. Fit PCA and keep the top direction: that line is the principal subspace, and the direction perpendicular to it is the residual subspace. Every feature splits into a principal part and a residual part. Out-of-distribution detectors based on residuals score only the second part, on the bet that new kinds of input show up in directions the training data never used.

Drag the test point. The shaded region is what each score accepts at its 99th training percentile. Compare shapes: distance to the mean accepts a disc; the residual norm accepts a band along the data (anything along the principal line passes, however far out); full Mahalanobis accepts an ellipse.

principal subspace residual subspace; residual of the test point accepted region
In 2-D with one principal direction the residual space is a single line, so "residual norm" and "residual Mahalanobis" rank points identically. In a 768-dimensional feature space the residual space has hundreds of directions with very different variances, and that is where MaRS's reweighting of the residual by its own covariance starts to matter.

Everything here is computed in the browser. The numbers quoted in the text notes come from code/subspaces.py.