Jacobian and Hessian: what moves where, and how it bends

jacobianhessiancurvaturechange-of-coordinatespositive-definite

In one paragraph

Both are matrices of derivatives. The Jacobian collects the first derivatives of a function that maps vectors to vectors, and it is the linear map that the function looks like when you zoom in. The Hessian collects the second derivatives of a function that returns one number, and it measures how the ground bends in each direction. The Hessian is the Jacobian of the gradient.

Jacobian: the local linear map

Take f that sends an input x=(x_1,\dots,x_n) to an output f(x)=(f_1,\dots,f_m). Its Jacobian at a point is the m\times n matrix of first partial derivatives,

J_{ij}=\frac{\partial f_i}{\partial x_j}.

Column j answers one question: if input x_j is nudged a little, where does the output move?

Left: a square grid with a small cell and its two edge steps, blue along the first axis and orange along the second. Right: the grid bent by a smooth map; near the marked point the small cell is almost a parallelogram, and the two steps have become the two columns of the Jacobian. — Zoom in on a smooth map and it looks linear. The two small input steps land on the two columns of the Jacobian, and the small square becomes a parallelogram.

Near the dot the bent grid is almost a grid of parallelograms. The Jacobian is that stretch, rotation and shear. It also tells how small areas scale: a small square becomes a parallelogram whose area is |\det J| times as large (for m=n), which is why |\det J| shows up when a change of variables is made inside an integral.

Information geometry uses it whenever it changes coordinates, for example from natural parameters \theta to expectation parameters \eta.

Hessian: the curvature

For f:\mathbb R^n\to\mathbb R the Hessian is the n\times n matrix of second partial derivatives,

H_{ij}=\frac{\partial^2 f}{\partial x_i\,\partial x_j}.

It is symmetric for smooth f. The gradient says which way is uphill; the Hessian says how the ground bends as you walk, and v^\top H v is the curvature along the direction v.

Three contour pictures around a point where the slope is zero. Left: ellipses, a minimum. Middle: hyperbolas, a saddle. Right: parallel lines, a trough with a flat direction. — Contour lines near a point where the slope is zero, sorted by the Hessian there: curving up both ways is a minimum, up one way and down the other is a saddle, and flat in one direction is a trough.

The same three, as surfaces

Three surfaces over a grey floor, each with two black slice curves through the origin along the two axes. Left: a bowl, both slices curve up. Middle: a saddle, one slice curves up and the other down. Right: a trough, one slice curves up and the other is straight. — The height is f and the two black curves are slices through the origin along the two axes. Each slice bends exactly as much as the Hessian says it does in that direction: up, down or not at all.

Read the black curves one at a time. A curve that bends up has positive curvature in its direction, one that bends down has negative curvature, and a straight one has none. The Hessian is the table of those bends for every direction at once; when all of them are positive the surface is a bowl.

This is the positive-definite picture of the earlier foundations page applied to curvature:

Positive definite: the ground curves up in every direction, so the point is a minimum.
Both signs: it curves up one way and down the other, so the point is a saddle.
Semi-definite: it is flat along one direction, so nothing pins the point down along that direction.

In information geometry

For a convex potential \psi(\theta), the Hessian is positive definite and serves as the metric; for an exponential family it is the Fisher information matrix. That is how a ruler comes out of a potential function: see Chapter 1: dually flat structure and, for the idea of a metric, distance, divergence and metric.

Potential: the function whose gradient is a vector field, and whose Hessian is the metric: Potential, function and vector field.

Takeaways

The Jacobian is the local linear map of a vector-valued function: its columns are where the coordinate steps land, and |\det J| is the local area scaling.
The Hessian is the matrix of curvature of a real-valued function: v^\top H v is the bend along v.
The Hessian is the Jacobian of the gradient.
A positive definite Hessian means a minimum; mixed signs mean a saddle; a zero direction means a trough.