Curvature of the likelihood: the Hessian
Chapter 2 · The log-likelihood of the Gaussian linear model is a function of (w0, w1). Its Hessian is the matrix of second derivatives, and it says how sharply the log-likelihood bends around its peak in each direction. Drag the red point in the left panel to try other parameter values.
Purple ellipses are the places where the log-likelihood has dropped by 0.5, 2 and 4.5 below its peak. The two arrows are the principal directions of the Hessian: the short arrow is the steep direction, the long arrow the flat one.
Hessian H = ∂²log L/∂w∂wᵀ:
Curvature along the principal directions (eigenvalues of −H):
steep flat
steep flat
━ maximum-likelihood line ━ line for the red point ● data
log-likelihood at the red point:
at the maximum:
For this model log L(w) = const − (1/2σ²) (t − Xw)ᵀ(t − Xw). Differentiating twice gives H = −(1/σ²) XᵀX, which does not depend on w: the log-likelihood is a quadratic bowl, and every point has the same curvature. A larger noise σ makes the bowl flatter. The curvature also changes with how the inputs are laid out: with years counted from zero the bowl becomes a long, thin, tilted valley, and w0 and w1 can trade off against each other with almost no loss in likelihood.