The quadratic bowl
Chapter 2 · The log-likelihood of the Gaussian linear model, drawn as a surface over (w0, w1). Its shape is an upside-down bowl whose top is the maximum-likelihood solution. Drag the picture to turn it, and move the red point with the sliders.
Height is the log-likelihood (the top, in green, is the maximum). The surface is cut off at a fixed depth. The red point sits on the surface at the chosen (w0, w1), with a dashed line down to the floor. The purple ellipses on the floor are the same contours as in the previous page.
log-likelihood at the red point (direct):
maximum − ½ (w − ŵ)ᵀ(−H)(w − ŵ):
drop below the top:
Because log L(w) = const − (1/2σ²)(t − Xw)ᵀ(t − Xw) and the residual is linear in w, log L is exactly a quadratic function of w. Writing it around its maximum ŵ gives
log L(w) = log L(ŵ) − ½ (w − ŵ)ᵀ(−H)(w − ŵ), with −H = XᵀX/σ².
The two numbers above are computed by different routes and agree. The Hessian sets the width and tilt of the bowl: large curvature means a narrow, steep bowl, and a larger σ flattens it.