Laplace approximation for the binary response model

Chapter 4 · The posterior over the two weights, g(w) / Z, has no standard form. The Laplace approximation replaces it by a Gaussian centred at its peak (the MAP weights, found by Newton–Raphson) with covariance Σ = −H⁻¹, the inverse curvature at that peak. Samples from the Gaussian then give a fan of decision boundaries and an averaged prediction.

20 samples are drawn for the plots, and 1000 for the averaged prediction.

Step 1: the posterior over the weights and its Gaussian stand-in

━ contours of the true unnormalised posterior g(w)   ╍ contours of the Laplace Gaussian N(ŵ, −H⁻¹) at the same drops in log height below each peak (0.5, 2, 4.5, 8)   ◆ ŵ, the MAP weights   grey dots: 20 sampled weight vectors. Near the peak the two agree. Further out, g is lopsided and the symmetric Gaussian is not.

MAP weights ŵ
Curvature −H at the peak
Posterior covariance Σ = −H⁻¹
Posterior std of w₁, w₂

Step 2: each sampled weight vector is a decision boundary

Each grey line is the boundary wᵀx = 0 for one sampled w (the 20 grey dots above). The green line is the MAP boundary. Where the grey lines fan out, the model is unsure which side a point belongs to. The red cross is your test point.

Step 3: predictions, a single weight vector against an average over the posterior

Using only the MAP weights: P(t = 1 | x, ŵ)

Averaged over 1000 posterior samples

Blue means P(t = 1) near 0, red means near 1, white means near 0.5. The red cross is your test point.

P(t = 1) at the test point, MAP weights only
P(t = 1) at the test point, averaged over samples
Share of the 1000 sampled boundaries that put the point in class 1
The prediction we would like is the average of the sigmoid over the posterior, which has no closed form. The Gaussian is easy to sample from, so we average the sigmoid over samples instead: (1/Ns) Σs 1/(1 + exp(−wsᵀx)). Far from the data, the sampled boundaries disagree, so the average sits near 0.5, while a single weight vector can be confidently wrong there.