Short revision notes. Exercises are not covered.
Chapter 3’s closed-form posteriors worked because the prior and likelihood were conjugate. Most useful models are not: for example, binary classification has no conjugate pair. We can still evaluate the numerator of Bayes’ rule, but the normalising integral cannot be done. Chapter 4 gives three ways to cope, all illustrated on one classification problem.
Each object has attributes \(\mathbf{x}_n\) and a label \(t_n \in \{0,1\}\).
The three options: (1) find the single best \(\mathbf{w}\), (2) approximate the posterior with a simpler density, (3) draw samples from the posterior.
Maximise \(\log g(\mathbf{w})\) (prior times likelihood). The normaliser does not depend on \(\mathbf{w}\), so it does not affect where the maximum is. There is no closed form, so use an iterative method, Newton–Raphson: repeatedly move to where the local tangent (or, for a maximum, the local quadratic model) crosses zero,
\[\mathbf{w} \leftarrow \mathbf{w} - \mathbf{H}^{-1}\,\nabla \log g,\]where \(\mathbf{H}\) is the Hessian. For this model
\[\nabla \log g = -\tfrac{1}{\sigma^2}\mathbf{w} + \sum_n \mathbf{x}_n\,(t_n - P_n), \qquad \mathbf{H} = -\tfrac{1}{\sigma^2}\mathbf{I} - \sum_n \mathbf{x}_n\mathbf{x}_n^T P_n(1-P_n).\]\(\mathbf{H}\) is negative definite for every \(\mathbf{w}\) (because \(0<P_n<1\)), so there is a single maximum and the iteration finds it.
The decision boundary is where \(P=0.5\), i.e. \(\hat{\mathbf{w}}^T\mathbf{x}=0\), a straight line. The weakness: one value of \(\mathbf{w}\) ignores the remaining uncertainty and gives over-confident predictions, even far from the data.
Replace the posterior with a Gaussian that matches its peak and its curvature there. Expand \(\log g\) to second order around the mode \(\hat{\mathbf{w}}\) (the first-order term vanishes at a maximum):
\[p(\mathbf{w}\mid\mathbf{X},\mathbf{t},\sigma^2) \approx \mathcal{N}(\hat{\mathbf{w}},\ \boldsymbol{\Sigma}), \qquad \boldsymbol{\Sigma} = -\mathbf{H}^{-1}\ \text{at}\ \hat{\mathbf{w}}.\]The mean is the MAP point and the covariance is the inverse curvature. We already have both from Newton–Raphson, so the approximation is nearly free.
Expectations can be estimated by averaging over samples, even if the density cannot be written down. (Darts example: the chance of hitting the target is just the fraction of throws that hit it.) Metropolis–Hastings produces samples from the posterior knowing only \(g\):
The chain moves uphill always and downhill sometimes, so it spends time in proportion to the posterior density.
| MAP | Laplace | Metropolis–Hastings | |
|---|---|---|---|
| Result | One parameter vector | A Gaussian over \(\mathbf{w}\) | Samples from the true posterior |
| Uncertainty in \(\mathbf{w}\) | Ignored | Kept, but only as Gaussian | Kept in full |
| Cost | Low | Low (reuses the MAP work) | Highest |
| Main risk | Over-confident predictions | Poor if the posterior is not Gaussian-like | Slow mixing, convergence checks needed |
Short descriptions of papers that use the chapter’s ideas on real data. Each paragraph links back to the option it illustrates.
“Two Projections Suffice for Cerebral Vascular Reconstruction” (MICCAI 2024) rebuilds a 3D vessel volume from only two X-ray-style views. A U-Net first gives a rough 3D estimate. Then a MAP stage takes over: the volume is adjusted iteratively until its re-projections match the two views, while a prior that favours connected vessels keeps the answer plausible. This is the chapter’s recipe in miniature: a data-fit term (the likelihood) plus a prior that regularises, and the single best volume is the one that maximises their product, just as MAP trades the likelihood against the prior in Chapter 3. The scans come from public brain MRA data, but the two views are simulated projections, and the authors note there is no validation on real fluoroscopy.
Wang, Wells, Golland and Zhang, “Efficient Laplace Approximation for Bayesian Registration Uncertainty Quantification” (MICCAI 2018), is the closest match to the chapter’s structure, and it does not use deep learning. A registration warps one brain scan onto another. The method first finds the MAP deformation by minimising the negative log posterior (a matching term plus a smoothness prior). It then approximates the posterior as a Gaussian centred on that deformation, with covariance equal to the inverse Hessian. The difficulty is size: a deformation of a 3D image has far too many parameters to form the Hessian directly, so the authors work in a low-dimensional frequency representation of the deformation, where the Hessian can be handled efficiently using products with vectors. The result is a map of how uncertain the registration is at each location, tested on real OASIS brain MRI.
Zepf et al., “Laplacian Segmentation Networks Improve Epistemic Uncertainty Quantification” (MICCAI 2024; arXiv), apply the recipe to a medical image segmentation network. First the network is trained to a single best set of weights, the MAP-style solution of Option 1. Then they treat that point as the centre of a Gaussian over the weights, using the curvature (Hessian) at the point to set its spread, which is Option 2. In this chapter the model has two weights, so the Hessian is a 2×2 matrix that can be formed and inverted. A segmentation network has millions of weights and a prediction for every pixel, so the authors use a much cheaper structured approximation of the Hessian, designed to cope with the network’s skip connections. The approximate posterior is then used as the chapter prescribes for predictions: draw several weight vectors from the Gaussian and look at all of the resulting predictions. Where the sampled networks disagree about a pixel, the model does not know the answer, which is epistemic uncertainty. Uncertainty from the data itself (aleatoric) is modelled separately in the network’s output. The disagreement flags out-of-distribution images, where an ordinary network tends to stay confidently wrong. The authors report evaluating it on three public medical segmentation datasets, with better out-of-distribution detection than MC dropout and deep ensembles.
“Last Layer Laplacian Pseudocoresets for Robust Medical Image Analysis” (MICCAI 2025) shrinks the problem differently: apply the Laplace approximation to the final layer only. After the network is trained, a Gaussian is placed over the last layer’s weights, which are few enough for the Hessian to be handled easily, the same trick as the two-weight example in this chapter but for one layer of a network. The paper then fits a small set of pseudo-examples (a “pseudocoreset”) against this posterior, aiming at robust, uncertainty-aware predictions under class imbalance without retraining the network. It is evaluated on skin-image datasets (ISIC 2019 and ASAN). A reviewer on the paper’s page notes that last-layer Laplace is not new in itself and that the novelty is the pseudocoreset step.
Madsen et al., “A Closest Point Proposal for MCMC-based Probabilistic Surface Registration” (ECCV 2020), is not a medical imaging paper, but it is motivated by medicine and uses CT-derived bone shapes. It treats registering one 3D surface onto another as inference: a Gaussian-process shape model is the prior, a point-distance model is the likelihood, and the resulting posterior over registrations can be evaluated up to a constant but not normalised, exactly the situation of this chapter. They sample it with Metropolis–Hastings. The standard random-walk proposal needs very long runs in high dimensions, so the paper designs an informed proposal that uses the geometry (a point on the template moves toward its closest point on the target) while staying a valid, stochastic proposal so the acceptance ratio remains well defined. The samples give an uncertainty for each point of the surface, and for completing partly missing shapes a set of posterior samples covers the ground truth better than the single most likely (MAP) shape. Data: 50 healthy femur meshes extracted from CT, and 10 face scans.