Bayesian evidence: three things to poke

Companion to the notes on Knuth et al. (2015). Each panel is drawn live in your browser.

1. The Occam factor

The parameter is a mean with a flat prior of width Δx. Data narrow the likelihood to a bump. Widen the prior, or add data, and watch the evidence move: the evidence is the area of the bump divided by Δx.

prior width Δx number of data n noise σ sample mean

Model M₀ has no free parameter: the mean is fixed at 0. M₁ has the mean free over the flat prior. The ratio shown is Z₁/Z₀ (the Bayes factor for the flexible theory).

2. Nested sampling, step by step

A narrow Gaussian likelihood on a square prior. N live points; each step deletes the worst and replaces it by a new draw from the prior restricted to higher likelihood (here by plain rejection, which is fine in two dimensions). The estimate of the prior mass uses X shrinking by e^−1/N per step.

live points N likelihood width

3. Estimating a ratio of normalisers by importance sampling

Target p (unnormalised) and proposal q (unnormalised): Gaussians with different widths. The true ratio Z_p/Z_q is known here. Below the picture, two estimates are printed: the plain average of p/q over samples from q (the estimator derived in eq. 37) and the expression printed as eq. (38), Σp²/q² ÷ Σp/q. Press the button repeatedly.

target width proposal width samples