Metropolis–Hastings: a random walk on the Poisson posterior

Chapter 4, Exercise 4.8 · The model is Poisson regression, t_n ~ Poisson(exp(w₁ + w₂u_n)), with a Gaussian prior N(0, σ²I) on the two weights. We can evaluate g(w) = prior × likelihood at any w but not the normaliser, so we sample with a random walk: propose a nearby w*, accept it with probability min(1, g(w*)/g(w)), otherwise stay put. The data are synthetic and are the same as in the Python script (the first n of 60 points).

Observations used n

Prior variance σ²

Proposal step size

Burn-in (samples discarded)

The walk in weight space

━ contours of the true unnormalised posterior g(w) at drops of 0.5, 2, 4.5 and 8 in log height ╍ Laplace Gaussian at the same drops ◆ mode ● burn-in samples (discarded) ● kept samples grey line: the last 150 steps ● black: the current point the dashed segment shows the latest proposal: green if accepted, red cross if rejected.

Trace plots

Each weight along the chain. A good chain wanders around a level without drifting. The dashed vertical line marks the end of burn-in. The red line is the Laplace mean and the pale band is ±2 Laplace standard deviations.

Marginal histograms against the Laplace approximation

Bars: kept samples, which approximate the true posterior. Red curve: the Laplace Gaussian. Dotted line: the sample mean.

Iterations / acceptance rate

Acceptance rate, last 200 steps

Latest proposal

Kept samples

Mean of samples vs the mode (Laplace mean)

Std of samples vs Laplace std

Correlation of w₁, w₂: samples vs Laplace

Lag-1 autocorrelation of w₁

What to try: (1) Restart far from the mode with a small burn-in and watch the walk head to the high-density region: those first points are the burn-in. (2) Make the step size tiny: almost every proposal is accepted, but the walk crawls, so the trace is smooth and the autocorrelation is close to 1. (3) Make it huge: almost every proposal is rejected, so the chain sits still for long stretches. The rule-of-thumb step (about 2.4/√D times the posterior standard deviation) gives an acceptance rate near 30%. (4) Lower n: the posterior becomes wider and more skewed, and the sample mean separates from the mode, while the Laplace Gaussian stays centred on the mode.