Evaluating the numerator of Bayes' rule at one w
Chapter 4 · For the binary dataset, pick a weight vector w and compute g(w) = prior × likelihood, the numerator of the posterior. Step 1 is the prior, which needs only w and σ². Step 2 is the likelihood, where each point gets the probability the model assigns to its observed label and the 40 numbers are multiplied.
Step 1: the prior
The prior is N(0, σ²I) on the two weights, so with D = 2: p(w) = (2πσ²)−1 exp(−wᵀw / (2σ²)). It depends only on the weights and says nothing about the data. The rings are 1, 2 and 3 prior standard deviations from the centre; the red dot is the chosen w.
Rings: prior standard deviations from the centre. The dashed line joins the centre to the chosen w.
| wᵀw = w₁² + w₂² | |
| Exponent −wᵀw / (2σ²) | |
| exp(exponent) | |
| Normaliser 1 / (2πσ²) | |
| Prior density = normaliser × exp(exponent) | |
| Distance of w from the centre, in prior standard deviations |
Step 2: the likelihood
For each point: z = wᵀx, then P = 1/(1 + e−z) is the probability of label 1, and the factor in the likelihood is P if t = 1 and 1 − P if t = 0.
Each point is coloured by P(observed label): green when high, red when low. The line is the boundary wᵀx = 0.
| # | x₁ | x₂ | t | z = wᵀx | P(t=1) | P(observed t) | log |
|---|
The product of 40 numbers below 1 can get extremely small, so in practice we add logs instead of multiplying probabilities.
Step 3: multiply (or add the logs)
g(w) is proportional to the posterior density at w. It is not the posterior itself, because we have not divided by Z, the integral of g over all w, which is the part with no closed form. Try different w: g is largest near the weights that fit the data and are not too large.
Step 4: g(w) for every w, the unnormalised posterior
Steps 1 to 3 gave one number, g at the chosen w. Now evaluate g on a grid of all (w₁, w₂) and colour each cell by its value. The picture is the shape of the posterior, only on the wrong scale. The red dot is your chosen w, the green diamond is the peak (the MAP weights).
Contours are drawn at fixed fractions of the peak value (90%, 70%, 50%, 30%, 10%, 1%). The grid covers w₁ and w₂ from −6 to 10.
With two weights, summing over a grid gives rough estimates of Z and of the posterior mean. That is feasible only because there are two weights: the number of grid points needed grows exponentially with the number of weights, which is why the general case needs MAP, Laplace or sampling.