Probabilistic PCA, Interactively

← text notes · Foundations › Probability. Three widgets: grow a data cloud from a hidden coordinate, locate a point's hidden coordinate from where it landed, and see where the noise level comes from.

1. The generative story: a line, blurred

Every point is made in two steps. First draw a hidden number z from a bell curve. Then walk that far along the arrow w (the z-th multiple of it), and finally add a little noise that is equally strong in every direction. The data you see is a stretched cloud around the line. Drag the sliders to rotate the arrow, lengthen it, or turn up the noise.

The solid ellipses are the model's own picture of the data, the Gaussian with covariance ww⊤ + σ²I. Press Fit from the points and the dashed ellipse is what maximum likelihood recovers from the dots alone: the long axis is the top principal direction, the noise is the short axis' variance, and the arrow length is what is left of the long axis after the noise is taken off.

true model (1σ, 2σ) fitted by maximum likelihood
Things to try. Set σ to its minimum and the dots collapse onto the line: that is plain PCA's picture, where every point is exactly on the subspace. Raise σ until the cloud is nearly round and the fit's angle starts to wobble; with this much noise the direction is hard to read. Shrink the length of w to zero and the cloud becomes round, with nothing to find. Press New sample a few times: the fit moves a little each time because it only sees 150 dots.

2. From a point back to its hidden coordinate

Here the model is fixed (a line at 30°). Drag the black dot anywhere. PCA says: drop a perpendicular onto the line, and that foot is the point's coordinate (orange). Probabilistic PCA gives a whole bell curve for the hidden coordinate instead of a single value; its centre is the blue dot and the blue bar is its two-standard-deviation spread. The coordinate axis along the line is marked in units of z.

PCA foot (σ → 0) PPCA posterior mean and ±2 spread
Things to try. Drag the dot far from the line: the PCA foot follows it all the way out, but the blue mean sits closer to the middle. It is pulled toward 0 because the prior says most points have a small z, and the pull grows with the noise. Set σ to its minimum and the blue and orange dots coincide, with the bar nearly gone. Make w short with large σ, and the blue dot barely moves from the centre however far you drag: with this much noise and so little signal, the position says almost nothing about z.

3. The noise is the variance PCA throws away

Each bar is one eigenvalue of the sample covariance: how much the data spreads along that principal direction. Choose how many directions q to keep. The model's noise variance σ² is the average of the bars you discard (dashed line), and each kept direction contributes its bar minus that floor. Change the preset to see when there is a natural place to cut.

signal kept: λᵢ − σ² noise floor σ² discarded directions
Things to try. With the clear gap, q = 2 puts the dashed line right at the level of the small bars, so the floor describes them well. Push q to 1 and the floor jumps up, because the second large bar is now being counted as noise. Push q to 7 and only one bar is discarded, so the floor is just the last eigenvalue. On the gradual-decay preset no q gives a floor that matches the discarded bars: there is no clean split between signal and noise, which is a statement about the data and not about the method.

Samples are generated in the browser from a fixed seed. Nothing here comes from the paper's data.