HSIC, Hands On

← text notes · Foundations › Linear algebra. Five widgets for the Hilbert–Schmidt independence criterion: where correlation fails, why more features help, HSIC as agreement between two Gram matrices, the permutation test and its bandwidth, and CKA for comparing representations.

1. Correlation misses it, HSIC does not

Pick a data set. Two tests look for dependence between x and y. The first uses a linear kernel, which is exactly the same test as Pearson correlation: it only sees straight-line trends. The second uses a Gaussian kernel, which sees any dependence given enough points. Both are judged the same way: shuffle y 499 times, and ask how often a shuffle scores as high as the real data. That fraction is the p-value.

A longer bar means stronger evidence. Past the dashed line means p < 0.05. Choose your own points to click in any shape you like.

Things to try. Parabola, circle, funnel, X shape: correlation is near 0 and the linear bar stays short, while the Gaussian bar passes the line. Slide n down to 10–15 and press New sample a few times: the Gaussian test now misses about half the time or more. Dependence you can see by eye can still be too little evidence. Checkerboard: neither test finds it, because the median bandwidth is wider than a square (section 4 fixes that). Independent: both bars stay short, and about one sample in twenty will cross the line by chance. That is what p < 0.05 means.

2. More features, more chances

If x and y are independent, then no feature of x is correlated with any feature of y. So give each variable several features, here its powers x, x², …, and look at every correlation between a feature of x and a feature of y. Degree 1 is plain correlation, a single cell. Raise the degree and new cells appear. Each kind of dependence lights up its own cell.

HSIC with a polynomial kernel adds up the squares of this table. The Gaussian kernel does the same with every power at once (§3 of the notes).

Parabola: nothing at degree 1, then x² against y lights up at degree 2. Circle: x² against y², negative, because x² + y² ≈ 1. Sine wave: the powers up to 4 never catch it well, because a wiggle needs higher powers. That is the case for a kernel with all the powers in it. One thing to notice: the sum of squares grows with the degree even for the plain line. The features x and x³ are nearly the same curve, so the one straight-line dependence gets counted several times. That is why a raw HSIC value only means something next to its shuffles.

3. Two similarity tables, and whether they agree

Forty points, sorted by x. K̃ says how much more alike each pair of samples is in x than a typical pair: blue for more alike, orange for less. L̃ says the same for y. The right-hand table multiplies them cell by cell: blue where x and y agree about a pair, orange where they disagree. HSIC is the average of that last table.

Hover over any table to see which pair a cell is. Then press Shuffle y. The x-table does not change, but the y-table is scrambled, the agreement table turns to noise, and its average drops.

For the parabola, look at the anti-diagonal of L̃: points at x and −x have the same y, so y calls them alike while x calls them far apart. Those cells are orange in the agreement table. The fold makes some pairs disagree, and HSIC comes out positive only because the neighbouring pairs near the diagonal agree more.

4. The shuffle test, and the bandwidth

The left chart is the test itself. Grey is HSIC for 400 shuffles of y (the "independent world" built from your own data), and red is the real data. The right chart repeats the test at every bandwidth σ, from a thousandth of the median distance to a thousand times it. Its height says how many shuffle standard deviations the real data sit above the shuffle average.

The checkerboard starts at the median bandwidth, where the test sees nothing. Drag σ to the left. Then try the parabola, whose curve peaks near the median, and notice that far to the right every curve settles on the dashed line: the linear kernel's answer.

Far left, every point is similar only to itself, so every shuffle gives the same HSIC and the grey histogram collapses to a spike. Far right, the Gaussian kernel behaves like the linear one and goes blind to anything that is not a straight line. The median heuristic is a guess at the middle, and the checkerboard shows it can guess wrong.

5. CKA: comparing two representations

Think of the left cloud as one network's embeddings of 90 inputs and the right cloud as another network's embeddings of the same inputs (colours mark which is which). CKA is HSIC normalised to lie between 0 and 1. It is the cosine of the angle between the two centred Gram matrices. Transform the right-hand representation and see what CKA ignores and what it notices.

Rotate and scale as much as you like: both CKAs stay at 1, because Gram matrices do not see orientation and the normalisation removes scale. Squashing an axis lowers linear CKA only as far as that axis carried variance. Warping and noise lower both. Shuffling which point is which sends both near 0: CKA compares the same inputs across two representations, so the pairing is everything.