Gram Matrices, Hands On

← text notes · Foundations › Linear algebra. Three widgets: a Gram matrix as a table of dot products, a kernel Gram matrix and its bandwidth, and the contrastive similarity matrix.

1. Points in, dot products out

Each point x_i is a row of X. The Gram matrix G = XXᵀ holds every dot product: G_ij = x_i · x_j. Its diagonal is the squared lengths. Drag the points and watch the table change. Then move the rotate slider, which turns the whole cloud: the points move and the Gram matrix does not change at all.

The eigenvalues underneath are never negative. Line the points up along one line and all but one of them drop to zero: the rank of G is the number of independent directions the points span.

Things to try. Drag two points until they are at right angles through the origin: their entry goes to 0. Drag a point further out: its whole row and column scale up. Tick centre: the mean is subtracted first, the cross marks the new origin, and every row of HGH sums to zero.

2. RBF kernel: how far does "similar" reach?

Swap the dot product for the Gaussian kernel K_ij = exp(−‖x_i − x_j‖² / 2σ²). It is still a Gram matrix, of some hidden feature vectors φ(x_i), so it is still PSD. The circle around the highlighted point has radius σ: roughly, points inside it count as "similar".

Two clusters of eight points each, sorted so cluster A comes first. At a good σ the matrix is two bright blocks. Too small and it collapses to the identity; too large and it goes to all ones. Hover a point to highlight its row.

The readout's "block contrast" is mean K within clusters minus mean K between them. It peaks at a middle σ and falls to 0 at both ends. MMD and HSIC are built from exactly these block means, which is why they depend so much on σ.

3. The contrastive Gram matrix

InfoNCE on a batch: four anchors a_k and their four positives p_k (two views of the same image), all normalised to the unit circle. The logit matrix is the Gram matrix between anchors and positives divided by a temperature, S = A Pᵀ / τ, i.e. cosine similarity over τ. Each row goes through a softmax, and the loss wants the diagonal (the true pair) to win.

Drag the dots around the circle (anchors are filled, positives are rings of the same colour). Lower τ sharpens every row: small differences in angle become confident decisions, and the hardest negative dominates the loss.

"Align pairs" puts each positive on its anchor and spreads the pairs evenly: the alignment and uniformity of Wang & Isola at once. The diagonal goes bright and the loss approaches its floor, which is above 0 unless τ is small, because the negatives are never infinitely far away on a circle.