← text notes · Three widgets for the panel-of-judges picture: one judge and the gap it earns, how four distances behave as two distributions move apart, and the MMD witness function as the kernel bandwidth changes. Everything is computed in your browser.
1. The judge: one function, one gap
Two distributions P (blue) and Q (orange) live on a line, say one coordinate of an embedding. A judge is a function h that scores every point. The gap is EPh − EQh: the average score under P minus the average score under Q. Here the judge is a ramp, h(x) = clip(L(x − c), −a, a), which climbs with slope L around the centre c and then flattens at height ±a. The Dudley budget splits 1 between the two limits: a (how tall) plus L (how steep) = 1. The lower panel shows the two CDFs' difference F − G; a ramp's gap is L times the signed area under that curve over the ramp's sloping window.
PQyour ramp judge h (right axis)best judge of any shape at this budget
F − G (P's CDF minus Q's)the ramp's sloping window
Things to try. On the default preset drag the centre c and watch the gap peak at c = 0.5, halfway between the means; then press find best ramp: the best gap (about 0.343) sits at a ≈ 0.57, well below W1 = 1, because a ramp with a sup-norm budget can only read a window of width 2a/L from the CDF gap. Tick ignore the budget: the ramp becomes a plain 1-Lipschitz judge, and with the clip level at its largest the gap approaches the difference of the means, which for this pair equals W1. Now choose Q two-humped, same mean and tick also show the best judge of any shape. The best ramp earns about half of what a zig-zag judge earns, because the CDF gap has two humps of opposite sign, and a ramp can read only one of them. In the no-budget mode the zig-zag judge (the derivative of h is the sign of G − F) earns exactly W1.
2. Distance versus separation
Move Q away from P and watch five numbers. TV can never exceed 1. Dudley (sup-norm plus Lipschitz constant at most 1) can never exceed 2. MMD with a Gaussian kernel has a ceiling too, and it is set by the bandwidth and the widths of the distributions. W1 and KL keep growing. The first tab shifts N(0, σ²) to N(m, σ²). The second tab uses two point masses, where every number has a closed form and KL is infinite from the first millimetre of separation.
TVDudley (sum norm)W1KL(P‖Q), Gaussians onlyMMD, Gaussian kernelceiling 2 for Dudley
Things to try. Gaussians, σ = 1: at m = 1 the four numbers are TV = 0.383, Dudley = 0.343, W1 = 1, KL = 0.5. Pull m to 6: TV is stuck at 1, Dudley creeps towards 2, W1 = 6, KL = 18 (off the chart at y ≤ 3). Now lower σ to 0.05 with the shift at 1: the two bumps no longer overlap and the Gaussian curves collapse onto the point-mass curves of the second tab. The Dudley curve becomes 2D/(D + 2), and KL = m²/(2σ²) is enormous. That is the sense in which KL "blows up" and TV "saturates" while W1 and Dudley keep a geometry. Point masses: Dudley = 2D/(D + 2) is below W1 = D everywhere, and equals W1 only in the limit D → 0; it crosses TV at D = 2 and tends to 2 as D grows. The Gaussian-kernel MMD saturates at √2 for any bandwidth. Shrinking the kernel bandwidth pulls the saturation in: the kernel sees separation only at its own length scale.
3. The MMD witness and the bandwidth
The judges of MMD are the functions in the unit ball of the kernel's function space. The best one has a closed form: the witnessw(t) = (1/n)Σik(xi, t) − (1/m)Σjk(yj, t), the P-sample "kernel density" minus the Q-sample "kernel density" (each sample point drops a Gaussian bump of width σ). Scaled to unit norm it is the judge f* = w/‖w‖, and the gap it earns is MMD = ‖w‖. Blue regions are where the P sample has more mass than Q, orange where Q has more. The lower panel plots the unbiased MMD² estimate for this sample against the bandwidth.
w > 0: P heavierw < 0: Q heavierP sampleQ sample
unbiased MMD² of this samplepopulation MMD²current σ
Things to try. Means differ: at σ around 1 to 2 the witness is a clean blue-to-orange step and MMD is largest. Drag σ down to 0.05: the witness turns into a spike at every sample point, the population MMD² falls towards zero (the kernel sees nothing beyond its own footprint), and the sample MMD² is just noise. Drag σ up to 20: MMD² ≈ (difference of means)²/σ², so the kernel has become a linear one and sees only the first moment. Switch to same mean, Q wider: the large-σ tail of the bottom curve falls much faster than for the shifted pair, because a Gaussian kernel with a huge bandwidth cannot see a difference in width. Same mean and variance, Q two-humped is harder still: only a mid-range bandwidth finds it. On same law the estimate hovers around zero and the permutation test returns a large p-value. On means differ it sits at its floor, 1/501, at every bandwidth here. On the other two, the p-value depends on σ and on the draw: with 40 points per sample the two-humped pair is often missed, so press resample a few times at σ ≈ 0.5, where the population curve peaks, and again at σ = 3, where it has died away.