Fisher information, by hand

Companion to the foundations page The Fisher information matrix: how sharply data pin down parameters. Back to the notes.

1. How much does one flip tell you?

A coin with probability $\theta$ of heads. The left plot is the average log-likelihood of a guess $t$ when the data come from $\theta$: its peak is at $t=\theta$, and its curvature there is the Fisher information $I(\theta)=1/(\theta(1-\theta))$ (dashed parabola). The right plot simulates 4000 experiments of $N$ flips each and histograms the estimate, the fraction of heads, against the normal curve with variance $1/(N I)$.

average log-likelihoodparabola of curvature $I(\theta)$
estimates from 4000 experimentsnormal, variance $1/(NI)$

2. The distance between two coins, three ways

Two coins, $\theta_1$ and $\theta_2$. Their distance in $\theta$ and in log-odds $\eta$ depends on the coordinates. The Fisher–Rao distance $2\lvert\arcsin\sqrt{\theta_1}-\arcsin\sqrt{\theta_2}\rvert$ does not. The ticks on the axis are spaced $0.1$ apart in Fisher–Rao distance: they crowd near 0 and 1, where coins are easy to tell apart.

Keep the gap at $0.1$ and slide both coins to the left edge: the Fisher–Rao distance more than doubles, from $0.201$ for $0.5\to0.6$ to $0.476$ for $0.01\to0.11$, and the KL divergence grows with it. The local approximation $\frac12 I\,\Delta\theta^2$ tracks the KL only while the step is small compared with $1/\sqrt{I}$.

3. Gaussians live in a hyperbolic half-plane

Each point of the plane is a normal distribution $\mathcal N(\mu,\sigma^2)$. Drag the two orange points. The dashed segment is the straight path; the blue arch is the geodesic, the shortest path in the Fisher–Rao metric $ds^2=(d\mu^2+2\,d\sigma^2)/\sigma^2$. Slide along it to see the distributions in between.

Horizontal: mean $\mu$. Vertical: standard deviation $\sigma$. The geodesic rises: widening the distribution first makes the mean cheap to move.
the two end pointsthe point on the geodesic

4. Natural gradient against plain gradient

Fitting $\mathcal N(\mu,\sigma^2)$ to data with mean $2$ and standard deviation $0.5$. Shading shows the negative log-likelihood. Click anywhere to start all three optimisers there: plain gradient descent in $(\mu,\sigma)$, plain gradient descent in $(\mu,\log\sigma)$, and natural gradient, which rescales the gradient by $F^{-1}$ and so takes the same path in any coordinates.

plain, in $(\mu,\sigma)$plain, in $(\mu,\log\sigma)$natural gradient

Everything runs in your browser. The same computations, with every number quoted in the notes, are in code/fisher.py.