Invariance, f-divergences and the Fisher metric, by hand

Companion to the notes on Chapter 3 of Amari's Information Geometry and Its Applications. Back to the notes. Everything here lives on a handful of outcomes, so every divergence is an exact sum and every "probability of a large deviation" an exact multinomial sum: no sampling anywhere. The numbers at the default settings match code/invariant_geometry.py.

1. Merging outcomes: the divergence can only drop, and drops by nothing exactly when the conditionals agree

Take two distributions $p,q$ on three outcomes and merge two of the outcomes into one cell, as if you could no longer tell them apart. For an $f$-divergence $D_f[p:q]=\sum_i p_if(q_i/p_i)$ the value after merging is never larger (Theorem 3.1, by Jensen's inequality), and it is unchanged exactly when $q_i/p_i$ is the same for the merged outcomes: then, inside the cell, $p$ and $q$ split their mass in the same proportions, so nothing about how they differ is lost. Press the button to make that happen. Now switch to the squared Euclidean distance or the non-convex $f$: after the same merge the number goes up, so they are not information-monotone. The variation distance never goes up but can stay put when the conditionals differ (strict convexity is missing).

pqunder each outcome: $u=q/p$

2. The alpha family, and the sign of the linear term in (3.95)

The $\alpha$-divergence is $D_f$ for the function $f_\alpha(u)=\frac4{1-\alpha^2}(1-u^{(1+\alpha)/2})$ of (3.38). To use it on positive measures (where the total masses need not match) the book adds a linear term so that $f(1)=f'(1)=0$: the standard function. The printed (3.95) has the opposite sign of that term. Move $\alpha$ and watch the slope at $u=1$: the standard version has slope $0$, the function (3.38) has $-2/(1-\alpha)$, the printed one $-4/(1-\alpha)$. On probability distributions the linear term cancels and all three agree; make the mass of $n$ differ from 1 and only the standard one stays a divergence (non-negative). The dashed curve is the dual $f^*(u)=uf(1/u)$, which for the standard family is exactly $f_{-\alpha}$.

$m=p=(.5,.3,.2)$ and $n=M\,q$ with $q=(.2,.2,.6)$, so $M=1$ is the case of two probability distributions. For $\alpha=\pm1$ the function (3.38) does not exist and (3.95) reduces to the standard one.

3. Every f-divergence has the Fisher metric as its quadratic form

Expand $D_f[p:p+\varepsilon d]$ for small $\varepsilon$: the first-order term vanishes (because $f'(1)=0$ and $\sum d_i=0$) and the second-order term is $\tfrac12 f''(1)\sum d_i^2/p_i$, which is $f''(1)$ times half the Fisher information of the direction $d$ (3.67)–(3.68). So $D_f/\varepsilon^2$ should level off at that number: the plot shows how fast. The Jensen–Shannon divergence has $f''(1)=1/4$. Two functions fail: $(u-1)^4$ is convex with $f(1)=f'(1)=0$ but $f''(1)=0$, so the limit is $0$ and there is no metric; the variation distance $|1-u|$ is not differentiable at $1$ and $D/\varepsilon^2$ blows up like $1/\varepsilon$.

4. Zero forcing and zero avoiding: what the best approximation does depends on alpha

Approximate the two-humped distribution $p$ below (half $N(10,2^2)$, half $N(28,3^2)$ on 40 points, with a gap in the middle where $p$ is almost $0$) by the best discretised Gaussian, measured by $D_\alpha[p:q]$. For $\alpha\le-1$ the divergence is infinite if $q$ vanishes where $p$ does not, so the best $q$ must cover both humps (zero avoiding, a wide Gaussian that puts mass in the gap). For $\alpha\ge1$ it is infinite if $q$ is positive where $p$ vanishes, so the best $q$ hides inside one hump (zero forcing). The book's statement is about $\pm1$; slide $\alpha$ through $0$ and see where the fit actually jumps.

pbest Gaussian q (grid search over μ, σ)

5. Large deviations: the exponent is the KL divergence to the nearest point of the set

Draw $N$ values of $x\in\{0,1,2\}$ from $p=(.5,.3,.2)$ and let $\hat p$ be the empirical distribution. What is the probability that its mean is at least $a$, i.e. that $\hat p$ lands in the shaded corner $A$? Sanov's lemma and the large deviation theorem (3.55)–(3.59) say it decays like $e^{-NI}$ with $I=\min_{q\in A}\mathrm{KL}[q:p]$, reached at the e-projection $q^*$ of $p$ on $A$, which is the exponential tilt $q^*_i\propto p_ie^{\lambda x_i}$. The dots on the right are exact multinomial sums; the curve is the exponent corrected by the polynomial factor (Bahadur–Rao). Increase $N$: the dots approach $I$ from above, like $\log N/N$.

6. Theorem 3.2: the Fisher information integrated along either geodesic is the symmetrised KL

Join $p$ and $q$ by the e-geodesic $p_t\propto p^{1-t}q^t$ and by the m-geodesic $(1-t)p+tq$, and measure the Fisher information $g(t)$ of the moving point at each $t$. The theorem says the area under $g_e$ and the area under $g_m$ both equal $D[p:q]+D[q:p]$. The book prints half of that sum on the left of (3.66); the readout shows the full sum matching and half of it not. The two curves are very different (the m-geodesic's information is large where $p_t$ is nearly empty), yet their areas agree. The last number is $s^2$, the squared Fisher–Rao distance: always at most the area (Cauchy–Schwarz), so half the sum could never be equal to it.

7. Chentsov: the Fisher weight is the only one that survives splitting an outcome

Split one outcome of $q$ into two with proportions $r=(\rho,1-\rho)$, so a distribution on $2$ outcomes becomes one on $3$ (a Markov embedding: you add randomness that does not depend on the parameter, so no information is gained). An invariant metric must give the pushed-forward vector the same length. Take a family of metrics $\sum_iZ_i^2/p_i^{\,a}$ ($a=1$ is the Fisher metric, $a=0$ the Euclidean one). The vector $e_1$ that carries the split cell has squared length multiplied by $\sum_ir_i^{2-a}$, equal to $1$ for every split only at $a=1$. Add the term $B(\sum Z_i)^2$ and nothing changes at $a=1$ for any $B$ (on tangent vectors, with $\sum Z_i=0$, it is invisible): on positive measures the invariant metric is not unique, which the book's proof glosses over.

8. The simplex is a sphere of radius 2, and scale makes it grow

In the coordinates $\xi_i=2\sqrt{p_i}$ the Fisher metric of the simplex becomes the ordinary Euclidean one and $\sum\xi_i^2=4$: the three-outcome simplex is an octant of a sphere of radius $2$, with Gauss curvature $1/4$. The Fisher–Rao geodesic is a great circle; the m-geodesic (straight in $p$) and the e-geodesic are longer, and the Hellinger divergence $D_0=4(1-\cos(s/2))$ is half the squared chord. Turn the view with the sliders. On positive measures $m=Mp$ the radius becomes $2\sqrt M$: the metric splits as $dM^2/M+M\,g_{\rm Fisher}$, distances between $Mp$ and $Mq$ scale like $\sqrt M$ and $D_f[Mp:Mq]=M\,D_f[p:q]$.

Fisher–Rao geodesic (great circle)m-geodesice-geodesicchord