The dually flat structure, by hand

Companion to the notes on Chapter 1 of Amari's Information Geometry and Its Applications. Back to the notes. After the first widget, everything lives on the simplest non-trivial manifold: the distributions $p=(p_0,p_1,p_2)$ on three outcomes (think: a 3-class softmax output). The two coordinate systems of the chapter are the logits $\theta_i=\log(p_i/p_0)$ and the probabilities $\eta_i=p_i$. Drag the grey points inside the triangles, or focus a triangle with Tab, press P or Q to choose a point and move it with the arrow keys.

1. One manifold, three charts: the Gaussians

The points are Gaussian curves; a chart is a way to name one with two numbers. Move $\mu$ and $\sigma$: the dot is the same Gaussian in all three charts, and the grid of chart 1 (blue: constant $\mu$, orange: constant $\sigma$) is bent by the change of labels. The moments are $(m_1,m_2)=(\mu,\mu^2+\sigma^2)$ and the natural parameters are $\theta=(\mu/\sigma^2,\,-1/(2\sigma^2))$.

2. A divergence, concretely: KL between two Gaussians

$p=N(0,1)$ is fixed; move $q$. The top-right bars are the integrand $p\log(p/q)$ (green positive, red negative) whose total is $\mathrm{KL}[p{:}q]$. It is never negative, it vanishes only when $q=p$, it is quadratic near $q=p$, and it is not symmetric: change $\sigma_q$ away from 1 and the two orders split.

3. Tangent space and the Riemannian metric on the Gaussians

A tangent vector at a Gaussian is a velocity $(\dot\mu,\dot\sigma)$: how fast its mean and width are changing. The metric measures its length by $|v|^2=(\dot\mu^2+2\dot\sigma^2)/\sigma^2$, so the same Euclidean step is long where the bell is narrow and short where it is wide. The ellipses (all drawn at the same scale: they are the steps of Riemannian length 0.15) show the metric at a grid of points; the bold one is at your chosen point. The orange arrow is a step $d$; the right plot compares the Gaussian at the start and at the end of the step, and the readout gives the divergence that step costs next to the quadratic $\tfrac12 ds^2$.

4. What "positive-definite Hessian" means

Near a point a function is its tangent plane plus a quadratic bowl $\tfrac12 d^\top H d$, and for a Bregman divergence that bowl is all that is left. Drag the entries of $H=\begin{bmatrix}a&b\\b&c\end{bmatrix}$. The left picture shades the sign of $d^\top Hd$ over all steps $d$ (blue positive, orange negative) with the unit ellipse $d^\top Hd=1$; the right plots the curvature along each direction $\varphi$. $H$ is positive-definite exactly when the curve never touches zero: an ellipse, two positive eigenvalues. Touch zero and there is a flat direction; go below and it is a saddle.

5. The Bregman divergence as a gap above a tangent

Choose a convex $\psi$. Draw its tangent at the base point $x_0$ (solid orange); the vertical gap at the probe $x$ is $D_\psi[x{:}x_0]$. Draw the tangent at $x$ instead (dashed blue) and measure at $x_0$ and you get the other order, $D_\psi[x_0{:}x]$. The right plot shows both as $x$ moves, with the parabola $\tfrac12\psi''(x_0)(x-x_0)^2$ that they both hug near $x_0$.

6. The Legendre transform: slope as a coordinate

Pick a convex $\psi$ and a point $\theta_0$. Its tangent has slope $\eta=\psi'(\theta_0)$ and crosses the vertical axis $\theta=0$ at height $\psi(\theta_0)-\eta\theta_0=-\psi^*(\eta)$. So the dual function is minus the intercept. Slide $\theta_0$ and the right plot traces $\psi^*$ point by point; its own tangent there has slope $\theta_0$, so the roles of point and slope are exchanged.

7. Two kinds of straight line

A path that is straight in $\theta$ is an e-geodesic (blue); one that is straight in $\eta$ is an m-geodesic (orange). Both join $P$ to $Q$. The green curve is the shortest path in the Fisher metric, the usual Riemannian geodesic, which is neither.

8. Pythagoras on the Gaussians: the 3-4-5 triangle's cousin

In the plane, $P=(0,0)$, $Q=(3,0)$, $R=(3,4)$ with $D=\tfrac12\|\cdot\|^2$ gives $4.5+8=12.5$. Here $P=N(0,1)$ is fixed and $Q$ is yours. The orange curve is the m-geodesic $P\to Q$ (straight in $\eta=(\mu,\mu^2+\sigma^2)$); the blue curve is the e-geodesic through $Q$ (straight in $\theta=(\mu/\sigma^2,-1/(2\sigma^2))$), leaving $Q$ in the direction whose pairing with $\eta_Q-\eta_P$ is zero. Move $R$ along it with $t$. The bars stack $\mathrm{KL}[P{:}Q]+\mathrm{KL}[Q{:}R]$ against the direct $\mathrm{KL}[P{:}R]$: they have the same length at zero tilt, for any $t$. Tilt the second leg and the gap is exactly the pairing $(\theta_Q-\theta_R)\cdot(\eta_Q-\eta_P)$.

9. The Pythagorean theorem on the probability triangle

Take the m-geodesic from $P$ to $Q$. Leave $Q$ along an e-geodesic whose direction is orthogonal to it (in the Fisher metric) and stop at $R$. Then $D(R{:}P)=D(Q{:}P)+D(R{:}Q)$ exactly, with $D$ the Bregman divergence of $\psi$, which on these coordinates is a KL divergence with its arguments reversed. Tilt the direction away from orthogonal and the sum stops matching, by exactly the amount the proof of Theorem 1.2 predicts.

10. Projection onto the zero-mean Gaussians

$S=\{N(0,\sigma^2)\}$ is the vertical axis. Drag $P$. The orange curve is the m-geodesic from $P$ (straight in $\eta$): it keeps $\mu^2+\sigma^2$ fixed, so it is a circle arc, and it lands on $S$ at $\hat R$ with $\hat\sigma^2=\mu_P^2+\sigma_P^2$ (the second moment is kept: this is maximum likelihood). The blue line is the e-geodesic (straight in $\theta$): horizontal, landing at the foot that keeps $P$'s width. The right plot shows $\mathrm{KL}[P{:}R]$ along $S$ and the Pythagorean split $\mathrm{KL}[P{:}\hat R]+\mathrm{KL}[\hat R{:}R]$ (they coincide), and the other order, with its own minimum. Slide $R$ along $S$ to test the split at any member.

11. Projecting onto a submanifold

Given $P$ and a curve $S$, which point of $S$ is closest in divergence? Pick the divergence, pick $S$, and drag $P$. For an e-flat $S$ (straight in $\theta$) and $\mathrm{KL}[P{:}R]$ there is always exactly one critical point on $S$, and it is the minimum. For a curved $S$ there can be several, and the orthogonality condition does not tell them apart.