The dually flat structure, by hand

Companion to the notes on Chapter 1 of Amari's Information Geometry and Its Applications. Back to the notes. Everything here lives on the simplest non-trivial manifold: the distributions $p=(p_0,p_1,p_2)$ on three outcomes (think: a 3-class softmax output). The two coordinate systems of the chapter are the logits $\theta_i=\log(p_i/p_0)$ and the probabilities $\eta_i=p_i$. Drag the grey points inside the triangles, or focus a triangle with Tab, press P or Q to choose a point and move it with the arrow keys.

1. Two kinds of straight line

A path that is straight in $\theta$ is an e-geodesic (blue); one that is straight in $\eta$ is an m-geodesic (orange). Both join $P$ to $Q$. The green curve is the shortest path in the Fisher metric, the usual Riemannian geodesic, which is neither.

2. The Pythagorean theorem

Take the m-geodesic from $P$ to $Q$. Leave $Q$ along an e-geodesic whose direction is orthogonal to it (in the Fisher metric) and stop at $R$. Then $D(R{:}P)=D(Q{:}P)+D(R{:}Q)$ exactly, with $D$ the Bregman divergence of $\psi$, which on these coordinates is a KL divergence with its arguments reversed. Tilt the direction away from orthogonal and the sum stops matching, by exactly the amount the proof of Theorem 1.2 predicts.

3. Projecting onto a submanifold

Given $P$ and a curve $S$, which point of $S$ is closest in divergence? Pick the divergence, pick $S$, and drag $P$. For an e-flat $S$ (straight in $\theta$) and $\mathrm{KL}[P{:}R]$ there is always exactly one critical point on $S$, and it is the minimum. For a curved $S$ there can be several, and the orthogonality condition does not tell them apart.