Dual connections, by hand

Companion to the notes on Chapter 6 of Amari's Information Geometry and Its Applications. Back to the notes. Two running examples: the Gaussian manifold in the chart $(\mu,\sigma)$, whose e-, m- and Levi-Civita connections are mixtures of one another, and the probability simplex on three outcomes in the chart $(p_1,p_2)$. The $\alpha$-connection is $\Gamma^{(\alpha)}=\Gamma^{(0)}-\tfrac\alpha2T$, with $\alpha=+1$ the e-connection and $\alpha=-1$ the m-connection. Every widget computes with the same mathematics as code/dual_connections.py (finite differences, RK4, Newton), in the browser, and the default state reproduces the numbers printed there.

1. Duality is conservation of the inner product

Take a loop in the Gaussian half-plane, a vector $A$ and a vector $B$. Carry $A$ along the loop by $\nabla^{(\alpha)}$ and $B$ by the dual connection $\nabla^{(-\alpha)}$: their Fisher inner product $\langle A,B\rangle$ never changes (equation (6.1)), although neither vector comes back to itself when the pair is curved. Carry both by the same connection and the inner product wanders, unless that connection is Levi-Civita ($\alpha=0$). At $\alpha=\pm1$ the pair is flat and both vectors do come home. Try $\alpha$ near $0.6$, shrink the loop (the wander and the holonomy scale with its area), then switch the pairing.

2. From a divergence to a geometry

A divergence $D[p:q]$ has a quadratic part, which gives the metric $g_{ij}=-D_{i;j}$, and a cubic part, which gives the pair of connections $\Gamma_{ijk}=-D_{ij;k}$ and $\Gamma^*_{ijk}=-D_{k;ij}$ (6.22)–(6.24). Pick a divergence between two nearby Gaussians: the table is computed by finite differences of $D$ alone and compared with the closed forms. $\mathrm{KL}[p{:}q]$ gives $\alpha=-1$, $\mathrm{KL}[q{:}p]$ gives $\alpha=+1$, the $\alpha$-divergence gives $\alpha$, half the squared distance gives $0$. Adding a quartic term changes nothing (6.51)–(6.54); adding a cubic term $\tfrac12\delta_\mu^3$ changes $\Gamma$ and $\Gamma^*$ by $\mp3$.

3. The α-geodesics of the probability triangle, and their curvature

Leave a point of the triangle $\{p_1+p_2+p_3=1\}$ with a fixed velocity and follow the geodesics of $\nabla^{(\alpha)}$ for $\alpha$ (thick), $-\alpha$ (dashed) and $0$ (thin, a great circle of the sphere of radius 2). The m-geodesic ($\alpha=-1$) is a straight line, the e-geodesic ($\alpha=+1$) is a geometric mixture. The curvature coefficient $K^{(\alpha)}$ computed from the connection coefficients is $(1-\alpha^2)/4$: flat at $\pm1$, a sphere at $0$, and the same number for $\nabla^{(\alpha)}$ and $\nabla^{(-\alpha)}$ (compare the identity $R_{ijkl}+R^*_{ijlk}=0$ in the notes).

4. The canonical divergence of a general dual pair, and the projection condition

For a pair $(\nabla,\nabla^*)$ the book builds a divergence from the geodesics: $\tilde D=\int_0^1t\,g(\dot\xi,\dot\xi)\,dt$ along the $\nabla$-geodesic from $p$ to $q$, $\tilde D^*$ with weight $1-t$ along the $\nabla^*$-geodesic, and $D=\tfrac12(\tilde D+\tilde D^*)$ (6.77)–(6.80); here the geodesic boundary-value problem is solved by Newton shooting. The readout compares it with the $\alpha$-divergences and half the squared distance, and measures the condition (6.81) under which projection onto a submanifold is geodesic: the angle between the normal $g^{-1}\partial_q D[p{:}q]$ of the divergence ball and the geodesic direction $-\exp_q^{-1}p$. On the simplex the $\alpha$-divergence satisfies it exactly; on the Gaussians it misses by a degree.

5. Mixed coordinates: rates against interaction

For two binary neurons the distribution has the dual coordinates $\eta=(\eta_1,\eta_2,\eta_{12})$ and $\theta=(\theta^1,\theta^2,\theta^{12})$. The mixed chart $(\eta_1,\eta_2;\theta^{12})$ has an orthogonal metric (left, lower matrix), with blocks $(G_{11})^{-1}$ and a Schur complement; the chart with the covariance $v=\eta_{12}-\eta_1\eta_2$ does not (upper matrix). Moving the sliders for $p$ and $q$, the intermediate distribution $r=(\eta\text{ of }p;\theta^{12}\text{ of }q)$ makes $\mathrm{KL}[p{:}q]=\mathrm{KL}[p{:}r]+\mathrm{KL}[r{:}q]$ exact, the Pythagorean theorem of (6.96) and (6.104); the other intermediate $r'=(\eta\text{ of }q;\theta^{12}\text{ of }p)$ decomposes the reversed divergence $\mathrm{KL}[q{:}p]$ instead.

6. Integrated information of a Gaussian channel

A channel $y=Ax+\varepsilon$ with $x\sim N(0,I)$ and noise correlation $\rho$. The split model $M_S$ (bars: M) removes the direct links $x_1y_2$, $x_2y_1$; its m-projection gives $GI$, and Ay's smaller model $M_S'$ (bars: M′; no $y_1y_2$ link either) gives $GI'$. The notes show $GI'\ge GI$, the reverse of the printed (6.146). Set the cross-talk to zero and raise $\rho$: there is no interaction between terminals, yet $GI$ and $GI'$ are positive and the projection $\hat A$ is not diagonal; only the curved model $M_S''$ ($\hat A$ diagonal, noise free) gives $GI''=0$. Set $A=0$: $I(X{:}Y)=0$ but $GI'>0$, the example (6.152).

7. Input–output tables: margins, interactions and RAS

A $3\times3$ transaction table $A$ has the e-coordinates $L_{ij}=\log A_{ij}$ and the m-coordinates $A_{ij}$. Replace the last row and column by the sums and the dual coordinates become the margins $(A_{i\cdot},A_{\cdot j},A_{\cdot\cdot})$ and the interactions $\tilde L_{ij}=\log\frac{A_{ij}A_{33}}{A_{i3}A_{3j}}$, orthogonal to each other (6.173)–(6.175). Scale rows by $\mu_i$ and columns by $\lambda_j$ (the RAS step, (6.180)): $\tilde L$ does not move, but the margins change by factors that are not $\mu_i$ and $\lambda_j$, and $S_{ij}=\log\frac{A_{ij}}{A_{i\cdot}A_{\cdot j}}$ of (6.177) does move. The bars decompose $D[A{:}B]=\sum B\log\frac BA-B+A$ for a fixed second table $B$ (rescaled to the total of $A$) through the RAS fit of $A$ to the margins of $B$.