Natural gradient, by hand

Companion to the notes on Chapter 12 of Amari's Information Geometry and Its Applications. Back to the notes. Seven small experiments on the chapter's two halves: how the natural gradient behaves (steepest descent in a metric, online efficiency, saturation, the learning constant) and what happens near the singular regions of a two-hidden-unit perceptron $f(x)=v_1\varphi(w_1x)+v_2\varphi(w_2x)$ with $\varphi(u)=\operatorname{erf}(u/\sqrt2)$ and input $x\sim N(0,1)$. Averages over $x$ are done with closed forms (arcsin kernels), so the learning flows are the averaged equations (12.129), not simulations. The numbers match code/natural_gradient.py (exactly for the deterministic parts, up to Monte Carlo error for the buttons marked as simulations).

1. Steepest descent depends on the metric

A step of length $\epsilon$ in the metric $G$ means $a^{\mathsf T}Ga=1$: an ellipse, not a circle. Of all steps on it, the one that lowers $L$ most is $G^{-1}\nabla L$ (the natural gradient, blue); the plain gradient (orange), rescaled to the same $G$-length, is only best when $G$ is a multiple of the identity or $\nabla L$ is an eigenvector. Move the sliders and watch the brute-force maximum over 3600 directions agree with $\sqrt{\nabla L^{\mathsf T}G^{-1}\nabla L}$. Switching the chart to $\xi'=J\xi$ with $J=\left[\begin{smallmatrix}1&0.3\\0&2\end{smallmatrix}\right]$ recomputes both directions in the new coordinates and maps them back: the natural gradient does not move, the plain gradient does (it is a covector, the natural gradient a vector).

2. Online learning: what the learning constant does to the error

For the Gaussian linear model $y=\xi\cdot x+\varepsilon$ with input covariance $G=\operatorname{diag}(1,0.04)$ the error covariance of online learning obeys an exact recursion (Isserlis' formula for the fourth moment of $x$), so no sampling is needed. The table shows $t\,\lambda_a\operatorname{Var}_a$, the ratio to the Cramér–Rao value $1/(t\lambda_a)$. With the natural gradient and $\eta_t=c/t$ the limit is $c^2/(2c-1)$ for $c>1/2$, equal to $1$ only for $c=1$ (Theorem 12.1); below $c=1/2$ it never settles. With the plain gradient the factor in direction $a$ is $(c\lambda_a)^2/(2c\lambda_a-1)$: the flat direction needs $c>12.5$ to have a $1/t$ rate at all and $c=25$ to be efficient, and then the steep one pays $12.8$. Try $c_{\rm GD}=1$: the flat direction's error decays like $t^{-0.08}$.

3. Stochastic relaxation: the natural gradient is a constant vector

To minimise $\mathbb E_p[f]$ over distributions on a finite set, write $p$ in the natural parameters $\theta_i=\log(p_i/p_0)$. The expected cost is linear in the expectation parameters $\eta_i=p_i$ and $G=\partial\eta/\partial\theta$, so the natural gradient is exactly $f_i-f_0$, whatever $p$ is: the flow is $p_t\propto p_0e^{-tf}$, a straight line in $\theta$. The plain gradient $p_j(f_j-L)$ carries the factor $p_j$, which dies near a vertex. Start close to the wrong vertex and compare the times to give the best outcome probability $0.9$ (outcome 0 costs $0$).

4. Saturation: the natural gradient does not vanish, but its step explodes

One erf neuron $f=\varphi(wx)$, true weight $w_0$, noise variance 1. The Fisher information $G(w)=\mathbb E[\varphi'(wx)^2x^2]$ falls like $w^{-3}$, and so does the plain gradient's squared size $\bar G$. The natural gradient's Riemannian size $\operatorname{tr}(\bar GG^{-1})$ never drops below $k=1$ (Theorem 12.2), which is why it escapes saturation (the averaged iteration from $w=4$ needs 4 steps against 124). But its length in $w$ units, $\sqrt{\bar G}/G$, grows like $w^{3/2}$: a single online sample at $w=8$ moves $w$ by about 56 units for a gain of 1. Run the online simulation and watch a gain of $0.2$ fling the runs away.

5. The Fisher matrix near the overlap singularity

Two hidden units with the same input weight, $w_1=w_2$, compute $(v_1+v_2)\varphi(wx)$ whatever $v_1-v_2$ is: the Fisher matrix of $(w_1,w_2,v_1,v_2)$ loses rank. In the chart $u=w_2-w_1$, $z=\frac{v_1-v_2}{v_1+v_2}$, $s=\frac{v_1w_1+v_2w_2}{v_1+v_2}$, $r=v_1+v_2$ the book's $R_o$ is $u=0$. The plot shows the four eigenvalues against $u$ at the current $(z,s,r)$: two stay of order one, the next falls like $u^2$ and the smallest like $u^6$ (determinant $u^8$): the book says only that $G^{-1}$ diverges. Tick on $R_o$ to put $u$ exactly $0$ and see two zero eigenvalues (the null space is two-dimensional although $R_o$ is a line). The dashed curves are a layer-wise block-diagonal approximation, which keeps the null vectors but is only $O(u^2)$ small.

6. Plateau: plain against natural gradient

Run the averaged flows (12.129) of the two-neuron perceptron from a point near $R_o$. Teacher inside $R$ (one neuron): the plain flow slows down like $u\sim t^{-1/2}$, $L\sim t^{-2}$ (critical slowdown); the natural flow has $L\sim e^{-2t}$ and ends on $R_e$ ($z\to1$), not on $R_o$. Teacher outside $R$, $K>0$: the stable part of $R_o$ is $\lvert z\rvert>1$; from there the plain flow stays at the plateau value $L_1$ for ever, from $\lvert z\rvert<1$ it leaves after a time of order $\ln(1/u_0)$; the natural flow leaves in about $0.4$ time units, from $u_0=0.3$ and $u_0=0.1$ alike. For $K<0$ the roles of $\lvert z\rvert<1$ and $>1$ swap. The natural flow started on the stable part is not integrated here (the parameters make huge excursions, see the notes), and for $u_0<0.1$ my integrator fails or does not converge (the Fisher matrix is too close to singular for double precision), so the natural flow is only run from $u_0\ge0.1$.

7. The adaptive learning constant

The rule $\eta_{t+1}=\eta_t\exp\{\alpha[\beta\,l_t-\eta_t]\}$ raises the learning constant when the loss is large. Its averaged equations are $\dot e=-2\eta e$, $\dot\eta=\alpha\beta\eta e-\alpha\eta^2$, and the book states $e_t\to\frac1\beta(\frac12-\frac1\alpha)\frac1t$, $\eta_t\to\frac1{2t}$. That needs $\alpha>2$: for $\alpha\le2$ the right side is $\le0$ although $e$ is a square, and what happens instead is $\eta_t\approx1/(\alpha t)$ with $e_t$ decaying faster than $1/t$. The table integrates the equations. The simulation button runs the stochastic rule on $y=\xi x+\text{noise}$ with $\beta=0.05$: without noise it follows the averaged analysis, with noise the instantaneous loss never reaches $0$ and $\eta_t$ stalls at $\beta\sigma^2/2$.