Hidden variables, by hand

Companion to the notes on Chapter 8 of Amari's Information Geometry and Its Applications. Back to the notes. The running example is a mixture of two unit-variance Gaussians, $p(y;\xi)=w\,N(\mu_1,1)+(1-w)\,N(\mu_2,1)$ with $\xi=(w,\mu_1,\mu_2)$, where the hidden variable $h$ says which component produced an observation. The sample of 200 points in the first widget is the one used by code/hidden_variables.py; the numbers in the readouts match that script's output (Monte Carlo ones up to the stated error).

1. EM, one projection at a time

A point of the data manifold $D$ is a table of responsibilities, one row per observation, each row adding to 1; the model manifold $M$ is the set of complete-data distributions of the mixture. The E-step goes from the point $\xi$ of $M$ to the nearest point of $D$ (the e-projection): the responsibilities are the posterior. The M-step goes from the table to the nearest point of $M$ (the m-projection): weights and means are responsibility-weighted averages, (8.26). Press step and read the readout: the divergence $F$ falls at the M-step by exactly $\mathrm{KL}[p_{\xi_{t+1}}\Vert p_{\xi_t}]$ and at the next E-step by exactly the posterior divergence shown as E part; their sum is the gain in log-likelihood per observation. Try the start with equal means: it is a stationary point (a saddle) and EM, in exactly symmetric arithmetic, stays there for ever; nudge the means by $10^{-6}$ and the gap grows by the sample variance, about $2.67$ per step, until EM leaves for the maximum.

2. The two curves and the staircase (one observation)

With a single observation $y$ and the symmetric model $\tfrac12N(-m,1)+\tfrac12N(m,1)$, the data manifold is an interval (the responsibility $a$ of the component at $+m$) and the model manifold is a line (the number $m$), so the whole algorithm fits in a plane. The grey contours are the divergence $F(a,m)$. The M-step is the straight line $m=(2a-1)y$, because the m-projection is linear in the point of the m-flat $D$; the E-step is the sigmoid $a=\sigma(2my)$. The staircase between them is EM. Move $y$ across $1$: for $\lvert y\rvert>1$ the crossing at $m=0$ is repelling (slope $y^2$), for $\lvert y\rvert<1$ it is the only fixed point and the maximum. Start exactly at $m=0$ and EM never leaves it.

3. How fast: the fraction of missing information

Near a fixed point EM contracts the error by the factor $1-I_o/I_c$: one minus the observed Fisher information over the complete-data information, i.e. the fraction of information the hidden label withholds, (8.30)–(8.31). For $\tfrac12N(-m,1)+\tfrac12N(m,1)$ the complete information is 1 and the factor is $E[y^2\operatorname{sech}^2(my)]$. Slide the true separation $m$: far apart (large $m$) the factor tends to $0$ and EM is exact in a step or two; as the components merge it tends to $1$. The second option feeds the algorithm data from a single Gaussian $N(0,1)$, where the fixed point is $m=0$ and the factor is exactly $1$: the iteration is $m\mapsto m-m^3+\dots$, and it crawls.

4. The identity (8.30) for the three-parameter mixture

For the mixture $w\,N(0,1)+(1-w)\,N(\Delta,1)$ the information about $\xi=(w,\mu_1,\mu_2)$ in the complete data is $G_X=\mathrm{diag}\big(1/w(1-w),\,w,\,1-w\big)$. The information in the visible data alone is $G_Y=E[(\partial\log p_Y)^2]$, computed here by quadrature. What is lost is $G_{X\mid Y}=E_y\mathrm{Cov}_{h\mid y}[\text{complete score}]$, and (8.30) says $G_X=G_Y+G_{X\mid Y}$. The eigenvalues of $G_X^{-1}G_{X\mid Y}$ (bars) are the fraction of the information about each direction that is missing; the largest is the speed of EM, and one eigenvalue is always $0$: the mixture mean $w\mu_1+(1-w)\mu_2$ is read off the data directly.

5. Throwing information away: grouping and censoring

Replace an observation $x\sim N(\mu,\sigma^2)$ by the bin it fell in, or by $\min(x,c)$. The Fisher information of the reduced data is $g^T$, the loss is $g^{X\mid T}=E_t[\mathrm{Cov}(\text{score}\mid T=t)]$, and $g^X=g^T+g^{X\mid T}$ exactly, (8.28)–(8.31). Small bins lose $h^2/12\sigma^2$ of the information about $\mu$ and twice that about $\sigma$; coarse bins lose less than that law says. The two ellipses show the covariance of an efficient estimator from $N=50$ observations: full data (blue) and reduced data (orange), and the orange one is always the larger.

6. A wrong model, and where it sends the data

The truth is the curved family $N(u,u^2)$ of Chapter 7, a parabola in the plane of the observed point $(\bar x,\overline{x^2})$. The model used is wrong; the estimator is its maximum likelihood estimator, which sends the observed point to the model along the straight q-ancillary leaves. Where the truth sits on a leaf labelled $v^*$, the estimator converges to $v^*$ (Theorem 8.3: consistent iff $v^*=u$). The loss of information is the angle between that leaf and the truth (8.42): the efficiency is $\sin^2$ of it, and it equals the squared correlation between the true score and the model's score. Switch models: $N(v,1)$ is the sample mean ($\tfrac13$), $N(0,v^2)$ is $\sqrt{m_2/2}$ ($\tfrac89$ after relabelling), and $N(v,c\,v^2)$ with the wrong coefficient of variation $c$ is inconsistent but, once relabelled by $f(v)=v/\kappa$, almost efficient. The button simulates $N=4000$ observations, $10^5$ times, from the exact law of $(\bar x,\overline{x^2})$.

7. The decoder that ignores correlation

Neurons respond with mean $r(u)$ and covariance $V$ (the truth); the model ignores $V$ and uses the identity. Its maximum likelihood estimator is least squares, consistent for every $V$ (Theorem 8.3), and its efficiency is $\lvert r'\rvert^4/\big(r'^{\mathsf T}Vr'\cdot r'^{\mathsf T}V^{-1}r'\big)$: it equals $1$ exactly when $r'$ is an eigenvector of $V$ (Cauchy–Schwarz). So “no loss” is not a property of the model but of the pair $(V,r')$. With two neurons turn the tuning direction against the correlation axis. With a ring of $n$ neurons, bump tuning of width $a$ and Gaussian correlation of range $b$: short-range correlation costs almost nothing, correlation as wide as the tuning curve costs most of the information, uniform correlation costs nothing again, and cosine tuning is an eigenvector of every translation-invariant $V$.