Time series geometry, by hand

Companion to the notes on Chapter 10 of Amari's Information Geometry and Its Applications. Back to the notes. A stationary Gaussian series is summarised by its power spectrum $S(\omega)=|H(e^{i\omega})|^2$, one variance per frequency; the sign convention is the book's, $H(z)=(1+bz^{-1})/(1+az^{-1})$, i.e. $x_t=-a\,x_{t-1}+\varepsilon_t+b\,\varepsilon_{t-1}$, so $a=-0.6$ means $x_t=0.6\,x_{t-1}+\varepsilon_t$. Integrals over frequency use a grid of 1024 points, exact for the trigonometric polynomials involved; Toeplitz computations (finite records) are exact linear algebra. Everything is the same mathematics as code/time_series_geometry.py, and the default readouts equal its printed numbers.

1. Two coordinate systems for one spectrum

The cosine coefficients of $S$ are the autocovariances $r^*_t=\mathbb E[x_sx_{s-t}]$, the m-affine coordinates; those of $1/S$ are the inverse autocovariances $r_t$, the e-affine coordinates (10.39)–(10.42). An AR($p$) model is exactly the one with $r_t=0$ for $t>p$, a set of linear conditions in the e-coordinates, hence e-flat; an MA($q$) model has $r^*_t=0$ for $t>q$, hence m-flat. Pick a family and move its parameters: the bars show $r^*_t$ (orange) and $r_t$ (blue) for $t=0,\dots,7$, and notice which sequence stops and which never does. ARMA(1,1) has neither stopping.

2. The e-geodesic and the m-geodesic between two models

Between two spectra there are two natural straight lines: the m-geodesic $S_t=(1-t)S_1+tS_2$ (mix the spectra) and the e-geodesic $1/S_t=(1-t)/S_1+t/S_2$ (mix the inverse spectra). Between two AR(2) models the e-geodesic never leaves AR(2): $r_3,r_4,\dots$ stay $0$. The m-geodesic does leave: its $r_3,r_4,\dots$ become non-zero. For MA(2) the roles swap, and for ARMA(1,1) (where $\gamma_{k+1}/\gamma_k=-a$ for $k\ge2$) neither geodesic keeps the ratio constant. Drag $t$ and read the table.

the two end models S₁, S₂e-geodesic (bars: left of each pair)m-geodesic (bars: right of each pair)

3. What a record of $n$ observations knows: the normalisations

The book writes the Kullback–Leibler divergence of two series as $\frac1{2\pi}\int(S_1/S_2-1-\log S_1/S_2)\,d\omega$ (10.37). Take two actual Gaussian records of $n$ observations, with their exact Toeplitz covariance matrices, and compute $\mathrm{KL}/n$: it tends to half of that, $\frac1{4\pi}\int(\dots)$. Likewise the exact Fisher information of $n$ observations of ARMA(1,1), divided by $n$, tends to $\frac1{4\pi}\int\partial_i\log S\,\partial_j\log S\,d\omega$, whose closed form is shown; the book's metric (10.36) is $2\pi$ times that. The difference between the exact value and the limit is a constant (a boundary term), not growing with $n$.

KL columns: exact Gaussian KL of $n$ observations of $S_1$ against $S_2$. Fisher block: $S_1$ in the coordinates $(a,b)$ at unit gain; the exact value uses central differences of the Toeplitz covariance.

4. Projection onto AR($p$): Yule–Walker, Pythagoras, prediction error

The m-projection of a spectrum $S$ onto AR($p$) is the Yule–Walker model: it has the same $r^*_0,\dots,r^*_p$ as $S$. Its divergence is $D[S:S_p]=\tfrac12\log(\sigma_p^2/\sigma_\infty^2)$, the log-ratio of the order-$p$ and infinite-order prediction error variances, and the Pythagorean relation (10.57) says $D[S:S_0]=D[S:S_p]+D[S_p:S_0]$ for white noise $S_0$. At $p=1$ there is a second projection, the e-projection (minimising $D[S':S]$), which is a different AR(1) model; it is shown dashed. The divergences here are Kullback–Leibler rates per observation, $\frac1{4\pi}\int$, half the printed (10.37).

Sm-projection S_p (Yule–Walker); bars: D[S_p : S₀]e-projection (p = 1 only); bars: D[S : S_p] on top

5. Maximum entropy, and the hypothesis the book leaves out

Theorem 10.2: among all spectra with the same $r^*_0,\dots,r^*_p$ as $S$, the Yule–Walker AR($p$) model has the largest entropy rate. Here $S_p$ is the AR($p$) projection of the MA(1) with $b=0.5$; add $\delta(\omega)=c_a\cos((p+1)\omega)+c_b\cos((p+2)\omega)$, which has no components of order $\le p$ and therefore keeps $r^*_0..r^*_p$ fixed, and watch the entropy fall (quadratically, with the second-order prediction $-\tfrac14\,\mathrm{mean}(\delta^2/S_p^2)$). Now switch on white noise $c$: that moves $r^*_0$ and the entropy grows without bound, like $\tfrac12\log c$, so the theorem as printed (which lists only $r^*_1,\dots,r^*_p$) needs the zeroth coefficient fixed too.

6. ARMA(1,1): the line where the model disappears

For $H(z)=(1+bz^{-1})/(1+az^{-1})$ numerator and denominator cancel when $a=b$: every such model is white noise. The per-observation Fisher information is $G=\left[\begin{smallmatrix}1/(1-a^2)&-1/(1-ab)\\-1/(1-ab)&1/(1-b^2)\end{smallmatrix}\right]$ with $\det G=(a-b)^2/((1-a^2)(1-b^2)(1-ab)^2)$, which vanishes quadratically on the diagonal. In the cepstral coordinates $(c_1,c_2)=(2(b-a),\,a^2-b^2)$, Euclidean coordinates for the book's metric, the whole diagonal goes to one point, and the two triangles $b>a$ and $b<a$ land on the two half-planes: the figure the book draws as two blobs touching at a point.

7. The edge of the stable region

For AR(1), $x_t=\phi x_{t-1}+\varepsilon_t$, the Fisher information per observation $1/(1-\phi^2)$ blows up at the unit root, but the Fisher arc length from $0$ is only $\arcsin\phi\to\pi/2$: the boundary is at finite distance. The Kullback–Leibler divergence from white noise to the AR(1) model stays below $\tfrac12$, while the divergence from the AR(1) model to white noise is infinite in the limit. For MA(1) with $b\to1$ the spectrum acquires a zero, so $1/S$ stops being integrable: the inverse autocovariances $r_t$ blow up while the autocovariances stay finite and $\int(\log S)^2$ stays finite, so (10.13) holds but the e-chart does not exist at the edge.

Fisher information 1/(1−φ²) (middle) and arc length (left)KL[AR(1) : white] (middle)KL[white : AR(1)] (right)

8. The periodogram is not the spectrum

The book calls $S=|X(\omega)|^2$ a deterministic function (10.12). Simulate an AR(1) record of length $n$ and compute $I=|\sum_t x_te^{-i\omega t}|^2/n$ at a fixed Fourier frequency $\omega=\pi/4$ over many records: its mean is $S(\omega)$, but its standard deviation relative to $S$ stays at $1$ however long the record, because $I/S$ is asymptotically a unit exponential variable. Averaging $9$ neighbouring ordinates reduces the standard deviation to about $1/3$. Neighbouring ordinates are uncorrelated and the phase of $X$ is uniform, as (10.25) says.