Information Geometry Notes
Study notes on Shun-ichi Amari, Information Geometry and Its Applications (Springer Japan, 2016): the fundamentals worked out from scratch, with checks you can run and visualisations you can poke at. Heavy use of AI went into these notes: they reflect my effort to understand this book using Claude as a tutor. The book itself is not included here.
Part I: Geometry of Divergence Functions: Dually Flat Riemannian Structure
- Chapter 1. Manifold, Divergence and Dually Flat Structureread
sections
- 1.1 Manifolds
- 1.2 Divergence Between Two Points
- 1.3 Convex Function and Bregman Divergence
- 1.4 Legendre Transformation
- 1.5 Dually Flat Riemannian Structure Derived from Convex Function
- 1.6 Generalized Pythagorean Theorem and Projection Theorem
- Chapter 2. Exponential Families and Mixture Families of Probability Distributionsreading
sections
- 2.1 Exponential Family of Probability Distributions
- 2.2 Examples of Exponential Family: Gaussian and Discrete Distributions
- 2.3 Mixture Family of Probability Distributions
- 2.4 Flat Structure: e-flat and m-flat
- 2.5 On Infinite-Dimensional Manifold of Probability Distributions
- 2.6 Kernel Exponential Family
- 2.7 Bregman Divergence and Exponential Family
- 2.8 Applications of Pythagorean Theorem
- Chapter 3. Invariant Geometry of Manifold of Probability Distributionsread
sections
- 3.1 Invariance Criterion
- 3.2 Information Monotonicity Under Coarse Graining
- 3.3 Examples of f -Divergence in Sn
- 3.4 General Properties of f -Divergence and KL-Divergence
- 3.5 Fisher Information: The Unique Invariant Metric
- 3.6 f -Divergence in Manifold of Positive Measures
- Chapter 4. α-Geometry, Tsallis q-Entropy and Positive-Definite Matricesread
sections
- 4.1 Invariant and Flat Divergence
- 4.2 α-Geometry in Sn and Rnþ
- 4.3 Geometry of Tsallis q-Entropy
- 4.4 ðu; vÞ-Divergence: Dually Flat Divergence in Manifold of Positive Measures
- 4.5 Invariant Flat Divergence in Manifold of Positive-Definite Matrices
- 4.6 Miscellaneous Divergences
Part II: Introduction to Dual Differential Geometry
- Chapter 5. Elements of Differential Geometryread
sections
- 5.1 Manifold and Tangent Space
- 5.2 Riemannian Metric
- 5.3 Affine Connection
- 5.4 Tensors
- 5.5 Covariant Derivative
- 5.6 Geodesic
- 5.7 Parallel Transport of Vector
- 5.8 Riemann–Christoffel Curvature
- 5.9 Levi–Civita (Riemannian) Connection
- 5.10 Submanifold and Embedding Curvature
- Chapter 6. Dual Affine Connections and Dually Flat Manifoldread
sections
- 6.1 Dual Connections
- 6.2 Metric and Cubic Tensor Derived from Divergence
- 6.3 Invariant Metric and Cubic Tensor
- 6.4 α-Geometry
- 6.5 Dually Flat Manifold
- 6.6 Canonical Divergence in Dually Flat Manifold
- 6.7 Canonical Divergence in General Manifold of Dual Connections
- 6.8 Dual Foliations of Flat Manifold and Mixed Coordinates
- 6.9 System Complexity and Integrated Information
- 6.10 Input–Output Analysis in Economics
Part III: Information Geometry of Statistical Inference
- Chapter 7. Asymptotic Theory of Statistical Inferenceread
sections
- 7.1 Estimation
- 7.2 Estimation in Exponential Family
- 7.3 Estimation in Curved Exponential Family
- 7.4 First-Order Asymptotic Theory of Estimation
- 7.5 Higher-Order Asymptotic Theory of Estimation
- 7.6 Asymptotic Theory of Hypothesis Testing
- Chapter 8. Estimation in the Presence of Hidden Variablesread
sections
- 8.1 EM Algorithm
- 8.2 Loss of Information by Data Reduction
- 8.3 Estimation Based on Misspecified Statistical Model
- Chapter 9. Neyman–Scott Problem: Estimating Function and Semiparametric Statistical Modelread
sections
- 9.1 Statistical Model Including Nuisance Parameters
- 9.2 Neyman–Scott Problem and Semiparametrics
- 9.3 Estimating Function
- 9.4 Information Geometry of Estimating Function
- 9.5 Solutions to Neyman–Scott Problems
- Chapter 10. Linear Systems and Time Seriesread
sections
- 10.1 Stationary Time Series and Linear System
- 10.2 Typical Finite-Dimensional Manifolds of Time Series
- 10.3 Dual Geometry of System Manifold
- 10.4 Geometry of AR, MA and ARMA Models
Part IV: Applications of Information Geometry
- Chapter 11. Machine Learningread
sections
- 11.1 Clustering Patterns
- 11.2 Geometry of Support Vector Machine
- 11.3 Stochastic Reasoning: Belief Propagation and CCCP Algorithms
- 11.4 Information Geometry of Boosting
- 11.5 Bayesian Inference and Deep Learning
- Chapter 12. Natural Gradient Learning and Its Dynamics in Singular Regionsread
sections
- 12.1 Natural Gradient Stochastic Descent Learning
- 12.2 Singularity in Learning: Multilayer Perceptron
- Chapter 13. Signal Processing and Optimizationread
sections
- 13.1 Principal Component Analysis
- 13.2 Independent Component Analysis
- 13.3 Non-negative Matrix Factorization
- 13.4 Sparse Signal Processing
- 13.5 Optimization in Convex Programming
- 13.6 Dual Geometry Derived from Game Theory
Foundations
- Bregman divergence: the gap to the tangent
- Conditional expectation is a projection
- Convex function and the Legendre transform
- Distance, divergence, discrepancy and metric
- Fisher information and the natural gradient
- Jacobian and Hessian: what moves where, and how it bends
- Kernel functions: similarity that behaves like a dot product
- Positive definite matrices: the bowl and the ruler
- Potential, function and vector field
- What a measure is