A First Course in Machine Learning
My own study notes, exercise solutions and interactive visualisations for A First Course in Machine Learning by Simon Rogers and Mark Girolami, with Python ports of the MATLAB examples. Work in progress. The book itself is not included here: please get a copy from the publisher.
I used AI heavily to help me understand the book. The derivations and explanations are mostly AI-written in response to the questions I asked while studying, so they are not in my own words, and they may contain mistakes.
Where to start
- Never seen the Bayesian chapters? Read the Chapter 3 summary, then the Chapter 4 summary.
- Prefer to poke at things? The featured visualisations below are the quickest way in.
- Looking for a worked exercise? They are listed at the end of each chapter (Exercises 4.1, 4.2, 4.6, 4.7, 4.8, 5.1 and 5.4 so far).
Featured visualisations
- Newton–Raphson step by stepHow the MAP weights are found when the posterior has no closed form.
- Laplace approximation for the binary response modelA Gaussian fitted at the posterior mode, against the true posterior.
- The Metropolis–Hastings random walkWatch the chain move, accept and reject, and compare its histograms with the Laplace approximation.
- When the random walk strugglesBimodal, skewed, curved and strongly correlated targets, where one step size cannot work.
- Figure 5.2: a Gaussian fitted to each classDrag points and see the class means, covariances and class probabilities change.
- The product of two GaussiansWhy the posterior mean is a precision-weighted average of prior and data.
Chapter 1: Linear modelling, a least-squares approach
Fitting a line
Mathematical background
Chapter 2: Linear modelling from a probabilistic view
The noise model and the likelihood
The shape of the likelihood
- Interactive: the quadratic bowl (log-likelihood surface)
- Interactive: curvature of the likelihood (the Hessian)
Model complexity and uncertainty
Another example: the Bernoulli parameter
Chapter 3: The Bayesian approach
Overview
The coin example: beta prior, binomial likelihood
- Interactive: the beta density
- Interactive: the binomial likelihood
- Notes: posterior with a flat prior
- Notes: mean and variance of a beta random variable
Gaussian prior and Gaussian likelihood
Chapter 4: Bayesian inference
Overview
The running example: the binary response model
Prior, likelihood and the unnormalised posterior
- Interactive: the likelihood read two ways (density of the data, likelihood of the weight)
- Interactive: evaluating the numerator of Bayes’ rule at one w
- Notes: the numerator of the posterior and what is intractable
MAP and Newton–Raphson
Laplace approximation
- Interactive: the gamma density and its Laplace approximation
- Interactive: Laplace approximation for the binary response model
Sampling: Metropolis–Hastings
- Interactive: the random walk, trace plots and histograms against the Laplace approximation
- Interactive: when the random walk struggles (bimodal, skewed, curved, correlated targets)
Exercises
- Notes: Exercise 4.1, the Laplace approximation is exact for a linear Gaussian model
- Notes: Exercise 4.2, the Laplace approximation to the coin posterior
- Notes: Exercise 4.6, Poisson regression and the MAP solution
- Notes: Exercises 4.7 and 4.8, Laplace and Metropolis–Hastings for Poisson regression
- Interactive: Exercise 4.3, the coin posterior and its Laplace approximation
Chapter 5: Classification
Overview
Gaussian class-conditional densities
Exercises
- Notes: Exercise 5.1, the posterior of a class mean with a Gaussian prior
- Notes: Exercise 5.4, maximum likelihood for the multinomial class-conditional