OLS as an attention mechanism

Goulet Coulombe (2026). Every OLS prediction is a weighted average of the training outcomes, ŷⱼ = Σᵢ ωⱼᵢ yᵢ with ωⱼᵢ = xⱼᵀ(XᵀX)⁻¹xᵢ. The two panels let you see those weights, the "attention pattern" of a regression, and what shrinkage and dimension reduction do to it. Training inputs are 300 points uniform on [−1, 1] and the features are powers 1, x, x², …

1. Who does the test point attend to?

OLS weight ωⱼᵢ (signed)Gaussian-kernel weight (softmax-like, ≥ 0)
Move the test point past ±1: the weights grow and swing in sign across the whole interval. That is extrapolation, and Σᵢ ωⱼᵢ² = xⱼᵀ(XᵀX)⁻¹xⱼ is exactly the factor multiplying the noise variance of the prediction. Raise the degree and the lobes multiply, because a more flexible regression is a more wiggly smoother. The weights always sum to 1 because the intercept is a regressor, but negative entries mean they are not a convex combination.

2. Ridge and PCR change the embedding, not the module

OLS, D = 1/λᵢridge, D = 1/(λᵢ + λ)PCR, D = 1/λᵢ for i ≤ L, 0 otherwise
All three curves come from Ω = U D Uᵀ with the same eigenvectors U of XᵀX (degree-5 polynomial, 6 features). Only the diagonal D changes (paper Eq. 23). Ridge damps the small-eigenvalue directions, the wiggly ones. PCR deletes them. Both pull the weights toward a smoother, less signed pattern.