Features Are Fate, Interactive

← text notes · Tahir, Ganguli & Rotskoff, ICML 2025. Seven widgets: the linear-transfer phase diagram (eq. 11), why transferability falls as the target set grows, fine-tuning and the null-space guess (eq. 15), the ridge result (eq. 13), and three from Appendix A: KL against transferability, the Dudley metric saturating, and a finite-sample Wasserstein distance that measures noise. Everything is computed in your browser from the closed forms in the notes; code/ holds the Python that checks them.

1. Eq. (11): the linear-transfer phase diagram

A source model has learned the single direction β_s. The target uses β_t at angle θ from it, and has n = γd noisy examples. Linear transfer can only re-scale the source direction, so it has a floor sin²θ of error it can never remove. Scratch training pays the double-descent risk R_sc(γ, σ) of eq. (8). The transferability is the difference, T = R_sc − sin²θ. Blue is T > 0 (transfer wins), red is T < 0. Click the left panel to move the point; the right panel is the slice of the surface at your θ.

Things to try. Set σ = 0: the band of anomalous positive transfer around γ = 1 disappears, and for γ > 1 the picture is one solid red, because scratch training is exact once n > d and T = −sin²θ. Raise σ to 1: negative transfer for γ < 1 vanishes altogether (needs 1 − cos θ > σ). Set θ = 90° (orthogonal tasks) and watch the blue sliver at γ = 1: transfer wins there only because scratch training is at its worst. Finally fix θ = 45° and σ = 0.2 and drag γ from 0 to 2: the same pair of tasks flips from positive to negative to positive to negative.

2. Why T falls with more target data: a moving curve against a floor

This is the same formula seen as two risks. The linear-transfer risk is sin²θ + (σ² + sin²θ)/(n−2) (Theorem 3.7): it is nearly flat in n, because one direction can be estimated from a handful of points, and it cannot go below sin²θ. The scratch risk is the double-descent curve, which keeps improving. T is the gap. Tick exact to draw the finite-d versions the proofs really give (thin lines): the scratch peak is finite at finite d, and the linear-transfer risk sits slightly above its floor at small n.

3. Eq. (15): fine-tuning and the null-space guess

Fine-tuning changes all the weights, but gradient flow can only move the model inside the space the data span (the row space of X, a random n-dimensional slice of R^d). In the remaining (1 − γ) of directions the data are silent, and the network keeps whatever it was initialised with. Scratch guesses 0 there; fine-tuning guesses β_s. Fine-tuning wins if β_s is closer to β_t than the origin is: inside the dashed circle, which is exactly θ < 60°. Drag β_s around the unit circle.

The threshold does not depend on γ: sample size only sets how much is at stake, (1 − γ). Drag θ across 60° and the sign flips at every γ. Tick the correction (derived in the notes and checked against real gradient descent): the threshold becomes cos θ = c/2 with c = 1 + O(σ²). At σ = 0.2 it barely moves (59.8° at γ = 0.25). Set σ = 1 and γ = 0.5: c = 1.22 and the break-even angle drops to 52°.

4. Eq. (13): ridge regularisation, and when it finally helps

Theorem 3.8 says ridge on the last layer only raises the risk of linear transfer. The proof is a law of large numbers: for large n the scalar being fitted is b = z·y/(‖z‖² + nλ) → cos θ/(1 + λ), which is biased and has no variance left to trade against. At finite n there is a variance term (σ² + sin²θ)/(n − 2) and a dip is possible. The dots are a Monte Carlo of the finite-n risk; the curve is the n → ∞ limit. The defaults (σ = 1.5, n = 10, a very noisy small set) show the dip; at σ = 0.2 there is essentially none.

5. Appendix A (1): KL against transferability, θ from 0 to π

Within the paper's own family (‖β_s‖ = ‖β_t‖ = 1) the two data distributions differ by a shift in the mean of y, and KL(p_s‖p_t) = ‖β_s − β_t‖²/(2σ²) = (1 − cos θ)/σ². It rises with θ all the way to π. But T_lt depends on θ only through sin²θ, which comes back down after π/2. Sweeping θ over [0, π] gives Figure 4(a)'s U. Restrict to [0, π/2], where the phase diagrams live, and KL orders T perfectly.

6. Appendix A (2): “arbitrarily far” in the Dudley metric, and the proof line that fails

Theorem A.2 takes g = c·f and lets c grow. KL then grows without bound. The Dudley metric cannot: every judge in its unit ball has |h| ≤ 1, so the distance is at most 2. The left panel gives a lower bound from an explicit judge h(y) = a(1 − 2 min(|y|/K, 1)); the right panel shows KL and the transferability of linear transfer (with the target g = c f, ‖g‖ = c) both growing like c². The table below tests line (29) of the proof.

7. Appendix A (3): Figure 4(b), a Wasserstein estimate that measures noise

Figure 4(b) plots T against a Wasserstein-1 distance estimated from finite samples, and the values (526 to 529) barely vary. Here both laws are samples of (x, y) with x ∈ ℝ^d Gaussian and y = β·x + noise. The x-marginals are identical; only the mean of y differs, by at most a Gaussian of size ‖β_s − β_t‖ ≤ 2. The population W₁ is therefore at most √(2/π)·‖β_s − β_t‖ ≤ 1.6. But two independent samples of a Gaussian in ℝ^d are far apart in ℓ₁ or ℓ₂, and the estimator sees that separation. The page solves the assignment problem exactly (Hungarian algorithm) between two samples of n points.

At d = 500 with the ℓ₁ metric and n around 250 the estimate is ≈ 527 for every θ, including θ = 0 where the two laws are the same. Turn d down to 2: now the estimate is informative and rises with θ, like KL. So the estimator in Figure 4(b) is dominated by dimension, not by the difference between tasks. (This page's n is capped at 200 for speed; the Python check uses n = 250 and gets the same 526 to 528.)