Imbalanced Overfitting, Interactive

← text notes · Lyu, Zhou & Zhong, A Statistical Theory of Overfitting for Imbalanced Classification, ICLR 2026. Six widgets. Every curve comes from the paper's limiting equations (Lemma E.9, Eqs 28, 38, 168), solved in your browser. The first widget can also train a real hard-margin SVM to compare against.

1. Training logits vs test logits: the truncation

The data are the paper's two-Gaussian mixture: y = +1 (minority) with probability π, and x | y ~ N(yµ, I_d), with n/d → δ. Train the max-margin classifier f(x) = ⟨x, β⟩ + β₀ with ‖β‖ = 1. Theorem 2.1 says that on a test point the signed logit y·f(x) is N(ρ*‖µ‖ + yβ₀*, 1). On the training points it is the same Gaussian, except that everything below the margin κ* is moved to a point mass at κ*. The shaded area below 0 is the test error, and training erases it. The numbers come from solving Eqs 76a–c.

π (minority) ‖µ‖ δ = n/d τ

simulate with n =

test logit density (TLD)training logit density (ELD) above κ*spike: point mass at κ*test mass pushed to κ*test errorssimulated training margins

The shared overfitting budget, E[(1−ρ²)ξ²] = (1−ρ*²)/δ, split into the minority's share π·g₂(t₊) and the majority's share (1−π)·g₂(t₋):

Things to try. At the default (Figure 1 of the paper), 61% of minority training points sit exactly on the margin, against 18% of majority points. The minority's per-point push g₁(t₊) is always exactly (1−π)/π times the majority's. The reason is Eq 79: each class's total push is equal, and the minority has fewer points to share it. Drag δ up towards δ*(0) and κ* shrinks to 0. Drag it down and the truncation gets more severe. τ leaves ρ* alone and slides β₀* (Corollary E.10). Press τ = τopt and the two test errors become equal. Now set π = 0.05 and δ = 0.5. τopt turns negative: no positive τ reaches β₀* = 0. Press train an SVM to overlay a real hard-margin SVM (dual SMO, d = n/δ) on the limit.

2. When is the training set separable? δ*(0)

Theorem D.1(a): with probability tending to 1 the data are linearly separable if δ < δ*(0), and not separable if δ > δ*(0). Here δ*(0) = max over ρ, β₀ of (1−ρ²) / E[(−ρ‖µ‖ − G − β₀Y)₊²]. The page finds it as the δ at which κ* crosses 0. With ‖µ‖ = 0 and π = ½ this is Cover's classical n/d = 2. The intercept, the signal and the imbalance all raise it.

3. Margin rebalancing: errors against τ and π

The rebalanced SVM (Eq 7) asks for a margin τ times larger on the minority side. Proposition C.1 says this changes only the intercept: β₀(τ) = β₀(1) + (τ−1)/(τ+1)·κ(1). Left panel: limiting errors against τ at widget 1's (π, ‖µ‖, δ), as in the paper's Figure 8. Right panel: errors against π at τ = 1 and at τ = τopt, as in Figure 4. Where τopt < 0 (grey), the τopt curve assumes a boundary no positive τ can reach.

4. High imbalance: three phases

Theorem 3.2 lets things grow with d: π ∝ d^−a, ‖µ‖² ∝ d^b, n ∝ d^c+1, and τ = d^r. The whole story is a competition between two lengths. One is the signal ‖µ‖ ≍ d^b/2. The other is the noise that the minority's few training points can be fitted to: the norm of their mean noise, √(d/n₊) ≍ d^(a−c)/2. Lemma H.1 bounds the margin by half the distance between the two training class means, ≈ √(d^b + d^a−c/4).

Left: the (a − c, b) plane. Above the line a − c = b the signal wins outright and any 1 ≤ τ ≪ d^b/2 works. Below a − c = 2b the signal is lost and no τ helps. In between, the moderate wedge, the plain SVM (r = 0) gets the minority wrong almost always. Only τ in the window d^a−b−c ≪ τ ≪ d^(a−c)/2 fixes it. Right: that window on the r axis. The numbers in the readout are order-of-magnitude estimates from Lemmas H.1–H.5 at the chosen d. They are not exact limits.

5. Non-separable data: the proximal shrinkage

When δ > δ*(0) the SVM does not exist, so the paper studies logistic regression instead (Theorem D.3). There a typical training logit is prox_λ*ℓ(test logit), where prox_λℓ(x) = argmin_t ℓ(t) + (t − x)²/(2λ), and ℓ(t) = log(1 + e^−t). A small λ* (large δ, little overfitting) gives nearly the identity. A large λ* (δ just above δ*(0)) gives a smooth version of the truncation max{κ, x}, up to scale. The paper does not give λ* in closed form (it solves a 4-equation system), so here λ is a free slider.

6. Calibration: the limiting reliability curve

With ‖β‖ = 1 the confidence is p̂ = σ(f). By Bayes' rule on the two Gaussian logit classes (Lemma I.1), the truth given that logit is p̂₀ = σ(2ρ*‖µ‖·(f − β₀*) + log π/(1−π)). So the limiting reliability diagram is one explicit curve, the paper's Figure 6 without the binning. There are two separate mismatches. The slope 2ρ*‖µ‖ need not equal 1, which is a scale that ‖β‖ = 1 fixes arbitrarily. And at τopt (β₀* = 0) the classifier ignores the prior log-odds. The toggle adds the prior back, to show how much of CalErr* each one accounts for.

Numerics: Φ via a Chebyshev erfc (relative error < 1.2·10⁻⁷). g₁(t) = tΦ(t) + φ(t), g₂(t) = (t² + 1)Φ(t) + tφ(t), g₁⁻¹ and ρ* by bisection. The SVM is the C = ∞ dual solved by maximal-violating-pair SMO, as in the notes' svm_simulation.py, with µ = (‖µ‖, 0, …, 0) by rotation invariance.