Power Transform Explorer

← back to the notes · Eftekhari & Papyan (ICML 2025), On the Importance of Gaussianizing Representations. Interactive companions to Eq 3 (the transform), Eqs 4–5 and Appendices C–D (choosing λ), and Lemma B.1 (why Gaussian means "uncorrelated ⇒ independent").

$$\psi(h;\lambda)=\begin{cases}\frac{1}{\lambda}\big((1+h)^{\lambda}-1\big), & h\ge 0,\ \lambda\ne 0\\ \log(1+h), & h\ge0,\ \lambda=0\\ \frac{-1}{2-\lambda}\big((1-h)^{2-\lambda}-1\big), & h<0,\ \lambda\ne2\\ -\log(1-h), & h<0,\ \lambda=2\end{cases}$$

1. What Eq 3 does to the number line

Each channel's pre-activations are standardised first, so h is mostly in [−3, 3]. The transform is monotone (it never reorders values), passes through 0 with slope 1 for every λ, and only bends the tails. For λ < 1 the right tail is pulled in and the left tail pushed out, which removes right skew. For λ > 1 it's the mirror image. λ = 1 is the identity.

2. Gaussianize a batch and see how λ is chosen

λ is chosen by maximum likelihood: pick the λ for which ψ(h) looks most like a Gaussian sample. The profile NLL (Eq 13, derived in Appendix C) has two competing parts:

variance term ½ log σ̂²(λ): smaller when ψ squashes the data. On its own it would pick whatever λ shrinks things most;
Jacobian term −(λ−1)·mean(sign(h)·log(1+|h|)): the change-of-variables charge for that squashing. You can't make a sample look tight just by compressing the axis.

The layer doesn't minimise this exactly. It takes one Newton step from λ = 1 on the quadratic Taylor expansion (Eq 5, Appendix D), shown as the dashed parabola.

Things to try. (1) Mild skew: Newton λ̂ lands almost on the exact minimum, which is the regime of the paper's Fig 13. (2) Strong skew (lognormal σ=0.8): the parabola is a poor fit far from λ = 1 and one step undershoots. (3) Student-t or bimodal: λ̂ ≈ 1 and nothing changes. Yeo–Johnson only removes skew. It cannot thin heavy symmetric tails or merge two modes, because a monotone map with slope 1 at 0 and one parameter doesn't have the freedom to.

3. Lemma B.1: the joint shape is what makes "uncorrelated ⇒ independent"

Both panels have exactly N(0,1) marginals and the same correlation ρ. The left one is jointly Gaussian. The right one mixes two Gaussians with correlations ρ + d and ρ − d. Lemma B.1 says the Gaussian joint has the smallest mutual information among all joints with these marginals and this ρ. Set ρ = 0 and increase d to get variables that are uncorrelated but strongly dependent.

Why this matters for the paper. The power transform acts one unit at a time, and mutual information doesn't change under an invertible map applied to each variable separately. So Gaussianizing each unit's marginal cannot, by itself, move a pair from the right panel to the left one. Any independence gain (the paper's Fig 12) has to come from how training responds to the layer, not from the transform.