Conditional entropy as a transferability score

Companion to the notes on Tran, Nguyen & Hassner, Transferability and Hardness of Supervised Classification Tasks (ICCV 2019). Back to the notes

The paper scores how well a network trained on one labelling of some images (the source, labels $z$) will carry over to another labelling of the same images (the target, labels $y$) by counting: how surprising is $y$ once $z$ is known? Four small experiments below take that apart: the counting itself, the one inequality in the proof, what the paper's own 40 attributes say about it, and a bias the counting has when each source class has few images.

1. Count, then take the entropy

Every image has a target label (a row below) and a source label (a column). Type counts into the grid. For the images with a given source label, look at how their target labels are spread. If nearly all agree, knowing $z$ nearly tells you $y$ and the conditional entropy $H(Y\mid Z)$ is close to $0$. If they are spread evenly it is as large as it can be. The unit is nats (natural logarithms): a fair coin is $\ln 2 = 0.693$ nats. The paper's bound then reads $\widetilde{\mathrm{Trf}}\ge \ell_Z - H(Y\mid Z)$, so a smaller $H(Y\mid Z)$ raises the guaranteed floor. Try the presets, then edit the counts.

Presets:
Counts of images: rows are target labels $y$, columns are source labels $z$.
Each column is $\hat P(y\mid z)$. Shading is the probability; the small number is the cell's share of $H(Y\mid Z)$, in nats.

2. The one inequality: drop all but one term

The proof builds a classifier $\bar k$ whose probability for the true target label $y_i$ is a sum over source labels, $\sum_z \hat P(y_i\mid z)\,p(z\mid x_i)$, then keeps only the term with $z=z_i$. That step is where the bound gives something away. With two source labels, the kept term is $a\,c$ (blue) and the dropped one is $b\,(1-c)$ (orange), where $c$ is the source model's confidence in the true $z_i$, $a=\hat P(y_i\mid z_i)$ and $b=\hat P(y_i\mid z_{\text{other}})$. The log of the full bar minus the log of the blue bar is the gap.

Presets:
The paper's proof does not quantify this. On six points with $Z=(0,0,0,1,1,1)$, $Y=(0,0,1,1,2,2)$ only the two points with $y=1$ are ambiguous, so the average gap is exactly $\tfrac13\ln(1/c)$.

3. The paper's own 40 attributes

Tables 3 and 4 of the paper's appendix give, for each of 40 CelebA face attributes, the conditional entropy given identity ($H(Y\mid\text{ID})$, the paper's measure), the attribute's own label entropy ($H(Y)$, which the paper calls its hardness), and the test accuracy of a linear classifier on face-recognition features. Pick what goes on each axis. The first view is the paper's claim (Figure 3): CE against error. Then swap the x-axis to $H(Y)$, which uses no source at all.

4. Counting with few examples per source class

The plug-in estimate of $H(Y\mid Z)$ splits the data by $z$ and measures how mixed $y$ is within each small group. A small group looks purer than the truth just by chance: three coin flips are rarely a perfect 50/50. So with few images per source label the count reads low even when $y$ has nothing to do with $z$. This widget draws $y$ independently of $z$ (true $H(Y\mid Z)=H(Y)$) and shows what counting would report. Set the numbers to the paper's identity source with the first button.