Annotator Trace Explorer

← back to the notes · Zhang, Tanno et al., Learning from multiple annotators for medical image segmentation, Pattern Recognition 2023. Three companions to §2.3–2.4: every explanation of one pixel's label and where the trace picks among them; the loss trained on a toy you can change; and the trace next to HSIC.

$$\hat p^{(r)}(x)=\hat A^{(r)}(x)\,\hat p(x),\qquad \mathcal L=\sum_{n,r}\operatorname{CE}\big(\hat A^{(r)}\hat p,\ \tilde y^{(r)}_n\big)+\lambda\operatorname{tr}\hat A^{(r)} .$$

1. One pixel: every explanation of the same label

The pixel is lesion. Its annotator writes "lesion" with probability t and "background" with probability 1 − t, so the data at this pixel are the distribution $q=(1-t,\,t)$. A model explains it with a segmentation output $\hat p=(1-s,\,s)$ and a confusion matrix $\hat A=\begin{bmatrix}1-b&a\\ b&1-a\end{bmatrix}$ (column = truth, row = what is written). Cross-entropy only asks for $\hat A\hat p=q$. That leaves one free number once $s$ and $b$ are chosen, so every point of the white band below is a perfect fit. On the band the trace is $\operatorname{tr}\hat A=1+(t-b)/s$, and the level sets are straight lines through the circle at $(0,t)$.

Things to try. (1) No constraint: the minimum is F, the anti-diagonal matrix with the segmentation inverted. The trace does not break the label-swap symmetry in favour of the truth; it breaks it against the truth. (2) Row dominance, the condition Theorem 1 states: the minimum slides onto the yellow line, where every column of Â equals q. The modelled annotator then ignores the truth, and s can be anything. (3) Column dominance, what the proof actually uses: the minimum is T, s = 1, which is the truth. (4) Drag t below ½ with column dominance on. The minimum jumps to s ≈ 0 and the pixel is called background. Per pixel, the trace does exactly what a majority vote would.

2. Train it: one careful and two sloppy annotators

The smallest problem where Eq 4 can be run as written: 48 × 48 pixels, each with a free segmentation logit (the limit of a very flexible network), one confusion matrix per annotator shared by all pixels, labels drawn from each annotator's (FPR, FNR). The loss is the paper's, $\sum_r$ mean cross-entropy $+\ \lambda\sum_r\operatorname{tr}\hat A^{(r)}$, trained with Adam. The warm-up start is what the paper describes: identity CMs, then 300 steps of maximising the trace.

preset λ start

truth

annotator 1

annotator 2

annotator 3

majority vote

model p̂ (lesion)

lesion, correct background marked lesion lesion missed

Dice of the model's segmentation against the truth

Sum of the three traces, Σ tr Â⁽ʳ⁾

Show the numbers (every 250 steps)

Things to try. (1) λ = 0: the model settles on "every annotator is perfect and the pixels are uncertain", and scores exactly the majority vote. (2) λ = 0.55 with the warm-up: Dice climbs to roughly the weighted vote, stays there for a while, then collapses to 0 as every label swaps. The right answer is somewhere training passes through, not where it ends. (3) The same λ without the warm-up swaps at once. (4) "Three equally good": majority vote is already optimal, and the trace only has room to do harm.

3. The trace next to HSIC, for one annotator

Treat the annotator's label ỹ and the true label y as random variables, with lesions a fraction π of pixels. The cross-covariance of their one-hot encodings is $C=\hat A\operatorname{diag}(\pi)-(\hat A\pi)\pi^\top$. HSIC with delta kernels is $\lVert C\rVert_F^2$. With the classes weighted equally, the paper's penalty is $\operatorname{tr}\hat A=1+L\operatorname{tr}C$. It is the same matrix, measured linearly instead of quadratically.

Press Swap: the trace jumps (1.70 → 0.30 at the default settings) and HSIC does not move. HSIC measures how much the label depends on the truth, whichever way round. The trace measures agreement and can go below 1, which is why it prefers anti-agreement unless something (the warm-up) points it the right way. Set b = 1 − a to make every column equal: the annotator then ignores the truth, and HSIC = 0 and tr Â = 1 together.