The Performance of Transferability Metrics does not Translate to Medical Tasks

Links

• arXiv:2308.07444 — preprint. Code at VirtualSpaceman/transfer-estimation-medical.
• The scores it evaluates: Tran 2019 — NCE, Nguyen 2020 — LEEP, You 2021 — LogME, Bao 2019 — H-score.
• Claßen 2026 — TE robustness — the direct follow-up, which takes up this paper’s own call for robustness work and separates the evaluation-metric question this one leaves open.
• Shao 2022 — SFDA — the measure that imitates fine-tuning dynamics rather than scoring a static representation.
• Achille 2019 — task reachability — one account of what frozen-feature scores leave out. Fine-tuning success also depends on whether SGD can reach a good target solution from the source weights (that paper’s reachability factor), which none of the scores measures. The linked section places each score against the paper’s two factors.
• Runnable code — not a reproduction, which is impossible here, but the statistical resolution of their Table 2: what a rank correlation over ten architectures can and cannot detect. make verify runs it.

In one paragraph

Seven transferability scores, ten ImageNet-pretrained architectures, three medical classification tasks — skin lesion, breast histopathology, brain tumour — and 2250 fine-tuned models with proper hyperparameter search behind every reference number. The conclusion is negative: no score reliably or consistently predicts target performance in a medical context, and the authors recommend practitioners not rely on them yet. The distinctive contribution is the out-of-distribution axis: each task gets a second, genuinely shifted test set, and the scores are evaluated against OOD performance too — the first study to do that. The surprise is that the OOD column is where the scores work best, and specifically where the label-based ones do. The reservation is about resolution: ten architectures is a small pool for a rank correlation, and that turns out to decide which of their forty-two numbers carry information.

What was run

Two things are done better than the norm here and deserve saying. The hyperparameter search is real — 75 configurations per architecture, so the reference ranking reflects each model at its best rather than at default settings, which is the usual shortcut. And balanced accuracy is used rather than plain accuracy, which is the right choice for imbalanced medical data. Both matter because the reference ranking is what everything is measured against.

The taxonomy that turns out to matter

They split the scores in two:

• feature-based (fb): need only the source feature extractor — H-Score, \mathcal{N}-LEEP, LogME, Regularised H-Score, GBC.
• label-based (lb): need the source classification head — NCE, LEEP.

This is the same split that runs through the Theory section, and it is exactly the axis along which their results separate.

The findings

In-distribution: unstable, and inconsistent across datasets. LogME is among the best on BrainTumor-Cheng (\tau=0.584) and BreakHis (0.378), and negative on ISIC2019 (-0.067). H-Score goes 0.270 \to 0.600 \to -0.244 across the three. No score keeps its sign, let alone its rank. Their hypothesis is domain dissimilarity: unlike the general-purpose benchmarks these scores were validated on, medical targets share neither classes nor low-level statistics with ImageNet.

Out-of-distribution: the label-based scores do well. NCE reaches 0.911 on PAD-UFES-20 and 0.778 on ICIAR2018; LEEP reaches 0.778 on ICIAR2018. This inverts the in-distribution picture, where NCE is often negative.

Their explanation is worth quoting because it is a real mechanism and not hand-waving: for binary targets with few classes, the source model’s predicted distribution concentrates on a single class, which inflates the label-based score. So the OOD success may be an artefact of task shape rather than genuine OOD sensitivity. They flag this themselves.

Two candidate causes, tested. They fine-tuned on each medical dataset and re-scored against the validation set: only label-based methods did well, so domain shift “helps to degrade the efficiency of such scores, but it is not the main reason”. They then collapsed OxfordPets from 37 classes to binary cats-vs-dogs: correlations dropped but stayed high. So neither shift nor class count alone explains the failure — a genuinely useful negative result that narrows the space.

The instrument’s resolution

The experiment is 2250 models and cannot be reproduced here. What can be checked is how much signal a rank correlation over ten items can carry, and it decides how to read Table 2.

Under the null, Kendall’s \tau at n=10 has standard deviation \sqrt{2(2n+5)/(9n(n-1))} = 0.248 — confirmed by simulation to three decimals. So:

At n=10, a reported \tau of 0.270 is barely one standard error from zero. Applying the threshold to all 42 cells of their Table 2:

• 6 of 42 clear p<0.05, against ~2.1 expected by chance with no correction.
• 3 clear Bonferroni (\lvert\tau\rvert \ge 0.733), and they are:

Every survivor is an out-of-distribution cell, and every one is a label-based scorer. That is exactly the paper’s positive finding, and it is the part that is statistically solid.

The headline negative finding is the underpowered part: the in-distribution cells are not distinguishable from zero in either direction. Power to detect a true \tau=0.3 at n=10 is 14%:

This is a criticism of the evidence, not of the conclusion. Underpowered null results err in the safe direction — failing to detect a correlation is not evidence that none exists — so “don’t rely on these scores in medical imaging yet” remains sound advice. But it is supported by far less than forty-two numbers suggest, and the paper reports point estimates with no interval anywhere.

Questions and doubts

• Ten architectures is the binding constraint. Every conclusion is a rank correlation over ten items, where the null standard deviation is 0.248. Confidence intervals would have made the strength of each claim visible; none are reported.
• The OOD result may be a task-shape artefact, as they say themselves: binary targets concentrate the source model’s predictions and inflate label-based scores. Since all three OOD sets are binary or near-binary, the cleanest interpretation of their strongest result is also the least flattering to it.
• Single seed, single subset. Claßen et al. later showed that rankings churn under reseeding alone, which means each of these 42 numbers carries an unreported sampling variance on top of the null variance above.
• Balanced accuracy is the right choice but creates a mismatch that goes unexamined: most of these scores were designed and validated against plain accuracy. Claßen et al. make this the explicit subject of their second experiment and find the reference ranking does depend on the criterion — so part of what is measured here as “the score fails” may be “the score optimises something else”.
• Table 2’s caption says \tau_w while the text says Kendall’s \tau. A small inconsistency, but the two weight disagreements differently and the distinction matters when the whole result is a table of correlations.
• The in-distribution instability is reported but not diagnosed. LogME swinging from +0.584 to -0.067 across datasets is the most interesting number in the paper, and the two causes they test (shift, class count) are both ruled out without a third being proposed.

Takeaways

• The OOD axis is the contribution. Being first to ask whether transferability scores predict out-of-distribution performance is the durable idea here, and it is the axis on which the results are cleanest — the only cells that survive correction are OOD ones.
• Feature-based versus label-based is the split that separates the results, and it separates them in the opposite direction from the general-purpose literature: label-based scores are the ones that work here, on OOD targets, possibly for the wrong reason.
• Read Table 2 with the null in mind. At n=10 the resolution is \pm 0.25, so most cells say nothing. The three that survive correction all point the same way, which is more informative than the forty-two taken at face value.
• Together with Claßen et al. the picture is consistent: these scores were validated where source and target overlap, and medical targets break a different assumption for each of them. Neither paper shows the scores are worthless — they show the evidence base for using them in this domain does not exist yet.