3 The pipeline

A walkthrough of every stage between metadata.csv and a reported score, with the reasoning for each decision.

This notebook does not train anything. It runs in seconds and needs no image files: the data stages work from the metadata alone, and the model stages are demonstrated on small synthetic inputs. Training is ham10000 train configs/<name>.yaml; results are in 02_results.ipynb.

The stages:

metadata.csv
   -> load and validate
   -> label scheme          (which classes, and how they are grouped)
   -> lesion-level split    (the leakage control)
   -> balancing             (training set only)
   -> validation expansion  (test-time augmentation)
   -> transform
   -> model                 (architecture and freezing)
   -> training
   -> inference             (per-image probabilities)
   -> aggregation           (per-image -> per-lesion)
   -> decision rule         (sensitivity bias)
   -> evaluation

3.1 1. Load and validate

Code

from ham10000.config import Settings
from ham10000.data import load_metadata

settings = Settings.resolve()
metadata = load_metadata(settings.metadata_csv)

print(f"{len(metadata):,} images, {metadata['lesion_id'].nunique():,} lesions")
metadata.head(3)

10,015 images, 7,470 lesions

	lesion_id	num_images	image_id	dx	dx_type	age	sex	localization
0	HAM_0000118	2	ISIC_0027419	bkl	histo	80.0	male	scalp
1	HAM_0000118	2	ISIC_0025030	bkl	histo	80.0	male	scalp
2	HAM_0002730	2	ISIC_0026769	bkl	histo	80.0	male	scalp

load_metadata adds a num_images column and refuses a file with duplicate image_id values. That check matters: nothing downstream would fail loudly on a repeated identifier, but the split would silently designate the same image for a lesion twice.

3.2 2. Label scheme

The seven diagnoses can be modelled individually, or grouped. LabelScheme turns a specification into the two mappings the pipeline needs: diagnosis to integer, and integer to class name.

Code

from ham10000.data import LabelScheme

present = set(metadata["dx"].unique())

seven = LabelScheme.build(["mel", "nv", "bkl", "bcc", "akiec", "vasc", "df"], present)
binary = LabelScheme.build(
    {"malignant": ["mel", "bcc", "akiec"], "benign": ["nv", "bkl", "df", "vasc"]},
    present,
)
# Naming two diagnoses does not by itself give a two-class problem: the other
# five are still in the data and have to go somewhere.
mel_nv = LabelScheme.build(["mel", "nv"], present)
# Restricting the data first is what makes it genuinely binary. This is what
# `restrict: {dx: [mel, nv]}` does in the demo configuration.
restricted = LabelScheme.build(["mel", "nv"], {"mel", "nv"})

print("seven-class     :", seven.codes)
print("malignant/benign:", binary.codes)
print("mel and nv named:", mel_nv.codes)
print("data restricted :", restricted.codes)
import pandas as pd

pd.DataFrame(
    {
        name: {
            "melanoma index": scheme.mapping["mel"],
            "called": scheme.codes[scheme.mapping["mel"]],
            "classes": scheme.n_classes,
            "chance": round(1 / scheme.n_classes, 3),
        }
        for name, scheme in {
            "seven-class": seven,
            "malignant/benign": binary,
            "mel and nv named": mel_nv,
            "data restricted": restricted,
        }.items()
    }
).T

seven-class     : {0: 'akiec', 1: 'bcc', 2: 'bkl', 3: 'df', 4: 'mel', 5: 'nv', 6: 'vasc'}
malignant/benign: {0: 'benign', 1: 'malignant'}
mel and nv named: {0: 'other', 1: 'mel', 2: 'nv'}
data restricted : {0: 'mel', 1: 'nv'}

Table 3.1: Where melanoma lands under four class schemes.

	melanoma index	called	classes	chance
seven-class	4	mel	7	0.143
malignant/benign	1	malignant	2	0.5
mel and nv named	1	mel	3	0.333
data restricted	0	mel	2	0.5

Naming mel and nv gives three classes, not two (Table 3.1). The other five diagnoses are still in the data, so they collect in other, and the model has a third answer available. Restricting the data first is what makes the problem genuinely binary, which is why the demo configuration sets restrict: {dx: [mel, nv]} before naming its classes.

Note that melanoma has a different integer under every scheme. That is not an inconsistency to be fixed, it is what an integer label means: a position in a particular scheme, assigned by sorting the class names, with other pinned to index 0 when it exists. A stored probability table has a column per class in that order, and a trained model has one output per class in that order, so a checkpoint and a scheme only make sense together. Scoring a seven-class checkpoint under a binary scheme would silently read the first two of seven outputs and produce entirely plausible nonsense, which is why predict_probabilities refuses when the widths disagree.

Two conventions are load-bearing, because stored artefacts depend on them.

Class names are sorted before indices are assigned, so the same specification always yields the same integer for the same class. Without that, a model trained in one session is silently mismatched with predictions decoded in another.
A specification that does not cover every diagnosis gains an other class at index 0, shifting the rest up. Groups must also be disjoint: a diagnosis appearing in two of them is rejected rather than quietly assigned to whichever was declared last.

Code

partial = LabelScheme.build({"mel": ["mel"], "nv": ["nv"]}, present)
print("partial specification:", partial.codes)
print("bkl falls into       :", partial.codes[partial.mapping["bkl"]])
print("probability columns  :", partial.probability_columns)

partial specification: {0: 'other', 1: 'mel', 2: 'nv'}
bkl falls into       : other
probability columns  : ['prob_other', 'prob_mel', 'prob_nv']

3.3 3. The lesion-level split

The most consequential decision in the project. 1,956 lesions are photographed more than once, and those images are near-duplicates. Splitting at the image level would place near-identical photographs of one lesion on both sides of the boundary, so the validation score would measure memorisation rather than generalisation.

Code

from ham10000.data import SplitConfig, assign_splits, lesion_overlap

metadata["label"] = metadata["dx"].map(seven.mapping)

split = SplitConfig(train_val_ratio=3, seed=0, keep_first=False, stratified=True)
assignment = assign_splits(metadata, split)
annotated = metadata.assign(set=assignment.sets)

print(f"train lesions {len(assignment.train_lesions):,}")
print(f"val lesions   {len(assignment.val_lesions):,}")
print(annotated["set"].value_counts().sort_index().to_string())
print("\nlesions on both sides:", lesion_overlap(annotated) or "none")

train lesions 5,600
val lesions   1,870
set
t1    5600
ta    1878
v1    1870
va     667

lesions on both sides: none

The set column encodes two experiments at once:

value	meaning
`t1`	training, and the one designated image for its lesion
`ta`	training, an additional image of a lesion already represented
`v1`	validation, designated image
`va`	validation, additional image

One image per lesion uses t1/v1; all images uses t1 | ta and v1 | va. Since ta is defined as the images of training lesions excluding t1, the union is exactly the images of the training lesions, with no double counting.

stratified=True applies the ratio within each class. With dermatofibroma at 73 lesions, an unstratified split can leave a rare class badly represented on one side by chance.

Code

# Both images of a multi-image lesion land on the same side, always.
multi = annotated[annotated["num_images"] > 1]
sides = multi.assign(side=multi["set"].str[0])
print("lesions whose images span both sides:",
      int((sides.groupby("lesion_id")["side"].nunique() > 1).sum()))

lesions whose images span both sides: 0

3.4 4. Balancing

Nevi are 72% of lesions. A classifier answering “nevus” unconditionally scores about 0.72 plain accuracy while being useless, and cross-entropy rewards it for doing so. Resampling toward uniform class counts removes the incentive.

Resampling is lesion-aware, not row-level. To draw N images of a class over its D lesions, write N = Q·D + R: every lesion contributes Q images and R randomly chosen lesions contribute one more. Within a lesion holding k images, Q = q·k + r: each image taken q times, r of them once more.

Sampling is therefore as uniform as integer arithmetic allows at both levels. A lesion photographed five times does not get five times the influence of one photographed once, which is the concern that motivates the split, applied here to resampling.

Code

from ham10000.data import balance

train = annotated[annotated["set"].isin(["t1", "ta"])].copy()
targets = {dx: 1000 for dx in train["dx"].unique()}
balanced = balance(train, targets, seed=0, one_image_per_lesion=False)

import pandas as pd

comparison = pd.DataFrame({"before": train["dx"].value_counts(),
                           "after": balanced["dx"].value_counts()})
comparison["factor"] = (comparison["after"] / comparison["before"]).round(1)
comparison["lesions"] = train.groupby("dx")["lesion_id"].nunique()
comparison.sort_values("factor")

Table 3.2: Resampling factors by class, for a target of 1,000 each.

	before	after	factor	lesions
dx
nv	5007	1000	0.2	4052
bkl	823	1000	1.2	545
mel	827	1000	1.2	460
bcc	388	1000	2.6	245
akiec	244	1000	4.1	171
vasc	107	1000	9.3	73
df	82	1000	12.2	54

Note what the factor column exposes (Table 3.2). Nevi are cut to a fifth; dermatofibroma is repeated twelve times over 54 distinct lesions. Random cropping means those repeats are different views rather than identical copies, but the underlying lesion count does not change, and a recall figure for df says more about those 54 lesions than about the class.

The same arithmetic handles both directions: when N < D, Q = 0, so no lesion contributes a base image and exactly N lesions contribute one each. That is the undersampling path, and it draws from distinct lesions rather than dropping rows arbitrarily.

Balancing is applied to training data only. balance refuses a frame containing validation rows, because repeating a validation lesion would count its errors twice and inflate the reported score.

Code

try:
    balance(annotated, targets)
except ValueError as error:
    print(error)

Frame contains 2537 validation row(s). Balancing must be applied to training data only, or validation performance is inflated by repeated lesions.

3.5 5. Validation expansion

Each validation lesion is scored several times under the evaluation transform and the predictions are combined into a single verdict. This is test-time augmentation, and it only makes sense because the transform is stochastic: with a deterministic transform it would produce identical predictions and cost runtime for nothing. Every lesion is repeated the same number of times, so the expansion cannot reweight the validation set across classes.

Whether this helps is measured in section 9, once there is a run to measure it on.

Code

from ham10000.data import expand_validation

validation = annotated[annotated["set"].isin(["v1", "va"])].copy()
expanded = expand_validation(validation, 3, seed=0)

print(f"{len(validation):,} images -> {len(expanded):,} after 3x expansion")
print("repeats per lesion:", sorted(int(v) for v in expanded["lesion_id"].value_counts().unique()))

2,537 images -> 5,610 after 3x expansion
repeats per lesion: [3]

3.6 6. Transform

Declared in the config rather than in code, so an experiment’s augmentation is part of its identity.

Code

from ham10000.experiment import build_transform, load_config

config = load_config(settings.root / "configs" / "04_balanced_random_crop.yaml")
transform = build_transform(config.transform)
transform

Compose(
    RandomCrop(size=(300, 300), padding=None)
    Resize(size=[224, 224], interpolation=bilinear, max_size=None, antialias=True)
    ToTensor()
    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
)

RandomCrop(300) then Resize(224) takes a random 300x300 window of the 600x450 image and scales it to the network’s input size. That is a meaningful augmentation for this domain: a dermatoscopic image has no canonical framing, so the lesion’s position and scale within the frame carry no diagnostic information and a model should not depend on them.

Normalize uses ImageNet channel statistics, matching the distribution the pretrained weights expect.

Note that this same transform is applied at validation time, which is what makes the expansion above work.

3.7 7. Model and freezing

Transfer learning: take an ImageNet-pretrained backbone, replace its head with one sized to the task, and train some suffix of the network.

Code

from ham10000.models import FreezeStrategy, apply_freezing, build_classifier

model = build_classifier(
    "resnet18", n_classes=seven.n_classes, strategy="last_block", pretrained=False
)

total = sum(p.numel() for p in model.parameters())
for strategy in FreezeStrategy:
    apply_freezing(model, strategy)
    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
    print(f"{strategy.value:11s} {trainable:>11,} / {total:,} trainable "
          f"({trainable / total:.2%})")

all          11,180,103 / 11,180,103 trainable (100.00%)
last_block    8,397,319 / 11,180,103 trainable (75.11%)
head_only         3,591 / 11,180,103 trainable (0.03%)

Early layers hold generic edge and texture filters that transfer well from ImageNet; late layers hold class-specific structure worth re-learning. Freezing the early ones is faster and reduces overfitting on a dataset this size. head_only is a genuine baseline rather than merely a cheap one: it measures how much of the task ImageNet features already solve, with no fine-tuning at all.

3.8 8. Training

train_model is a loop over epochs: forward, loss, backward, step; then a validation pass. Loss is cross-entropy, the optimiser is Adam at 1e-4, a conservative rate appropriate to fine-tuning, since a larger one can destroy pretrained features in the first few steps.

Only the trainable parameters go to the optimiser. Handing frozen tensors to Adam would allocate optimiser state that never updates, and for optimisers with weight decay can perturb weights that were meant to stay fixed.

The final epoch is saved rather than the best. With no early stopping, a run that begins overfitting at epoch 5 still ships epoch 10, which is a real weakness and visible in the loss curves in 02_results.ipynb. Setting save_best keeps the lowest-validation-loss epoch instead, with a caveat: choosing an epoch by validation loss and then reporting validation metrics is selection on the evaluation set, so the reported score is optimistic by an unknown amount. Note also that the selection criterion and the reported metric are different quantities, so the lowest-loss epoch is not necessarily the most accurate one.

Code

print(f"epochs        {config.training.epochs}")
print(f"batch size    {config.training.batch_size}")
print(f"learning rate {config.training.learning_rate}")
print(f"save best     {config.training.save_best}")

epochs        10
batch size    32
learning rate 0.0001
save best     False

3.9 9. Inference and aggregation

The model scores each image independently. Something has to reconcile several images of one lesion into a single answer, because a lesion is what a clinician decides about. There are two places to do it, and they are not equivalent.

Code

import pandas as pd

# A small synthetic probability table: one lesion, three views.
example = pd.DataFrame({
    "lesion_id": ["L1", "L1", "L1"],
    "prob_mel":  [0.20, 0.55, 0.30],
    "prob_nv":   [0.80, 0.45, 0.70],
})
example

	lesion_id	prob_mel	prob_nv
0	L1	0.20	0.80
1	L1	0.55	0.45
2	L1	0.30	0.70

Code

from ham10000.models import aggregate_probabilities, predicted_label

# Combine probabilities first, then take one argmax.
combined = aggregate_probabilities(example, {"max": ["mel"], "min": ["nv"]})

comparison = example[["prob_mel", "prob_nv"]].copy()
comparison.columns = ["mel (per image)", "nv (per image)"]
comparison["per-image verdict"] = predicted_label(example)
comparison["mel (lesion)"] = combined["prob_mel"]
comparison["nv (lesion)"] = combined["prob_nv"]
comparison

Table 3.3: Per-image probabilities and the lesion-level values that replace them.

	mel (per image)	nv (per image)	per-image verdict	mel (lesion)	nv (lesion)
0	0.20	0.80	nv	0.55	0.45
1	0.55	0.45	mel	0.55	0.45
2	0.30	0.70	nv	0.55	0.45

Choosing max for melanoma (Table 3.3) says: if any view of this lesion looked malignant, treat the lesion as that likely to be malignant. That is an explicitly sensitivity-favouring rule, appropriate to screening. The lesion columns are constant by construction, since the whole point is to replace each image’s estimate with one figure for the lesion.

The alternative is to let each image vote and combine the labels.

Code

from ham10000.models import aggregate_predictions

voted = example.assign(pred=predicted_label(example))
aggregate_predictions(voted, seed=0)[["lesion_id", "pred", "pred_final"]]

	lesion_id	pred	pred_final
0	L1	nv	nv
1	L1	mel	nv
2	L1	nv	nv

Ties are broken at random, with a seed. The alternative is to take the first value in sorted order, which for integer labels means the lowest, and since other and nv occupy low indices that would quietly resolve tied lesions toward the benign class: the wrong direction for this problem.

Both rules are defensible. Whether the choice matters is a separate question, and a measurable one. The rest of this section uses the predictions from a completed run rather than the toy example above.

Code

run = next(
    p for p in sorted((settings.root / "models").iterdir())
    if p.name.startswith("balanced-with-random-crop") and not p.name.endswith("-smoke")
)
predictions = pd.read_csv(run / "predictions.csv")


views_per_lesion = predictions.groupby("lesion_id")["pred"].nunique()
print(f"lesions whose views disagreed: {(views_per_lesion > 1).sum():,} "
      f"of {len(views_per_lesion):,} ({(views_per_lesion > 1).mean():.1%})")

lesions whose views disagreed: 432 of 1,870 (23.1%)

Code

# The route the pipeline used: vote on per-image labels -> pred_final.
# The alternative: combine probabilities first, then take one argmax.
by_probability = aggregate_probabilities(predictions, {"max": ["mel"]})
by_probability["alt"] = predicted_label(by_probability, seven.codes)

per_lesion = by_probability.drop_duplicates("lesion_id")
differ = per_lesion["alt"] != per_lesion["pred_final"]
print(f"routes disagree on {differ.sum():,} of {len(per_lesion):,} lesions "
      f"({differ.mean():.1%})")

routes disagree on 175 of 1,870 lesions (9.4%)

Views disagreed on 432 of 1,870 lesions (23.1%), but most of those are settled the same way by both rules. On 175 lesions (9.4%, about one in eleven) the two routes reach different verdicts. For those, the choice of aggregation rule is the answer.

The 23.1% also explains why the score moves between evaluations. Those lesions sit near the decision boundary, and re-cropping shifts some of them across it.

Code

disagreeing = per_lesion[differ]

# Prefer a case where combining probabilities lands on melanoma, since that is
# the direction the `max` rule is chosen for.
melanoma = seven.mapping["mel"]
preferred = disagreeing[disagreeing["alt"] == melanoma]
chosen = (preferred if len(preferred) else disagreeing).iloc[0]["lesion_id"]

views = predictions[predictions["lesion_id"] == chosen]
table = views[seven.probability_columns].round(3)
table["per-image verdict"] = views["pred"].map(seven.codes)

print(f"lesion {chosen}, one of {len(disagreeing)} where the two routes differ")
print(f"true diagnosis        : {views['dx'].iloc[0]}")
print(f"vote on labels        : {seven.codes[views['pred_final'].iloc[0]]}")
print(f"combine probabilities : "
      f"{seven.codes[disagreeing.set_index('lesion_id').loc[chosen, 'alt']]}")
table

lesion HAM_0002730, one of 175 where the two routes differ
true diagnosis        : bkl
vote on labels        : bcc
combine probabilities : mel

Table 3.4: One validation lesion where the two aggregation routes differ.

	prob_akiec	prob_bcc	prob_bkl	prob_df	prob_mel	prob_nv	prob_vasc	per-image verdict
0	0.009	0.001	0.150	0.000	0.840	0.000	0.000	mel
1	0.071	0.537	0.185	0.008	0.189	0.004	0.006	bcc
2	0.012	0.242	0.690	0.004	0.047	0.002	0.003	bkl

Lesion HAM_0002730 is a benign keratosis. One view read it as melanoma at 0.84, another as basal cell carcinoma at 0.54, the third as keratosis at 0.69. Voting gives bcc; combining probabilities with max on melanoma gives mel.

Both are wrong, which makes this a more useful example than one where the sensitivity-favouring rule is vindicated. It shows the cost of the choice rather than only the benefit: max turns a single confident view into a melanoma call, and here that view was mistaken. In screening terms this is a false alarm and a biopsy, which is the price paid for catching the cases where the one suspicious view is right.

This also lets us check whether the expansion in section 5 earns its place. Over ten evaluations at each setting, three views scored 0.694 against 0.686 for one view, and the run-to-run spread fell from 0.018 to 0.014. The 23.1% figure above is why that spread exists at all: those lesions sit near the boundary, and re-cropping moves some of them across it. Neither difference is significant at this sample size, so the technique is retained because it is standard and costs little, not because this dataset shows it earning its keep.

3.10 10. The decision rule

Plain argmax is wrong for melanoma screening. A lesion at 45% melanoma against 50% nevus is called a nevus, and the two errors are not symmetric: a missed melanoma delays treatment of a cancer with a sharp survival gradient in time to diagnosis, while a false alarm costs a biopsy.

Two rules are available. They are different rules, not two implementations of one, and they disagree on roughly one lesion in six.

Code

from ham10000.models import apply_cost_sensitive_weights, apply_priority_thresholds

cases = pd.DataFrame({
    "prob_mel": [0.45, 0.35, 0.42],
    "prob_bcc": [0.05, 0.00, 0.45],
    "prob_nv":  [0.50, 0.65, 0.13],
})

rules = [("plain argmax", cases)]
rules.append(("priority (A)",
              apply_priority_thresholds(cases, promote=[("mel", 0.4), ("bcc", 0.4)])))
rules.append(("cost-sensitive (C)",
              apply_cost_sensitive_weights(cases, thresholds=[("mel", 0.4), ("bcc", 0.4)])))
pd.DataFrame({name: predicted_label(frame) for name, frame in rules})

Table 3.5: Three decision rules on the same three probability vectors.

	plain argmax	priority (A)	cost-sensitive (C)
0	nv	mel	mel
1	nv	nv	mel
2	bcc	mel	bcc

Rule A, priority-ordered (Table 3.5), walks the list and promotes the first class clearing its bar, then stops. The ordering carries the clinical priority: melanoma is checked before basal cell carcinoma because it matters more. It is a gate: below the bar it does nothing at all, which is why row 2 stays nv. The rule is unorthodox: most work applies per-class thresholds independently or reweights, rather than walking an ordered list and stopping. However, the ordering makes the clinical priority explicit in a way a set of weights does not.

Rule C, cost-sensitive, divides each probability by its threshold and takes the argmax. It is an unconditional reweighting, so it acts even when nothing crosses a bar, and row 2 flips to mel. Its effective condition is p_mel > 0.4 · p_nv rather than p_mel > 0.4 (much weaker).

Rule A is the default. It is easy to reason about, and its priority ordering is an explicit statement of which class matters most, which suits a screening problem. Rule C is the more standard method and is available as an alternative; switching would change every reported figure, so the two are kept separate rather than swapped.

Row 3 shows why a third variant is unattractive. If both classes over 0.4 were promoted to 1.0, the tie would fall to column order, which is alphabetical, so bcc would beat mel because “b” precedes “m”. That puts an arbitrary rule where a clinical one belongs, and it is the reason Rule A stops at the first class rather than promoting all of them.

3.11 11. Evaluation

Per lesion, not per image, so a lesion photographed five times counts once.

Code

import numpy as np

from ham10000.evaluation import evaluate
from ham10000.reporting import confusion_with_recall

target = np.array([0, 0, 0, 0, 1, 1])
prediction = np.array([0, 0, 0, 1, 0, 1])
codes = {0: "nv", 1: "mel"}

report = evaluate(target, prediction)
print(f"plain accuracy    {report.accuracy:.3f}")
print(f"balanced accuracy {report.balanced_accuracy:.3f}")
print(f"F2               {report.f2:.3f}")
print()
print(confusion_with_recall(target, prediction, codes).to_string())

plain accuracy    0.667
balanced accuracy 0.625
F2               0.625

pred  nv  mel  support  recall
true                          
nv     3    1        4    0.75
mel    1    1        2    0.50

Balanced accuracy, the mean of per-class recall, is the headline, because plain accuracy rewards predicting the majority class. Its chance level is 1/n_classes, so scores from tasks with different class counts are not directly comparable.

F2 weights recall four times as heavily as precision, which is the direction a screening tool should err in. F1/2 is reported alongside for contrast, not because it is appropriate here.

Support sits beside recall because a recall of 1.00 over 18 lesions is not evidence of much.