Metrics

Description

What gets reported. Unlike the losses these are computed on the hard prediction - the mask you will actually be handed - because a metric taken on probabilities reads higher than the model deserves.

Usage

metric_iou(smooth = 1, include_background = TRUE)

metric_dice(smooth = 1, include_background = TRUE)

metric_tversky(alpha = 0.5, smooth = 1, include_background = TRUE)

metric_cldice(iterations = 5, smooth = 1, include_background = FALSE)

metric_surface_dice(tau, smooth = 1, include_background = FALSE)

Arguments

smooth	Smoothing. Zero is allowed and is the honest choice for a reported number: with smoothing, a class absent from an image scores a perfect 1.
include_background	Average over the background class as well. In medical images the background is usually most of the picture, so leaving it in can turn a model that found nothing into one that looks respectable. Papers report foreground only. It defaults to TRUE for the three overlap metrics and FALSE for metric_cldice() and metric_surface_dice(), where the background class does not merely flatter a model but measures something that is not the question - its skeleton and its boundary are both artefacts of the frame rather than of what was found. The paragraphs above give both measurements.
alpha	For Tversky, the weight on false negatives.
iterations	For clDice, how many times the skeleton is peeled. It has to reach the half-width of the thickest structure, or its core is never reduced to a centreline and the score is refused rather than reported - an empty skeleton otherwise reads as a perfect 1.
tau	For metric_surface_dice(), the tolerance a boundary is allowed to be wrong by, and it is required because no default would be honest. A tolerance belongs to the structure and the dataset rather than to a package: conventionally it is the inter-observer variability for that structure - the distance two people drawing the same boundary differ by - and published values exist per organ and per challenge. Take one from the protocol you are comparing against. In millimetres where target_spacing puts every case on a common grid, and in voxels where it does not, which is the same pair of answers the signed distance map in loss_boundary() gives and for the same reason: a 0.5 mm and a 2.5 mm acquisition are not comparable in voxels at all.

Details

Three of these ask the same question and two ask different ones. metric_iou(), metric_dice() and metric_tversky() all ask how much of the mask coincides and differ in how they weigh the ways of being wrong. metric_cldice() asks whether the structure coincides, and metric_surface_dice() asks how far wrong the boundary is - which is the question a contour somebody has to approve is actually judged by, and the one nothing here reported until the pin moved to 0.8.0a26.

For something thin and connected clDice is the question worth asking, and it and Dice disagree in a way worth knowing. Measured on a nine-pixel vessel: one drawn two pixels thin scores Dice 0.875, and the same vessel severed in the middle scores 0.947 - Dice prefers the severed one. clDice gives them 1.000 and 0.943.

Its limit, also measured: at one pixel wide a displacement of one pixel leaves no overlap at all, and every overlap-based metric reads 0, this one included.

metric_surface_dice() is the other kind of disagreement. loss_boundary()’s own page reports a model at Dice 0.9525 whose volumes run 5% small and 17% wrong per case, and no overlap column says that: a displaced boundary and a correct one of the same area score alike. This is normalised surface Dice (Nikolov et al., 2018), symmetric on purpose - the truth’s surface near the prediction is the boundary’s recall, the prediction’s surface near the truth is its precision - and it is the one metric here that reads the voxel spacing, so tau is in millimetres when target_spacing has put the data on a common grid and in voxels when it has not. It is also what loss_boundary() optimises, so until now that loss was being judged by the metric it exists to improve on.

Like clDice it is a property of a whole mask rather than of a batch of counts, so on a tiled run (splits) it is reported per case by evaluate_cases() and is absent from the epoch history - a per-batch average of per-tile boundaries would be a number about nothing. A callback asking to watch val_surface_dice on such a run is refused when the configuration is read, the way one watching val_cldice is - so the choice to tile and the choice of what to stop on are settled together, before anything is trained.

include_background defaults to FALSE for those two and TRUE for the three overlap metrics, and each has its own measured reason. The background’s skeleton lies inside the background by construction, so that class scores 1 for clDice whatever the model did - averaging it in moved 0.7865 to 0.8933 on a three-pixel vessel. The background’s boundary is the foreground’s boundary plus the edge of the picture, and that edge agrees between any two masks whatever happened in them: measured on four random pairs at tau = 2, the background scored 0.6110 to 0.7030 against the foreground’s 0.5583 to 0.6266 - higher every time. That is agreement on the field of view rather than on the object.

Value

A metric specification.

Examples

library("rplatypus")

metric_dice(include_background = FALSE)

$name
[1] "dice"

$smooth
[1] 1

$include_background
[1] FALSE

# Vessels, airways, neurons, cracks: thin and connected, where Dice asks the wrong thing.
metric_cldice()

$name
[1] "cldice"

$iterations
[1] 5

$smooth
[1] 1

$include_background
[1] FALSE

# How far wrong the boundary is, rather than how much of the area coincides. 1.5 mm
# because that is this structure's inter-observer variability, not because of anything
# here.
metric_surface_dice(tau = 1.5)

$name
[1] "surface_dice"

$tau
[1] 1.5

$smooth
[1] 1

$include_background
[1] FALSE