InstanceEngine.evaluate

InstanceEngine.evaluate(split='validation')

One row per model and decoding: the comparison this task is for.

A decoding costs a hundredth of an epoch, so the table gains an axis for nearly nothing.

This paragraph used to say connected_components is in every model’s rows whether it was asked for or not, and it is not. It is in the pair decode derives to when a model stayed silent, which is the signal’s own decoding plus the yardstick; a model that writes a decode list gets the list it wrote and nothing added. That is the right behaviour - the engine does not put a row in somebody’s table that they did not ask for - and the claim was wrong rather than the code: the only test on it covered the derived case, and examples/compare_instance_separations.py keeps the yardstick in its own --decode handling precisely because nothing below does it. test_a_written_decode_list_is_what_you_get is the half that was missing.

The reason the yardstick belongs in a comparison is unchanged and is why the derivation includes it: it ignores the separation signal, so it is the row the signal’s row has to be read against.

Parameters

Name	Type	Description	Default
split	str	Which data to score on.	'validation'

Returns

Name	Type	Description
	list[dict[str, Any]]	One row per (model, decoding): what the model is, what it was trained on, the
	list[dict[str, Any]]	mask’s own numbers, and the instance average precision with the object counts
	list[dict[str, Any]]	that explain it. average_precision is the mean over images, which is what the
	list[dict[str, Any]]	Data Science Bowl ranked on.
	list[dict[str, Any]]	**foreground_dice is the mask column to read across models and dice is
	list[dict[str, Any]]	not.** The first is a binary overlap of anything the model called an object
	list[dict[str, Any]]	against anything that is one, computed identically whatever the separation
	list[dict[str, Any]]	signal is, unsmoothed and averaged over images - the same pooling as the
	list[dict[str, Any]]	average_precision beside it. The second is the training metric, a mean over
	list[dict[str, Any]]	the class channels that model’s head has, so it is read down one model’s rows
	list[dict[str, Any]]	and never across two. Both are the same for every decoding of one model.
	list[dict[str, Any]]	pyplatypus#222 has the measurement that separates them: 0.09 apart in the
	list[dict[str, Any]]	class column where the masks are 0.0062 apart, and in the other direction.

Raises

Name	Type	Description
	EngineError	If nothing has been trained, or the split carries no masks.