split_dataset

split_dataset(
    root,
    out_dir,
    *,
    mode=DataMode.NESTED_DIRS,
    subdirs=('images', 'masks'),
    column_sep=';',
    fractions=(0.7, 0.15, 0.15),
    group_by=None,
    seed=0,
    strict=True,
    relative=True,
    label_column='masks',
)

Split one folder of data into three CSVs a specification can point at.

The whole point of the function: a researcher has one directory and needs train_path, validation_path and test_path. Returns the paths it wrote plus the counts, so the caller can see what happened without opening the files.

label_column names the second column, and it has to match what the specification will look for - masks for segmentation, annotations for detection. Not derived from subdirs, although that would read well: someone whose directories are called img and lbl has always got a masks column out of this, and deriving it would quietly write a file their existing configuration could no longer read.

The other way to divide one folder is the specification’s own split block, which partitions the samples in memory and writes nothing. Use this one when the CSVs are the point - to keep, to hand to a colleague, or to cite in a paper.

Parameters

Name	Type	Description	Default
root	str | Path	The folder holding the data.	required
out_dir	str | Path	Where the three CSVs go.	required
mode	DataMode | str	nested_dirs for one directory per sample, config_file for a CSV.	DataMode.NESTED_DIRS
subdirs	tuple[str, str]	For nested_dirs, the image and label subdirectories.	('images', 'masks')
column_sep	str	What separates several paths inside one cell - a sample with one mask file per object, or one file per channel. Not the CSV’s own delimiter, which is a comma: that is the distinction this argument’s name does not make.	';'
fractions	Sequence[float] | Mapping[str, float]	Two or three, as for split_samples.	(0.7, 0.15, 0.15)
group_by	str | None	A pattern picking the group out of each sample’s key - a patient, usually. As for split_samples, and for the same reason.	None
seed	int	Which shuffle.	0
strict	bool	Whether a sample that cannot be read is an error rather than a skip.	True
relative	bool	Write paths relative to out_dir, so the CSVs survive being moved together with the data.	True
label_column	str	The name of the second column.	'masks'

Returns

Name	Type	Description
	dict[str, object]	paths - the CSV written for each split that got any; samples and groups -
	dict[str, object]	how many of each went where, test included and zero when there are two fractions;
	dict[str, object]	and skipped, which is not zero when strict is False and something could not be
	dict[str, object]	read.

import pathlib, tempfile root = pathlib.Path(tempfile.mkdtemp()) for patient in range(1, 5): … for slice_no in range(2): … case = root / f”patient{patient:02d}slice{slice_no}” … (case / “images”).mkdir(parents=True) … (case / “masks”).mkdir(parents=True) … = (case / “images” / “scan.png”).write_bytes(b”“) … _ = (case /”masks” / “mask.png”).write_bytes(b”“) out = split_dataset( … root, … root /”splits”, … fractions=(0.5, 0.5), … group_by=r”^(patient)_“, … ) sorted(out) [‘groups’, ‘paths’, ‘samples’, ‘skipped’] out[“samples”] {‘train’: 4, ‘validation’: 4, ‘test’: 0}

Two groups each side, because a patient is never divided:

out[“groups”] {‘train’: 2, ‘validation’: 2, ‘test’: 0}

And the CSVs are what a specification points at. Note the header: the file is comma-separated and carries the key and the group it was assigned, while column_sep separates several paths within a single cell.

written = pathlib.Path(out[“paths”][“train”]).read_text().splitlines() written[0] ‘key,group,images,masks’ len(written) - 1 4