Train/test leakage audit

Classifier accuracy is invalid when related crops occur on both sides of an evaluation boundary. spaCR now audits the permanent train//test/ split, the ordinary train/validation holdout, every outer and inner CV boundary, and the CV partition as a whole before fitting.

What is verified

The audit checks:

• the same path or crop identity never crosses a boundary;

• exported augmentations such as _rot90, _flip_h and _aug3 remain with their source object;

• byte-identical crops are detected by streaming SHA-256 even if renamed;

• the requested cv_group_by identity (field, well or plate) stays intact;

• every CV sample is held out exactly once;

• no related family is assigned to more than one held-out fold; and

• related crops do not carry conflicting class labels.

Filename identities follow spaCR’s <plate>_<well>_<field>_..._<object>.png crop convention. With leakage_require_identity=True (the default), a filename that cannot prove its requested identity fails the audit; an unknown relationship is not reported as independent.

Ordinary validation is now group-aware too. cv_group_by='well' uses the same group-stratified partitioner as CV and selects the candidate fold closest to val_split and the full dataset’s class distribution. Augmentation is applied only after the split and only to training.

Run the audit directly

spacr-leakage /data/classifier_dataset
spacr-leakage /data/classifier_dataset --group-by plate \
    --output /tmp/leakage.json

The command prints JSON and exits 0 for a verified split, 1 when leakage or unverifiable identities are found, and 2 when the dataset cannot be audited. --no-content-hash reduces I/O but cannot detect renamed copies. --allow-unverifiable is intended for diagnosing legacy datasets; it does not make their performance estimate trustworthy.

Classify settings

leakage_audit_train_test

Audit train/ against test/ before fitting. Default True.

leakage_hash_content

Stream SHA-256 for copy/rename detection. Default True.

leakage_require_identity

Fail when protected identity or content cannot be verified. Default True.

evaluation_fail_on_leakage

Stop before fitting when an audit fails. Default True. Setting it to False records a failed audit but cannot make the resulting metric valid.

Reports are written beside the classifier run as train_test_leakage_audit.json and train_validation_leakage_audit.json. CV audit records are also included in the Classifier Evaluation workbench’s leakage.json.

API

Classifier evaluation, calibration, and split-leakage diagnostics.

The module is model-agnostic: it consumes labels, probabilities, fold ids, and sample paths. Deep-learning CV and future classical-ML pipelines can therefore write the same evaluation bundle and the Qt workbench can display one stable artifact format.

class spacr.classifier_evaluation.FoldLeakageAudit(group_by: str, n_samples: int, n_folds: int, validation_membership_missing: ~typing.List[int] = <factory>, validation_membership_duplicate: ~typing.List[int] = <factory>, overlap_counts: ~typing.Dict[str, int] = <factory>, examples: ~typing.Dict[str, ~typing.List[str]] = <factory>, critical_levels: ~typing.List[str] = <factory>, warnings: ~typing.List[str] = <factory>, unverifiable_counts: ~typing.Dict[str, int] = <factory>, hash_errors: ~typing.List[str] = <factory>, split_name: str = 'all_cv_folds')

Whole-CV proof that each related sample family belongs to one fold.

property passed: bool

Return True only when the fold partition is complete and isolated.

to_dict() → Dict[str, Any]

Return a JSON-serializable audit record.

class spacr.classifier_evaluation.LeakageReport(group_by: str, train_samples: int, validation_samples: int, overlap_counts: ~typing.Dict[str, int], examples: ~typing.Dict[str, ~typing.List[str]], split_name: str = '', critical_levels: ~typing.List[str] = <factory>, warnings: ~typing.List[str] = <factory>, unverifiable_counts: ~typing.Dict[str, int] = <factory>, hash_errors: ~typing.List[str] = <factory>)

Overlap counts and examples for one train/validation boundary.

Variables:

• group_by – protected split level (none, field, well, or plate).

• train_samples – number of training paths.

• validation_samples – number of validation paths.

• overlap_counts – overlap count at each identity level.

• examples – up to ten shared identities per level.

• critical_levels – levels that invalidate the requested split.

• warnings – non-fatal caveats.

property passed: bool

Return True when no protected identity crosses the boundary.

to_dict() → Dict[str, Any]

Return a JSON-serializable report.

spacr.classifier_evaluation.audit_cv_folds(paths: Sequence[Any], folds: Sequence[Tuple[Sequence[int], Sequence[int]]], *, labels: Sequence[Any] | None = None, group_by: str = 'well', hash_content: bool = False, require_identity: bool = True, raise_on_leakage: bool = False) → FoldLeakageAudit

Verify partition coverage and identity isolation across every CV fold.

This checks the fold assignment as one object, rather than trusting a sample of pairwise boundaries. Each index must be validation exactly once; exact paths, byte-identical content, source/augmentation families and the requested plate/well/field group must map to one held-out fold only.

spacr.classifier_evaluation.audit_dataset_splits(root: Any, *, group_by: str = 'well', hash_content: bool = True, require_identity: bool = True, raise_on_leakage: bool = False) → LeakageReport

Audit the permanent train/ versus test/ dataset boundary.

spacr.classifier_evaluation.audit_split_leakage(train_paths: Sequence[Any], validation_paths: Sequence[Any], *, group_by: str = 'well', raise_on_leakage: bool = False, split_name: str = '', hash_content: bool = False, require_identity: bool = False) → LeakageReport

Detect related images crossing a train/validation boundary.

Exact/object/augmentation-family overlap is always critical. The requested group_by level is also critical: a well-grouped split permits the same plate on both sides but never the same well.

Parameters:

• train_paths – source paths used to fit the model.

• validation_paths – paths used only for evaluation.

• group_by – none, field, well, or plate.

• raise_on_leakage – raise LeakageError on a critical overlap.

• split_name – optional fold/split label stored in the report.

Returns:

LeakageReport.

spacr.classifier_evaluation.write_leakage_audit(path: Any, audit: Any) → Path

Atomically write a leakage report/audit as JSON and return its path.