Checkpoint and resume

spaCR checkpoints long work only after a safe unit has finished. A checkpoint never means killing a write halfway through: it means that a later invocation can prove which fields, trials, or plate jobs settled and continue after them. The Qt Stop button uses the same boundaries: it requests cooperative cancellation, lets the current safe unit finish, and retains the checkpoint for the next run. See Threading and cancellation audit.

Supported workflows

Mask

Masks are matched to fields by exact filename. An object role is skipped only when every .npy field in stack/ has a structurally complete mask of the same name; a missing, empty, or truncated mask sends the role back through segmentation. An unrelated mask cannot stand in for a missing field, and extra masks do not force complete fields to run again. Enable resume under Workflow & Test Run to also keep verified merged/*.npy fields instead of rebuilding them. New mask arrays are written with temporary-file plus atomic replace semantics. Cell adjustment first checks that the pathogen, cell, and nucleus folders hold the same field filenames with complete masks of matching dimensions; if they differ, it stops before changing any mask. With timelapse and motility_analysis on, the motility assay runs once per plate, after every object role and merged frame is complete, and rebuilds its measurements from the current masks rather than reusing an earlier table.

Measure

Enable resume under Runtime & Reliability. A field is skipped only when every Measure-owned table in measurements.db is complete. Partial field rows are cleared in one transaction before remeasurement, while tables owned by conversion, alignment, or other modules are never deleted.

Format Converter

Enable the Apple-style Resume switch. The converter writes .spacr_conversion.checkpoint.json after a whole field is complete. Resume reopens each target’s TIFF metadata before accepting the field and atomically repairs a missing or corrupt target. Source identity and the full mapping plan must match.

Image UMAP search

Open UMAP settings, then enable Resume checkpoint. Trial metadata is written after every trial and embeddings are stored as adjacent NumPy artifacts. Grid searches skip completed configurations. Adaptive 2×2 searches also persist their centre, best score, completed-round count, and partial-round corners, so only missing corners run after an interruption.

Batch Runner

A saved queue is written after every job transition. Loading and resuming it leaves successful jobs alone. A Mask, Measure, or Format Converter job that was running when the machine stopped is restarted with resume=True, so its own verified field boundary is reused. Other jobs restart at the job boundary.

Interrupted normalization archives

Mask also checks the normalized masks/*.npz archives before segmentation. It checks ZIP member extents, a readable numeric data.npy header, enough declared pixel bytes for the shape and dtype, and one filename per batch field. This structural check does not inflate every pixel array or perform a full pixel CRC scan.

Damaged archives are set aside as <name>.damaged. With preprocessing disabled, the run stops and explains how to recover them. The same quarantine still prevents a later run from silently omitting those fields. It is resolved by a valid same-name replacement, or by readable archives covering every existing field in stack/. Legacy object-valued filename arrays cannot prove that coverage. Enabling preprocessing lets recoverable fields be rebuilt from the available source data; valid archives and quarantined evidence are preserved.

Safety rules

Checkpoint files are atomic JSON documents with a workflow name, format version, boundary, timestamps, input/settings signature, status, completed units, and workflow state. spaCR refuses a resume when this signature differs; start with Resume disabled to create a fresh checkpoint. A corrupt checkpoint is reported and preserved for diagnosis rather than silently discarded.

The persistence API is spacr.checkpoint.CheckpointStore. Conversion uses spacr.convert.convert(), UMAP search uses spacr.hyperparam.umap_search(), Measure uses spacr.resume.plan_measure_resume(), and queues use spacr.batch.resume_queue().