Checkpoint and resume

A checkpoint can be initialized before any unit finishes. Supported workflows mark completed units of work, such as fields, trials or plate jobs, at their checkpoint boundaries. A later resume validates the stored workflow identity and uses its completion records and workflow-specific output checks to decide what can be reused. A checkpoint is not a guarantee against forced process termination or damaged output files. The Qt Stop button uses the same boundaries: it requests cooperative cancellation, lets the current safe unit finish, and retains the checkpoint for the next run. See Threading and cancellation audit.

Supported workflows

Mask

Masks are matched to fields by exact filename. An object role is skipped only when every .npy field in stack/ has a structurally complete mask of the same name; a missing, empty, or truncated mask sends the role back through segmentation. An unrelated mask cannot stand in for a missing field, and extra masks do not force complete fields to run again. Enable resume under Workflow & Test Run to also keep verified merged/*.npy fields instead of rebuilding them. New mask arrays are written with temporary-file plus atomic replace semantics. Cell adjustment first checks that the pathogen, cell, and nucleus folders hold the same field filenames with complete masks of matching dimensions; if they differ, it stops before changing any mask. With timelapse and motility_analysis on, the motility assay runs once per plate, after every object role and merged frame is complete, and rebuilds its measurements from the current masks rather than reusing an earlier table.

Measure

Enable resume under Postprocessing → Runtime & Reliability. A field is skipped only when every Measure-owned table in measurements.db is complete. Partial field rows are cleared in one transaction before remeasurement, while tables owned by conversion, alignment, or other modules are never deleted.

Format Converter

Enable the Resume switch. The converter writes .spacr_conversion.checkpoint.json after a whole field is complete. Resume reopens each target’s TIFF metadata before accepting the field and atomically repairs a missing or corrupt target. Source identity and the full mapping plan must match.

Image UMAP search

Open UMAP settings, then enable Resume checkpoint. Trial metadata is written after every trial and embeddings are stored as adjacent NumPy artifacts. Grid searches skip completed configurations. Adaptive 2×2 searches also persist their centre, best score, completed-round count, and partial-round corners, so only missing corners run after an interruption.

Batch Runner

A saved queue is written after every job transition. Loading and resuming it leaves successful jobs alone. A Mask, Measure, or Format Converter job that was running when the machine stopped is restarted with resume=True, so its own verified field boundary is reused. Other jobs restart at the job boundary. Failed and skipped jobs are left unchanged unless retry_failed=True is passed to spacr.batch.resume_queue().

Interrupted normalization archives

Mask also checks the normalized masks/*.npz archives before segmentation. It checks ZIP member extents, a readable numeric data.npy header, enough declared pixel bytes for the shape and dtype, and one filename per batch field. This structural check does not inflate every pixel array or perform a full pixel CRC scan.

New normalized V1 Mask archives record the ordered original channel IDs in source_channels. Reuse checks this order against the current object-role channel choices; saved cellpose_<role>_channel values are derived positions, not overrides. Missing or incompatible channel provenance stops segmentation with preprocessing off. Enable preprocessing to rebuild from the original stack/ fields; if those fields are unavailable, restore the original images before retrying. When rebuilding archives with incompatible channel provenance, prior mask stacks and merged/ outputs are moved to previous_segmentation_channel_order_* for safekeeping rather than reused with the new channel order.

Damaged archives are set aside as <name>.damaged. With preprocessing disabled, the run stops and explains how to recover them. The same quarantine still prevents a later run from silently omitting those fields. It is resolved by a valid same-name replacement, or by readable archives covering every existing field in stack/. Archives that store filenames as object-valued arrays cannot prove that coverage. Enabling preprocessing lets recoverable fields be rebuilt from the available source data. Quarantined evidence is preserved. Compatible whole archives can be retained; corrected or channel-incompatible archive sets may need to be rebuilt completely.

Safety rules

Checkpoint files are atomic JSON documents with a workflow name, format version, boundary, timestamps, input/settings signature, status, completed units, and workflow state. spaCR refuses a resume when this signature differs; start with Resume disabled to create a fresh checkpoint. A corrupt checkpoint is reported and preserved for diagnosis rather than silently discarded.

The persistence API is spacr.checkpoint.CheckpointStore. Conversion uses spacr.convert.convert(), UMAP search uses spacr.hyperparam.umap_search(), Measure uses spacr.resume.plan_measure_resume(), and queues use spacr.batch.resume_queue().