spacr.pipeline_v2

Streaming mask pipeline (v2).

An alternative to the multi-copy disk chain that preprocess_generate_masks runs by default (pipeline_style='v1'):

originals

→ renamed + split into channel folders → orig/ backup → per-channel npy → batch npz on disk → cellpose → per-field mask npy → concatenated into merged/

This module instead runs a two-pass streaming pipeline that keeps only what the downstream measure module reads:

Pass 1 — assemble

walk originals, parse metadata regex, build one npy stack per field with all image channels in the C axis. Emit filename_map.csv recording every original → stack mapping.

Pass 2 — segment

stream the plate in batches of N fields, hand each batch to Cellpose, append the mask channels to the SAME stack file. Each batch is written to a scratch NPZ on the way through and deleted again unless keep_npz=True.

Output — merged/ folder holds one file per field, each shape (H, W, C_image + C_mask) in uint16, plus:

channel_order.json {“channels”: […]} filename_map.csv original path, plate/well/field/…, stack idx

Public API:

from spacr.pipeline_v2 import (
    FilenameMapper, stream_originals_to_stack,
    stream_masks_from_stack, run_v2,
)

# High-level (one call):
run_v2(src_folder, channels=(0,1,2,3), model="cyto", diameter=60)

# Low-level (two passes, run each explicitly):
mapper = FilenameMapper.discover(src_folder,
                                   metadata_type="cellvoyager")
stacks = stream_originals_to_stack(src_folder, mapper, channels=(0,1,2,3))
stream_masks_from_stack(stacks, model="cyto", diameter=60)

This module is opt-in: spacr.core.preprocess_generate_masks() runs it when pipeline_style='v2', and the default 'v1' runs the disk chain above.

Classes

FilenameMapper	Walks a folder of microscopy images, parses each filename's
FilenameRecord	One entry in the filename map.
StackFile	One field's on-disk stack: merged/stack_<id>.npy with

Functions

run_v2(, channel_names, model_name, ...)	Run the entire v2 pipeline against src. Convenience wrapper.
stream_masks_from_stack(, diameter, batch_fields, ...)	Batch the field stacks through Cellpose, then append the mask
stream_originals_to_stack(, channel_names, dst, *, ...)	Write one merged/stack_<field>.npy per field.

Module Contents

class spacr.pipeline_v2.FilenameMapper(records: List[FilenameRecord], metadata_type: str, regex: str)

Walks a folder of microscopy images, parses each filename’s metadata via a regex, and records the mapping to a per-plate CSV.

The CSV is written next to the merged/ folder (at the plate root) so users can Excel-open filename_map.csv and see the original path of every image in the run.

Parameters:

• records – parsed filename records in file-system order.

• metadata_type – name of the metadata convention used to parse them.

• regex – regular-expression source used for parsing.

Variables:

• records – list of FilenameRecord in file-system order.

• metadata_type – which regex was used ("cellvoyager" / "yokogawa" / "custom").

• regex – compiled regex pattern that matched.

Store parsed filename records and the metadata rule that made them.

by_field() → Dict[str, List[FilenameRecord]]

Group records by stack_field_id — one entry per field, with one record per channel inside.

classmethod discover(src: pathlib.Path, metadata_type: str = 'auto', custom_regex: str | None = None, exts: Sequence[str] = ('.tif', '.tiff', '.png', '.jpg', '.jpeg')) → FilenameMapper

Scan src for images + parse each name with the metadata regex. Falls back through cellvoyager → yokogawa on metadata_type="auto".

Parameters:

• src – folder to scan. Not recursive: images are expected at the top level, as in the current spacr layout.

• metadata_type – "auto" / "cellvoyager" / "yokogawa" / "custom". When "custom", custom_regex must be given.

• custom_regex – user-supplied regex; required for metadata_type="custom".

• exts – image file extensions to include.

Returns:

a populated FilenameMapper.

Raises:

ValueError – when no images are found or no regex fits.

field_ids() → List[str]

Return the sorted list of unique stack_field_id values.

classmethod load_csv(path: pathlib.Path) → FilenameMapper

Rehydrate a mapper from a previously-saved CSV.

Parameters:

path – mapping CSV previously written by save_csv().

save_csv(path: pathlib.Path) → pathlib.Path

Write the mapping to path as a CSV that Excel opens cleanly. One row per (original image, resulting stack slot).

Parameters:

path – destination CSV path; its parent directory is created.

class spacr.pipeline_v2.FilenameRecord

One entry in the filename map.

Variables:

• original_path – absolute path to the source image on disk.

• plate – plate id parsed from the filename.

• well – well id parsed from the filename.

• field – field index parsed from the filename.

• channel – channel index parsed from the filename.

• time – time index parsed from the filename (defaults to 1).

• z – z-slice index parsed from the filename (defaults to 1).

• stack_field_id – the field id used in merged/stack_<X>.npy.

class spacr.pipeline_v2.StackFile

One field’s on-disk stack: merged/stack_<id>.npy with shape (H, W, C).

Populated by stream_originals_to_stack() before Cellpose runs (C = image channels only). After stream_masks_from_stack() the same file has additional mask channels appended.

Variables:

• field_id – stable field identifier used in the stack filename.

• path – path to the on-disk NumPy stack.

• shape – (height, width, channels) shape at write time.

• channels – human-readable channel names in array order.

spacr.pipeline_v2.run_v2(src: pathlib.Path, channels: Sequence[int] = (0, 1, 2, 3), channel_names: Sequence[str] | None = None, model_name: str = 'cyto', channels_for_cellpose: Sequence[int] = (0, 0), diameter: float | None = None, batch_fields: int = 8, metadata_type: str = 'auto', custom_regex: str | None = None, keep_npz: bool = False, cellprob_threshold: float = 0.0, flow_threshold: float = 0.4, min_size: int = 15, resample: bool = True, postprocess_settings: Dict[str, Any] | None = None, object_type: str = 'cell', illumination_settings: Dict[str, Any] | None = None) → Dict[str, Any]

Run the entire v2 pipeline against src. Convenience wrapper.

Equivalent to:

mapper = FilenameMapper.discover(src, metadata_type, custom_regex)
stacks = stream_originals_to_stack(src, mapper, channels, channel_names)
stream_masks_from_stack(stacks, model_name, channels_for_cellpose,
                        diameter, batch_fields, keep_npz=keep_npz)

Parameters:

• src – plate folder holding the originals, scanned non-recursively. Normally the mapping CSV lands here and output goes to <src>/merged. Calibrated runs use a fresh calibrated_segmentation output directory instead, preserving source metadata and existing merged arrays. Occupied names receive a numeric suffix; calibrated results are never resumed.

• channels – channel numbers as parsed from the filename (C01 gives 1), not C-axis positions. The default (0, 1, 2, 3) therefore misfits stock CellVoyager/Yokogawa names: channel 0 never exists, so plane 0 is all zeros, and C04 is dropped. Pass (1, 2, 3, 4) for those layouts.

• channel_names – names recorded in channel_order.json and on each StackFile. A length mismatch with channels trips a bare assert, so it goes unchecked under python -O.

• model_name – resolved by spacr.utils._resolve_cellpose_pretrained(), not passed as Cellpose’s model_type. On Cellpose 4 every legacy name ("cyto", "nuclei", …) collapses to cpsam, so only a fine-tuned checkpoint path actually changes the weights, and a path with no file behind it raises instead of falling back.

• channels_for_cellpose – despite the name this never reaches Cellpose’s channels= argument; it selects C-axis indices from the assembled stack, taken modulo the channel count (7 with C=4 becomes 3) and de-duplicated in order. The default (0, 0) collapses to one plane, and an empty sequence falls back to plane 0. Calibrated runs require explicit in-range positions instead.

• diameter – forwarded to model.eval; None leaves Cellpose to size objects itself. Unlike model_name it is still honoured on Cellpose 4, which rescales the image by 30 / diameter.

• batch_fields – fields loaded and segmented per batch, the memory-versus-speed dial. 0 raises ValueError; a negative value segments nothing at all, yet channel_order.json is still rewritten to claim a mask channel that no stack received. Calibrated batches are additionally bounded to 64 * 1024**2 packed input or padded model-input values; lower this field count when a batch refuses.

• metadata_type – "cellvoyager", "yokogawa", "custom" or "auto". Only "custom" is special-cased; every other unrecognised string quietly behaves as "auto" instead of raising.

• custom_regex – required when metadata_type="custom", else ValueError. It must supply the named groups (plateID, wellID, fieldID, chanID, timeID, sliceID); any group it omits silently defaults, so a regex without chanID makes every image channel 1 and therefore its own single-plane field.

• keep_npz – the per-batch NPZ is compressed into merged/_scratch either way — this only decides whether it and the scratch folder survive, so False does not save the write.

• cellprob_threshold – forwarded to model.eval through float().

• flow_threshold – forwarded to model.eval through float().

• min_size – forwarded to model.eval through int(); None raises TypeError rather than meaning “no minimum”.

• resample – coerced with bool(), so any non-empty string — "false" included — is True, while None is False.

• postprocess_settings – None hands the raw selected planes to Cellpose and skips post-processing entirely. Any dict, {} included, switches on both spacr.io._normalize_img_batch() and spacr.object.merge_split_filter_masks(). The four *_channel role keys are rewritten on a copy (the caller’s dict is left alone): all cleared to None, then object_type set to 0, plus nucleus_channel=1 when object_type is "cell" and two or more planes were selected. An explicit unmix_calibration with unmix selects the organized 2D calibrated cell route: signed valid-ROI crops are retained separately, segmentation uses a nonnegative view and labels are restored to the original reference canvas with incomplete boundary objects removed. Unsupported preprocessing, custom/intensity/ perimeter filters and quality exclusion refuse before inference. Optional unmix_per_plate selects complete physical-channel controls from a JSON master map for one exact source plate/time/plane; mixed sources and missing assignments refuse before output creation.

• object_type – picks the weights during model_name resolution, names the role given channel 0 in normalisation, and is passed to the mask post-processor. An unrecognised value does not raise — it adds a dead <value>_channel key and leaves every real role unset.

• illumination_settings – full Mask settings mapping. When it enables illumination_correction, one model is prepared from the raw persisted stacks after pass 1 and its session corrects only the private Cellpose inputs in pass 2; omitted/off preserves the previous byte-level output contract.

Returns:

dict with mapper (FilenameMapper), stacks (list of StackFile), and dst (Path to merged/). Calibrated output also names its separate output_root; its float32 display intensities are aligned and nonquantitative, never Measure inputs. Signed valid-ROI float32 diagnostics live under output_root/signed; its application record binds both artifacts and remains incomplete on error.

A nonempty unmix_matrix_file in postprocess_settings supplies a version-1 JSON matrix instead of estimating one. This requires calibrated organized 2D cell V2 processing, no control wells, and unmix_per_plate=False. Its physical channels must match the packed order exactly; no transpose, channel reordering or diagonal renormalization is inferred. matrix[i][j] is the camera reading in row-channel i per unit of the dye measured in its own column-channel j. The diagonal is exactly one; outputs retain camera-linear units and signed residuals, not calibrated concentrations. For physical order [3, 1], the file has exactly these keys:

{
  "artifact": "spacr_spectral_matrix",
  "version": 1,
  "source_context": {"plate": "plate1", "time": "", "z": ""},
  "channels": [3, 1],
  "orientation": "rows_readings_columns_dyes",
  "matrix": [[1, 0.4], [0.7, 1]],
  "background_percentile": 0,
  "background_operator": "per_field_percentile_subtract_solve_restore",
  "calibration_source_sha256": "<SHA256 of the exact calibration-file bytes>",
  "acquisition_compatibility": "user_asserts_same_channel_gains_exposures_and_linear_response"
}

Replace the example calibration hash with 64 lowercase hexadecimal digits. Background percentile must equal unmix_background_percentile. Plate, time and Z must equal every mapped field’s exact acquisition IDs, including their string or integer types; an absent time or Z uses the empty string. This supplied-file route requires POSIX descriptor-relative access without symbolic links; unsupported platforms refuse before creating outputs. The acquisition-compatibility token is an unverified user assertion: unknown gains, exposures and response remain unknown. The file is captured before outputs and copied byte-for-byte into psf/spectral_matrix_input.json. Signed-cell readers validate that copied bundle and never depend on the original external path. An empty path preserves control-estimation behavior.

With unmix_partition_plates=True and unmix_per_plate=True, one mixed-plate source produces independent calibrated child datasets. Supply unmix_plate_calibrations as a JSON string mapping every exact plate ID to calibration_file, positive YX pixel_size_um, source_context containing exact plate, time and z IDs, and acquisition_compatibility set to user_asserts_same_channel_gains_exposures_and_linear_response. Relative optical paths resolve against the original source folder. Each plate has one time/Z context, complete physical-channel controls in the master map, and its own optical assignment and pixel spacing. The assertion concerns controls and targets within that plate, remains unverified, and does not assert equal acquisition settings between plates. Global optical settings and supplied matrix files cannot be combined with partitioning.

Before creating outputs, partitioning captures optical bytes, admits all source headers and batch bounds, and estimates each plate’s sampled-control matrix sequentially. Unread target values are validated later by the child route before model construction. Canonical control-value fingerprints are compared after assembly; actual NPY file hashes remain the child record’s authority. Limits are 8192 fields, 16 physical channels and 16 MiB retained metadata, with the ordinary per-field, control and batch budgets unchanged. Supported source formats are unchanged; QC supports report or off only. Directory-relative, no-follow and nonblocking descriptor access is required; unsupported hosts refuse before creating outputs.

A fresh calibrated_plate_batch collection directory contains a plate_batch.json manifest and numbered plate_000001 child datasets. Existing names receive a suffix. The collection is never a measurable dataset; each child retains the ordinary version-1 calibrated signed-cell contract and original global field/source identities. Original files and existing merged data remain untouched. Cancellation or failure preserves completed children and rethrows the original error. Segmentation completion is distinct from aggregate completion, which also requires completed or disabled QC. The opt-in return has partitioned=True, plate_results with independent resolved settings and output_root per child, output_roots, and manifest_path; it has no singular output fallback. With partitioning disabled, the existing return and processing are unchanged.

Raises:

ValueError – from FilenameMapper.discover() when src holds no images, when metadata_type="custom" has no custom_regex, or from batch_fields=0.

spacr.pipeline_v2.stream_masks_from_stack(stacks: List[StackFile], model_name: str = 'cyto', channels_for_cellpose: Sequence[int] = (0, 0), diameter: float | None = None, batch_fields: int = 8, mask_channel_name: str = 'mask', keep_npz: bool = False, npz_dir: pathlib.Path | None = None, cellprob_threshold: float = 0.0, flow_threshold: float = 0.4, min_size: int = 15, resample: bool = True, postprocess_settings: Dict[str, Any] | None = None, object_type: str = 'cell', illumination_session: Any | None = None, psf_session: Any | None = None) → List[StackFile]

Batch the field stacks through Cellpose, then append the mask channel(s) to the SAME npy files.

Parameters:

• stacks – list produced by stream_originals_to_stack().

• model_name – Cellpose model to use ("cyto", "nuclei", …).

• channels_for_cellpose – C-axis indices selected out of each assembled stack before it is handed to Cellpose — taken modulo the channel count and de-duplicated in order. It is NOT forwarded as Cellpose’s channels= argument; [0, 0] therefore yields a single plane, not a grayscale pair. Calibrated sessions instead require explicit in-range positions. The first selected channel is the object’s own channel for absolute mean-intensity filtering.

• diameter – expected object diameter in px (None → Cellpose auto).

• batch_fields – how many field stacks to load into memory at once. Larger = faster but more RAM. Calibrated batches refuse above 64 * 1024**2 packed input or padded model-input values.

• mask_channel_name – human name to record for the appended mask channel (default "mask").

• keep_npz – the intermediate batch is compressed to an NPZ under npz_dir on every batch regardless; this flag only decides whether that file and the scratch folder survive the run.

• npz_dir – where to write the (optional) intermediate NPZ files. Defaults to a scratch subfolder under the stack folder.

• illumination_session – optional spacr.illumination.SegmentationIlluminationSession. Its corrector sees private selected-channel copies immediately before normalisation/Cellpose; persisted intensity planes and scratch NPZs remain raw, and completion is recorded only after the combined stack has been atomically replaced.

• psf_session – optional captured PSF session. Unmixes each whole field first when unmixing is on, then processes selected intensities after illumination and before normalization, padding or Cellpose. Stored image channels stay raw; only the appended labels depend on PSF processing. A calibrated session instead supplies a signed valid crop for a nonnegative segmentation view, restores labels to the reference canvas and stores separate nonquantitative float32 display channels in its separate output tree, with signed crops under signed/. These display channels are not Measure inputs; calibrated outputs cannot be resumed and completion requires both durable display and signed files.

Returns:

the same list, with each StackFile.shape / .channels updated to reflect the appended mask channel.

spacr.pipeline_v2.stream_originals_to_stack(src: pathlib.Path, mapper: FilenameMapper, channels: Sequence[int] = (0, 1, 2, 3), channel_names: Sequence[str] | None = None, dst: pathlib.Path | None = None, *, metadata_root: pathlib.Path | None = None, calibrated: bool = False) → List[StackFile]

Write one merged/stack_<field>.npy per field.

Reads originals directly (no rename-into-channel-folders step), stacks the selected channels along the C axis, and writes one npy per field. Also emits a channel_order.json sidecar describing which C-index holds which channel.

Parameters:

• src – plate folder containing the original images.

• mapper – FilenameMapper produced from src.

• channels – which channel numbers (as parsed from filenames) to include, in the order they should occupy the C axis.

• channel_names – human names for those channels (must match channels length). Default: ["ch0", "ch1", …].

• dst – override the output folder; defaults to <src>/merged.

• metadata_root – mapping CSV destination, defaulting to the source folder.

• calibrated – preserve numeric planes and refuse missing, duplicate, non-2D or oversized channels for the separate calibrated route.

Returns:

list of StackFile, one per field written.