Remote and distributed execution

spaCR can submit any module exposed by spacr-run to another workstation, a Slurm cluster, or a cloud/HPC command-line client. Jobs remain on the remote system when the spaCR GUI closes. Their identifiers, status, settings hashes and latest logs are stored locally and appear in Help → Distributed Jobs.

Execution profiles

An execution profile describes how to reach compute, not an analysis. Profiles never store passwords or API keys:

SSH workstation

Uploads the small resolved settings JSON over SSH, starts spacr-run in a durable background process and records its exit code.

Slurm cluster

Uploads settings, submits an sbatch script, polls squeue and then sacct, and cancels with scancel. The SSH host may be blank when the Slurm commands are installed locally.

Cloud / custom command

Runs configured submit/status/cancel argument templates. This supports cloud CLIs and site-specific schedulers without embedding vendor credentials in spaCR. The command must print a safe job identifier and the status command should print a conventional state such as PENDING, RUNNING, SUCCEEDED, FAILED or CANCELLED.

Configure SSH keys, cloud authentication and VPN access outside spaCR. Test the same connection in a terminal before submitting a long run.

Shared data and path mapping

spaCR transfers the settings document, not an image dataset. Images must already be visible to the execution target through shared storage, a mirrored mount or a cloud-aware wrapper command.

When the mount paths differ, set a Local dataset root and Remote dataset root. Every absolute path nested in the settings below the local root is rewritten beneath the remote root. Paths outside it are preserved. For example:

local root:   /mnt/microscopy
remote root:  /cluster/projects/microscopy
local src:    /mnt/microscopy/experiment-7/plate-A
remote src:   /cluster/projects/microscopy/experiment-7/plate-A

Settings can be dragged onto the Distributed Jobs screen, selected with Browse, or handed off directly with Submit remote… in any ordinary module.

Command-line workflow

The GUI and CLI share the same profile and job stores. Create a workstation profile and submit a settings export:

spacr-remote profile add gpu-box \
    --backend ssh \
    --host scientist@gpu-box \
    --workdir /shared/spacr \
    --local-root /mnt/lab \
    --remote-root /shared/lab

spacr-remote submit mask \
    --settings mask-settings.csv \
    --profile gpu-box

spacr-remote list --refresh
spacr-remote status JOB_ID --logs
spacr-remote watch JOB_ID --logs
spacr-remote cancel JOB_ID

A Slurm profile can be local or reached through a login host:

spacr-remote profile add lab-slurm \
    --backend slurm \
    --host scientist@login.cluster \
    --workdir /project/lab \
    --runner /project/lab/env/bin/spacr-run \
    --scheduler-options "--partition=gpu --gres=gpu:1 --time=12:00:00"

Custom/cloud templates

Templates are parsed as argument vectors; spaCR does not execute them with shell=True. Supported placeholders are:

{job_id}

The permanent local spaCR identifier.

{module}

The canonical spacr-run module name.

{settings}

The absolute local resolved-settings JSON. A cloud wrapper is responsible for uploading it if needed.

{external_id}

The identifier printed by the submit command, used by status/cancel/log commands.

{profile}

The profile display name.

If a cloud CLI prints JSON, set a Job-ID regular expression with a named id group. For example "jobId":\s*"(?P<id>[A-Za-z0-9-]+)".

Reliability and limitations

• A temporary polling failure is retained as a visible error and does not falsely mark a remote job failed.

• SSH jobs write an exit-code file atomically. Slurm uses accounting after a job leaves the queue.

• Local job records and profiles use advisory locks plus atomic replacement, so a GUI and spacr-remote process do not partially write JSON.

• Cancellation is a request. A scheduler or remote process may take time to stop, and pipeline-level checkpoint semantics still determine how much work can be resumed.

• Cloud templates intentionally do not provide an embedded shell. Put pipelines, uploads, quoting and vendor-specific JSON handling in a reviewed wrapper executable.

Python API

Persistent remote and distributed execution for spaCR pipelines.

The local GUI and spacr.cli already agree on one headless contract:

spacr-run MODULE --settings SETTINGS.json

This module transports that contract to a workstation over SSH, submits it to Slurm, or hands it to a user-configured cloud/HPC command. Submitted jobs are recorded locally and can be polled or cancelled after spaCR itself has closed.

No command uses shell=True. SSH necessarily invokes a remote login shell; all user-controlled values interpolated into its small fixed scripts are quoted with shlex.quote(), and host names are validated separately. Custom command profiles are parsed into argument vectors and substitute each placeholder inside one argument, so shell operators have no special meaning.

class spacr.remote_execution.CommandResult(returncode: int, stdout: str = '', stderr: str = '')

Bases: object

Result returned by the injectable command runner.

Parameters:

• returncode – process exit status, where zero denotes success.

• stdout – captured standard-output text.

• stderr – captured standard-error text.

returncode: int

stderr: str = ''

stdout: str = ''

class spacr.remote_execution.ExecutionProfile(name: str, backend: str, host: str = '', workdir: str = '', local_root: str = '', remote_root: str = '', runner: str = 'spacr-run', scheduler_options: str = '', submit_command: str = '', status_command: str = '', cancel_command: str = '', log_command: str = '', job_id_pattern: str = '', poll_seconds: int = 10)

Bases: object

Connection and scheduler settings for one execution target.

local_root and remote_root describe the same shared or mirrored dataset. Every absolute string nested in the settings is rewritten when it lies below local_root. Image datasets are deliberately not copied: accidental recursive transfer of a multi-terabyte plate is worse than a clear pre-flight error.

For command profiles, command strings are tokenized with shlex.split() and support {job_id}, {module}, {settings} and {external_id} placeholders. They are argument templates, not shell scripts.

Parameters:

• name – human-readable profile name used to select this target.

• backend – execution mechanism, one of BACKENDS.

• host – OpenSSH target for SSH and remote Slurm; blank permits a Slurm scheduler running on the local host.

• workdir – absolute execution directory where spacr-jobs artifacts are created.

• local_root – local dataset prefix paired with remote_root for nested absolute-path rewriting.

• remote_root – equivalent dataset prefix visible to the target host.

• runner – headless executable invoked as runner module --settings path by SSH and Slurm backends.

• scheduler_options – additional shell-free arguments passed to sbatch.

• submit_command – custom-backend argument template containing {settings} and printing a job identifier.

• status_command – custom status template containing {external_id} and returning a state spaCR can normalize.

• cancel_command – custom cancellation template containing {external_id}.

• log_command – optional custom template for fetching job output.

• job_id_pattern – optional regular expression extracting a named id, first capture, or full match from submission output.

• poll_seconds – recommended client polling interval, from 2 through 3600 seconds.

backend: str

cancel_command: str = ''

classmethod from_dict(value: Mapping[str, Any]) → ExecutionProfile

Construct and validate a profile from JSON-compatible data.

Parameters:

value – stored profile fields keyed by their dataclass names.

host: str = ''

job_id_pattern: str = ''

local_root: str = ''

log_command: str = ''

name: str

poll_seconds: int = 10

remote_root: str = ''

runner: str = 'spacr-run'

scheduler_options: str = ''

status_command: str = ''

submit_command: str = ''

to_dict() → Dict[str, Any]

Return a JSON-safe representation.

validate() → ExecutionProfile

Validate the profile and return self for fluent callers.

workdir: str = ''

class spacr.remote_execution.JobStore(path: PathLike | None = None)

Bases: object

Atomic persistent store for remote job metadata.

Parameters:

path – optional jobs JSON path; by default state_directory()/jobs.json. The advisory lock uses a sibling path with .lock appended.

get(job_id: str) → RemoteJob

Return a job by full ID or unambiguous prefix.

Parameters:

job_id – complete job identifier or an unambiguous prefix.

list() → List[RemoteJob]

Return newest jobs first.

save(job: RemoteJob) → None

Insert or replace one job atomically.

Parameters:

job – remote-job record to persist.

class spacr.remote_execution.ProfileStore(path: PathLike | None = None)

Bases: object

Atomic persistent store for execution profiles.

Parameters:

path – optional profiles JSON path; by default state_directory()/profiles.json. The advisory lock uses a sibling path with .lock appended.

delete(name: str) → bool

Delete one profile; return whether it existed.

Parameters:

name – profile name to delete case-insensitively.

get(name: str) → ExecutionProfile

Return a named profile or raise a user-facing error.

Parameters:

name – profile name to match case-insensitively.

list() → List[ExecutionProfile]

Return profiles sorted by case-insensitive name.

save(profile: ExecutionProfile) → None

Insert or replace one profile atomically.

Parameters:

profile – validated execution profile to persist.

exception spacr.remote_execution.RemoteExecutionError

Bases: RuntimeError

A profile, submission, polling, or cancellation error.

These errors are safe to show directly in the GUI: passwords and environment variables are never included in command rendering.

class spacr.remote_execution.RemoteJob(job_id: str, module: str, profile_name: str, backend: str, status: str = 'submitting', external_id: str = '', created_utc: str = <factory>, updated_utc: str = <factory>, settings_path: str = '', settings_sha256: str = '', remote_settings_path: str = '', remote_job_dir: str = '', log_reference: str = '', log_tail: str = '', exit_code: int | None = None, error: str = '', profile: ~typing.Dict[str, ~typing.Any] = <factory>)

Bases: object

Persistent local record of one submitted job.

Parameters:

• job_id – locally generated stable identifier for the job.

• module – resolved spaCR command-line module key to execute.

• profile_name – name of the execution profile used for submission.

• backend – backend recorded at submission time.

• status – compact lifecycle state, initially "submitting".

• external_id – validated process, scheduler, or provider identifier.

• created_utc – ISO-8601 construction timestamp.

• updated_utc – timestamp replaced whenever JobStore persists the record.

• settings_path – retained local path of the mapped settings JSON.

• settings_sha256 – SHA-256 digest of the exact serialized settings.

• remote_settings_path – uploaded settings path for SSH and Slurm.

• remote_job_dir – target-side directory containing job artifacts.

• log_reference – backend log location or custom-backend marker.

• log_tail – most recently fetched output or availability message.

• exit_code – normalized process exit status, or None until known.

• error – latest submission, polling, or cancellation failure; cleared after a successful refresh or cancellation.

• profile – submission-time profile snapshot, retained so the job remains operable after profile edits or deletion.

backend: str

created_utc: str

error: str = ''

exit_code: int | None = None

external_id: str = ''

classmethod from_dict(value: Mapping[str, Any]) → RemoteJob

Construct a job from a stored mapping, tolerating future fields.

Parameters:

value – stored job fields keyed by their dataclass names.

job_id: str

log_reference: str = ''

log_tail: str = ''

module: str

profile: Dict[str, Any]

profile_name: str

remote_job_dir: str = ''

remote_settings_path: str = ''

settings_path: str = ''

settings_sha256: str = ''

status: str = 'submitting'

to_dict() → Dict[str, Any]

Return a JSON-safe representation.

updated_utc: str

class spacr.remote_execution.RemoteJobManager(profile_store: ~spacr.remote_execution.ProfileStore | None = None, job_store: ~spacr.remote_execution.JobStore | None = None, runner: ~typing.Callable[[...], ~spacr.remote_execution.CommandResult] = <function _run_command>)

Bases: object

Submit, monitor, cancel and inspect persistent remote jobs.

All methods are synchronous and may perform network I/O. GUI callers must invoke them on a worker thread; the shipped Distributed Jobs screen does.

Parameters:

• profile_store – injectable persistent profile store; omitted creates the default ProfileStore.

• job_store – injectable persistent job store; omitted creates the default JobStore.

• runner – shell-free callable accepting an argument vector, timeout, and optional input and returning CommandResult.

cancel(job_id: str) → RemoteJob

Request cancellation and persist the result.

Parameters:

job_id – complete local job identifier or unambiguous prefix.

logs(job_id: str, lines: int = 200) → str

Retrieve and persist the tail of one remote job’s log.

Parameters:

job_id – complete local job identifier or unambiguous prefix.

refresh(job_id: str, *, include_logs: bool = True) → RemoteJob

Poll one non-terminal job and optionally retain its latest log tail.

Parameters:

job_id – complete local job identifier or unambiguous prefix.

refresh_all(*, include_logs: bool = False) → List[RemoteJob]

Poll every active job and return the complete newest-first list.

submit(module: str, settings: Mapping[str, Any], profile_name: str) → RemoteJob

Submit resolved settings through a named execution profile.

Parameters:

• module – headless spaCR module name accepted by spacr-run.

• settings – resolved module settings to serialize for the job.

• profile_name – name of the execution profile to use.

spacr.remote_execution.map_settings_paths(value: Any, local_root: str, remote_root: str) → Any

Recursively map absolute paths below one local root to a remote root.

Non-path strings and paths outside the configured root are unchanged. Mapping keys are intentionally preserved: setting names are not paths.

Parameters:

• value – nested settings value whose absolute path strings are mapped.

• local_root – local dataset root that mapped paths must lie below.

• remote_root – remote dataset root that replaces local_root.

spacr.remote_execution.state_directory() → Path

Return the persistent directory for profiles, jobs, settings and logs.

SPACR_REMOTE_STATE_DIR is intentionally supported for tests, portable deployments, and managed lab installations. Otherwise XDG state storage is used on Linux and a conventional per-user directory elsewhere.