Remote and distributed execution

spaCR can submit any module exposed by spacr-run to another workstation, a Slurm cluster, or a cloud/HPC command-line client. Jobs remain on the remote system when the spaCR GUI closes. Their identifiers, status, settings hashes and latest logs are stored locally and appear in Data & batch → Distributed Jobs.

Execution profiles

An execution profile describes how to reach compute, not an analysis. Profiles never store passwords or API keys:

SSH workstation

Uploads the small resolved settings JSON over SSH, starts spacr-run in a durable background process and records its exit code.

Slurm cluster

Uploads settings, submits an sbatch script, polls squeue and then sacct, and cancels with scancel. The SSH host may be blank when the Slurm commands are installed locally.

Cloud / custom command

Runs configured submit/status/cancel argument templates. This supports cloud CLIs and site-specific schedulers without embedding vendor credentials in spaCR. The command must print a safe job identifier and the status command should print a conventional state such as PENDING, RUNNING, SUCCEEDED, FAILED or CANCELLED.

Configure SSH keys, cloud authentication and VPN access outside spaCR. Test the same connection in a terminal before submitting a long run.

Shared data and path mapping

spaCR transfers the settings document, not an image dataset. Images must already be visible to the execution target through shared storage, a mirrored mount or a cloud-aware wrapper command.

When the mount paths differ, set a Local dataset root and Remote dataset root. Every absolute path nested in the settings below the local root is rewritten beneath the remote root. Paths outside it are preserved. For example:

local root:   /mnt/microscopy
remote root:  /cluster/projects/microscopy
local src:    /mnt/microscopy/experiment-7/plate-A
remote src:   /cluster/projects/microscopy/experiment-7/plate-A

Settings can be dragged onto the Distributed Jobs screen, selected with Browse, or handed off directly with Submit remote… in any ordinary module.

Command-line workflow

The GUI and CLI share the same profile and job stores. Create a workstation profile and submit a settings export:

spacr-remote profile add gpu-box \
    --backend ssh \
    --host scientist@gpu-box \
    --workdir /shared/spacr \
    --local-root /mnt/lab \
    --remote-root /shared/lab

spacr-remote submit mask \
    --settings mask-settings.csv \
    --profile gpu-box

spacr-remote list --refresh
spacr-remote status JOB_ID --logs
spacr-remote watch JOB_ID --logs
spacr-remote cancel JOB_ID

A Slurm profile can be local or reached through a login host:

spacr-remote profile add lab-slurm \
    --backend slurm \
    --host scientist@login.cluster \
    --workdir /project/lab \
    --runner /project/lab/env/bin/spacr-run \
    --scheduler-options "--partition=gpu --gres=gpu:1 --time=12:00:00"

Custom/cloud templates

Templates are parsed as argument vectors; spaCR does not execute them with shell=True. Supported placeholders are:

{job_id}

The permanent local spaCR identifier.

{module}

The canonical spacr-run module name.

{settings}

The absolute local resolved-settings JSON. A cloud wrapper is responsible for uploading it if needed.

{external_id}

The identifier printed by the submit command, used by status/cancel/log commands.

{profile}

The profile display name.

If a cloud CLI prints JSON, set a Job-ID regular expression with a named id group. For example "jobId":\s*"(?P<id>[A-Za-z0-9-]+)".

Reliability and limitations

• A temporary polling failure is retained as a visible error and does not falsely mark a remote job failed.

• SSH jobs write an exit-code file atomically. Slurm uses accounting after a job leaves the queue.

• Local job records and profiles use advisory locks plus atomic replacement, so a GUI and spacr-remote process do not partially write JSON.

• Cancellation is a request. A scheduler or remote process may take time to stop, and pipeline-level checkpoint semantics still determine how much work can be resumed.

• Cloud templates intentionally do not provide an embedded shell. Put pipelines, uploads, quoting and vendor-specific JSON handling in a reviewed wrapper executable.

Python API

Persistent remote and distributed execution for spaCR pipelines.

The local GUI and spacr.cli already agree on one headless contract:

spacr-run MODULE --settings SETTINGS.json

This module transports that contract to a workstation over SSH, submits it to Slurm, or hands it to a user-configured cloud/HPC command. Submitted jobs are recorded locally and can be polled or cancelled after spaCR itself has closed.

No command uses shell=True. SSH necessarily invokes a remote login shell; all user-controlled values interpolated into its small fixed scripts are quoted with shlex.quote(), and host names are validated separately. Custom command profiles are parsed into argument vectors and substitute each placeholder inside one argument, so shell operators have no special meaning.

class spacr.remote_execution.CommandResult(returncode: int, stdout: str = '', stderr: str = '')

Bases: object

Result returned by the injectable command runner.

returncode: int

stderr: str = ''

stdout: str = ''

class spacr.remote_execution.ExecutionProfile(name: str, backend: str, host: str = '', workdir: str = '', local_root: str = '', remote_root: str = '', runner: str = 'spacr-run', scheduler_options: str = '', submit_command: str = '', status_command: str = '', cancel_command: str = '', log_command: str = '', job_id_pattern: str = '', poll_seconds: int = 10)

Bases: object

Connection and scheduler settings for one execution target.

local_root and remote_root describe the same shared or mirrored dataset. Every absolute string nested in the settings is rewritten when it lies below local_root. Image datasets are deliberately not copied: accidental recursive transfer of a multi-terabyte plate is worse than a clear pre-flight error.

For command profiles, command strings are tokenized with shlex.split() and support {job_id}, {module}, {settings} and {external_id} placeholders. They are argument templates, not shell scripts.

backend: str

cancel_command: str = ''

classmethod from_dict(value: Mapping[str, Any]) → ExecutionProfile

Construct and validate a profile from JSON-compatible data.

host: str = ''

job_id_pattern: str = ''

local_root: str = ''

log_command: str = ''

name: str

poll_seconds: int = 10

remote_root: str = ''

runner: str = 'spacr-run'

scheduler_options: str = ''

status_command: str = ''

submit_command: str = ''

to_dict() → Dict[str, Any]

Return a JSON-safe representation.

validate() → ExecutionProfile

Validate the profile and return self for fluent callers.

workdir: str = ''

class spacr.remote_execution.JobStore(path: PathLike | None = None)

Bases: object

Atomic persistent store for remote job metadata.

get(job_id: str) → RemoteJob

Return a job by full ID or unambiguous prefix.

list() → List[RemoteJob]

Return newest jobs first.

save(job: RemoteJob) → None

Insert or replace one job atomically.

class spacr.remote_execution.ProfileStore(path: PathLike | None = None)

Bases: object

Atomic persistent store for execution profiles.

delete(name: str) → bool

Delete one profile; return whether it existed.

get(name: str) → ExecutionProfile

Return a named profile or raise a user-facing error.

list() → List[ExecutionProfile]

Return profiles sorted by case-insensitive name.

save(profile: ExecutionProfile) → None

Insert or replace one profile atomically.

exception spacr.remote_execution.RemoteExecutionError

Bases: RuntimeError

A profile, submission, polling, or cancellation error.

These errors are safe to show directly in the GUI: passwords and environment variables are never included in command rendering.

class spacr.remote_execution.RemoteJob(job_id: str, module: str, profile_name: str, backend: str, status: str = 'submitting', external_id: str = '', created_utc: str = <factory>, updated_utc: str = <factory>, settings_path: str = '', settings_sha256: str = '', remote_settings_path: str = '', remote_job_dir: str = '', log_reference: str = '', log_tail: str = '', exit_code: int | None = None, error: str = '', profile: ~typing.Dict[str, ~typing.Any] = <factory>)

Bases: object

Persistent local record of one submitted job.

backend: str

created_utc: str

error: str = ''

exit_code: int | None = None

external_id: str = ''

classmethod from_dict(value: Mapping[str, Any]) → RemoteJob

Construct a job from a stored mapping, tolerating future fields.

job_id: str

log_reference: str = ''

log_tail: str = ''

module: str

profile: Dict[str, Any]

profile_name: str

remote_job_dir: str = ''

remote_settings_path: str = ''

settings_path: str = ''

settings_sha256: str = ''

status: str = 'submitting'

to_dict() → Dict[str, Any]

Return a JSON-safe representation.

updated_utc: str

class spacr.remote_execution.RemoteJobManager(profile_store: ~spacr.remote_execution.ProfileStore | None = None, job_store: ~spacr.remote_execution.JobStore | None = None, runner: ~typing.Callable[[...], ~spacr.remote_execution.CommandResult] = <function _run_command>)

Bases: object

Submit, monitor, cancel and inspect persistent remote jobs.

All methods are synchronous and may perform network I/O. GUI callers must invoke them on a worker thread; the shipped Distributed Jobs screen does.

cancel(job_id: str) → RemoteJob

Request cancellation and persist the result.

logs(job_id: str, lines: int = 200) → str

Retrieve and persist the tail of one remote job’s log.

refresh(job_id: str, *, include_logs: bool = True) → RemoteJob

Poll one non-terminal job and optionally retain its latest log tail.

refresh_all(*, include_logs: bool = False) → List[RemoteJob]

Poll every active job and return the complete newest-first list.

submit(module: str, settings: Mapping[str, Any], profile_name: str) → RemoteJob

Submit resolved settings through a named execution profile.

spacr.remote_execution.map_settings_paths(value: Any, local_root: str, remote_root: str) → Any

Recursively map absolute paths below one local root to a remote root.

Non-path strings and paths outside the configured root are unchanged. Mapping keys are intentionally preserved: setting names are not paths.

spacr.remote_execution.state_directory() → Path

Return the persistent directory for profiles, jobs, settings and logs.

SPACR_REMOTE_STATE_DIR is intentionally supported for tests, portable deployments, and managed lab installations. Otherwise XDG state storage is used on Linux and a conventional per-user directory elsewhere.