spacr.artifacts

The artifact registry: what produced every file in a spaCR project.

A spaCR project is a folder of derived data. merged/ came from raw images, measurements.db came from merged/, the model came from the crops the database indexes, and the hit list came from the model. Nothing on disk records any of that, so “is this result still current?” has never had an answer — and re-running Mask with a different diameter leaves every downstream number quietly wrong.

This module records it. Every output registers with:

• the producing module ("mask", "measure", … — the keys spacr.ports and spacr.validate use);

• a settings hash over the settings that could change the numbers (spacr.resume.COSMETIC_SETTINGS decides which cannot);

• the spaCR version that produced it;

• the input artifact ids it was derived from, making the project a DAG;

• a timestamp, a path, and a content fingerprint.

Storage is SQLite — artifacts.db in the project root, one row per artifact plus one edge row per input — because two Measure workers, a GUI and a batch runner can all be touching a project at once, and a pickle cannot survive that. Writes go through spacr.database_concurrency.transaction() with an explicit lock budget, which is also why nothing here does its own retry arithmetic.

Public API

Registry

The registry for one project: register(), get(), by_kind(), by_project(), latest(), downstream_of(), upstream_of(), is_stale(), forget().

open_registry, and module-level register / by_kind / by_project / latest / downstream_of / is_stale

The same, resolved from a project path.

register_run_outputs

The one call a finished run makes; walks spacr.ports.declared_outputs() and registers what is there.

Artifact, Staleness, Fingerprint

The records that come back.

settings_hash, material_settings, content_fingerprint

The provenance primitives, usable on their own.

Attributes

ARTIFACTS_DB_ENV
ARTIFACTS_DB_NAME
FULL_HASH_LIMIT_BYTES
SCHEMA_VERSION
STATUS_COMPLETE
STATUS_FAILED
STATUS_PARTIAL

Classes

Artifact	One registered output and everything known about where it came from.
Fingerprint	A content fingerprint plus how it was arrived at.
Registry	The artifact registry for one project.
Staleness	Whether an artifact still matches what it was made from.

Functions

by_kind(→ List[Artifact])	Every artifact of kind; see Registry.by_kind().
by_project(→ List[Artifact])	Every artifact in a project; see Registry.by_project().
content_fingerprint(→ Fingerprint)	Fingerprint whatever is at path: a file, a folder, or nothing.
downstream_of(→ List[Artifact])	What an artifact invalidates; see Registry.downstream_of().
is_stale(→ Staleness)	Whether an artifact is out of date; see Registry.is_stale().
latest(→ Optional[Artifact])	The newest artifact of kind; see Registry.latest().
material_settings(→ Dict[str, Any])	Return only the settings that can change the numbers.
open_registry(→ Registry)	Return the Registry for one project.
register(→ Artifact)	Register one artifact; see Registry.register().
register_run_outputs(→ Tuple[Artifact, Ellipsis])	Register everything a finished run declared it would write.
registry_path(→ str)	Return the registry file for project.
settings_hash(→ str)	Return a digest over the material settings of a run.

Module Contents

class spacr.artifacts.Artifact

One registered output and everything known about where it came from.

Parameters:

• artifact_id – 16 hexadecimal characters, derived from the identity below. Registering byte-identical content from the same module, project, role and settings yields the same id, so a repeated registration updates one row rather than growing the table.

• project – absolute project root.

• kind – a spacr.ports kind, e.g. "measurements-db".

• role – the producing module’s port role, e.g. "merged".

• path – absolute path of the file or folder.

• module – producing module key, e.g. "mask".

• run_id – the run this came out of, when the caller knows it.

• settings_hash – digest of the material settings; see settings_hash().

• spacr_version – the version that produced it.

• created_ns – registration time, time.time_ns().

• created_utc – the same instant, ISO-8601.

• fingerprint – content digest; see content_fingerprint().

• fingerprint_method – how that digest was computed.

• size_bytes – bytes on disk at registration.

• n_files – files covered.

• status – STATUS_COMPLETE, STATUS_PARTIAL or STATUS_FAILED.

• settings – the material settings, kept so a later run can be diffed against this one rather than only compared by hash.

• extra – anything else the producer wanted recorded.

• inputs – artifact ids this was derived from.

• schema_version – the registry layout that wrote the row.

__str__() → str

One line: id, kind, module and path.

to_dict() → Dict[str, Any]

Return a JSON-serializable copy of the record.

artifact_id: str

created_ns: int

created_utc: str

property exists: bool

True when the artifact is still on disk where it was registered.

extra: Dict[str, Any]

fingerprint: str

fingerprint_method: str

inputs: Tuple[str, Ellipsis] = ()

kind: str

module: str

n_files: int

path: str

project: str

role: str

run_id: str

schema_version: int = 1

settings: Dict[str, Any]

settings_hash: str

size_bytes: int

spacr_version: str

status: str

class spacr.artifacts.Fingerprint

A content fingerprint plus how it was arrived at.

Parameters:

• digest – hexadecimal SHA-256, or "" when nothing was there.

• method – "sha256" (a file read end to end), "sampled" (a large file: size plus its first and last megabyte), "tree" (a folder: every file’s relative path, size and mtime) or "missing".

• size_bytes – total bytes covered.

• n_files – number of files covered; 1 for a single file.

__bool__() → bool

True when something was actually fingerprinted.

digest: str

method: str

n_files: int = 0

size_bytes: int = 0

class spacr.artifacts.Registry(path: str | os.PathLike | None = None, *, project: str | os.PathLike | None = None, timeout: float = 30.0, create: bool = True)

The artifact registry for one project.

One SQLite file, opened per operation and closed again: Measure workers, the GUI and a batch runner all register into the same project, and a connection held open across a whole run is a lock held across a whole run. Writes go through spacr.database_concurrency.transaction() with an explicit lock budget, so a second registration arriving mid-write waits rather than failing.

Parameters:

• path – the registry file. Defaults to registry_path() for project.

• project – the project root recorded on artifacts that do not name one of their own.

• timeout – seconds a write may wait on the lock, in total.

• create – create the file and its tables when missing. Pass False for a read-only consumer that must not conjure an empty registry.

Raises:

• FileNotFoundError – when create is False and there is no registry.

• ValueError – when neither path nor project is given.

all(*, limit: int | None = None) → List[Artifact]

Return every artifact in this registry, newest first.

Parameters:

limit – cap the number of rows returned.

by_kind(kind: str, *, project: str | None = None, module: str | None = None, limit: int | None = None) → List[Artifact]

Return every artifact of kind, newest first.

Parameters:

• kind – a spacr.ports kind, e.g. "merged-arrays".

• project – restrict to one project root.

• module – restrict to one producing module.

• limit – cap the number of rows returned.

by_project(project: str | None = None, *, kind: str | None = None, module: str | None = None, limit: int | None = None) → List[Artifact]

Return every artifact belonging to one project, newest first.

Parameters:

• project – the project root; defaults to this registry’s own. Pass "" explicitly for “every project in this file”, which is what a shared registry (see ARTIFACTS_DB_ENV) holds.

• kind – restrict to one kind.

• module – restrict to one producing module.

• limit – cap the number of rows returned.

downstream_of(artifact: str | Artifact, *, transitive: bool = True) → List[Artifact]

Return the artifacts derived from artifact.

Transitive by default: the question a user asks — “what does this invalidate?” — is about everything downstream, not only the immediate children.

Parameters:

• artifact – id or Artifact.

• transitive – follow the edges all the way down.

Returns:

registered descendants, newest first.

forget(artifact: str | Artifact) → int

Delete one artifact row and the edges pointing out of it.

Edges pointing at it are left alone on purpose: a downstream artifact that names a vanished input must keep saying so, which is what makes it report as stale rather than as current.

Parameters:

artifact – id or Artifact.

Returns:

number of artifact rows deleted — 0 or 1.

get(artifact: str | Artifact) → Artifact | None

Return one artifact by id, or None when it is not registered.

Parameters:

artifact – id or Artifact.

is_stale(artifact: str | Artifact, *, settings: Mapping[str, Any] | None = None) → Staleness

Answer whether an upstream artifact or setting changed since this.

An artifact is stale when any of these hold:

• one of its recorded inputs is no longer in the registry;

• an input was registered again after this artifact was;

• a newer artifact of the same kind now sits at an input’s path — the shape “Mask was re-run with a different diameter” takes;

• an input is itself stale, transitively;

• settings is supplied and its material hash differs from the one recorded here.

Parameters:

• artifact – id or Artifact.

• settings – the settings a caller is about to use, compared against the ones that produced the artifact.

Returns:

a Staleness; bool(result) is the answer and Staleness.reasons is what to show a user.

latest(kind: str, *, project: str | None = None, module: str | None = None, role: str | None = None, path: str | None = None) → Artifact | None

Return the most recent artifact matching the filters, or None.

The call auto-chaining makes: “what is the current measurements-db for this project?”.

Parameters:

• kind – a spacr.ports kind.

• project – restrict to one project root.

• module – restrict to one producing module.

• role – restrict to one port role.

• path – restrict to one exact path.

register(*, module: str, kind: str, path: str | os.PathLike, role: str = '', project: str | os.PathLike | None = None, settings: Mapping[str, Any] | None = None, settings_digest: str = '', inputs: Sequence[str | Artifact] = (), run_id: str = '', status: str = STATUS_COMPLETE, extra: Mapping[str, Any] | None = None, fingerprint: Fingerprint | None = None) → Artifact

Record one output and what it was made from.

The id is derived from the identity — project, module, kind, role, path, settings hash, content fingerprint — so registering the same content twice updates the existing row (refreshing its timestamp, run id and inputs) instead of adding a duplicate. Registering different content, or the same content under different settings, creates a new row, which is exactly what makes the older downstream artifacts stale.

Parameters:

• module – producing module key, e.g. "mask".

• kind – a spacr.ports kind.

• path – the file or folder produced.

• role – the module’s port role for this output.

• project – project root; defaults to the registry’s own.

• settings – the run’s settings. Only the material ones are hashed and stored.

• settings_digest – use this digest instead of hashing settings — for a caller that already computed one.

• inputs – artifact ids (or Artifact objects) this was derived from.

• run_id – the run this came out of.

• status – STATUS_COMPLETE, STATUS_PARTIAL or STATUS_FAILED.

• extra – any additional JSON-safe provenance.

• fingerprint – a precomputed Fingerprint; omit to compute one from path.

Returns:

the stored Artifact.

Raises:

ValueError – when module, kind or path is empty.

upstream_of(artifact: str | Artifact, *, transitive: bool = False) → List[Artifact]

Return the artifacts artifact was derived from.

Parameters:

• artifact – id or Artifact.

• transitive – follow inputs of inputs, to the roots of the DAG.

Returns:

registered ancestors, newest first. Input ids that are no longer registered are simply absent — is_stale() is what reports them.

path = b'.'

project = b'.'

timeout

class spacr.artifacts.Staleness

Whether an artifact still matches what it was made from.

Stale means an upstream artifact or a material setting changed after this was produced. A file that has simply been deleted is reported by missing instead — that is an availability problem, not a provenance one, and conflating the two hides both.

Parameters:

• artifact_id – the artifact asked about.

• stale – the answer. bool(staleness) is the same value.

• reasons – human-readable sentences, one per finding.

• causes – machine codes for the same findings, e.g. "upstream-newer".

• missing – the artifact’s own path is no longer on disk.

__bool__() → bool

True when the artifact is stale.

__str__() → str

A one-line verdict with its reasons.

artifact_id: str

causes: Tuple[str, Ellipsis] = ()

missing: bool = False

reasons: Tuple[str, Ellipsis] = ()

stale: bool

spacr.artifacts.by_kind(kind: str, *, project: str | os.PathLike | None = None, registry: Registry | None = None, **kwargs: Any) → List[Artifact]

Every artifact of kind; see Registry.by_kind().

Parameters:

• kind – a spacr.ports kind.

• project – the project root.

• registry – an open registry to use instead of opening one.

• kwargs – passed through to Registry.by_kind().

spacr.artifacts.by_project(project: str | os.PathLike | None = None, *, registry: Registry | None = None, **kwargs: Any) → List[Artifact]

Every artifact in a project; see Registry.by_project().

Parameters:

• project – the project root.

• registry – an open registry to use instead of opening one.

• kwargs – passed through to Registry.by_project().

spacr.artifacts.content_fingerprint(path: str | os.PathLike, *, full_hash_limit: int = FULL_HASH_LIMIT_BYTES) → Fingerprint

Fingerprint whatever is at path: a file, a folder, or nothing.

A regular file is hashed end to end while it is small enough to be worth it, and above full_hash_limit from its size plus its first and last megabyte — enough to notice a rewritten database without spending minutes of I/O at the end of every run. A folder is fingerprinted from its file inventory: every file’s relative path, size and modification time, in sorted order.

Parameters:

• path – file or folder.

• full_hash_limit – byte size above which a file is sampled instead of read end to end.

Returns:

a Fingerprint; method="missing" when nothing is there.

spacr.artifacts.downstream_of(artifact: str | Artifact, *, project: str | os.PathLike | None = None, registry: Registry | None = None, **kwargs: Any) → List[Artifact]

What an artifact invalidates; see Registry.downstream_of().

Parameters:

• artifact – id or Artifact.

• project – the project root.

• registry – an open registry to use instead of opening one.

• kwargs – passed through to Registry.downstream_of().

spacr.artifacts.is_stale(artifact: str | Artifact, *, project: str | os.PathLike | None = None, registry: Registry | None = None, **kwargs: Any) → Staleness

Whether an artifact is out of date; see Registry.is_stale().

Parameters:

• artifact – id or Artifact.

• project – the project root.

• registry – an open registry to use instead of opening one.

• kwargs – passed through to Registry.is_stale().

spacr.artifacts.latest(kind: str, *, project: str | os.PathLike | None = None, registry: Registry | None = None, **kwargs: Any) → Artifact | None

The newest artifact of kind; see Registry.latest().

Parameters:

• kind – a spacr.ports kind.

• project – the project root.

• registry – an open registry to use instead of opening one.

• kwargs – passed through to Registry.latest().

spacr.artifacts.material_settings(settings: Mapping[str, Any] | None) → Dict[str, Any]

Return only the settings that can change the numbers.

Verbosity, worker counts, plot cosmetics and the environment snapshot are dropped, using the same deny-list (spacr.resume.COSMETIC_SETTINGS, and spacr.resume.ENV_SETTINGS) that decides whether a resume is allowed. Any key nobody has classified counts as material, so a new knob is conservatively assumed to matter.

Parameters:

settings – a settings dict, or None.

Returns:

a new dict with the inconsequential keys removed.

spacr.artifacts.open_registry(project: str | os.PathLike | None = None, *, path: str | os.PathLike | None = None, create: bool = True) → Registry

Return the Registry for one project.

Parameters:

• project – the project root.

• path – an explicit registry file, overriding project’s default.

• create – create the file and tables when missing.

spacr.artifacts.register(*, registry: Registry | None = None, project: str | os.PathLike | None = None, **kwargs: Any) → Artifact

Register one artifact; see Registry.register().

Parameters:

• registry – an open registry to use instead of opening one.

• project – the project root, used to find the registry and recorded on the artifact.

• kwargs – passed through to Registry.register().

spacr.artifacts.register_run_outputs(module: str, settings: Mapping[str, Any] | None = None, *, roots: Iterable[Any] | None = None, run_id: str = '', status: str = STATUS_COMPLETE, inputs: Sequence[str | Artifact] | None = None, registry: Registry | None = None, strict: bool = True) → Tuple[Artifact, Ellipsis]

Register everything a finished run declared it would write.

The one call a pipeline entry point makes on completion. For each project root it walks spacr.ports.declared_outputs(), registers every produced port that is actually on disk, and links each one to the artifacts currently sitting at the module’s input ports — which is what turns a folder of files into a provenance DAG.

Parameters:

• module – producing module key, e.g. "mask".

• settings – the settings the run used; hashed into every artifact.

• roots – project roots to record. Defaults to the one spacr.ports.project_root() derives from settings. A multi-plate run passes its whole src list.

• run_id – the run this came out of.

• status – STATUS_COMPLETE, STATUS_PARTIAL or STATUS_FAILED — a run that lost fields should say so.

• inputs – explicit input artifact ids, overriding the lookup.

• registry – an open registry to use for every root, instead of one per project.

• strict – raise on failure. Pipelines pass False: a registry that cannot be written is worth one printed line, never a lost run.

Returns:

the artifacts registered, in declaration order.

spacr.artifacts.registry_path(project: str | os.PathLike | None = None) → str

Return the registry file for project.

Parameters:

project – the project root. Ignored when ARTIFACTS_DB_ENV is set, which points every project at one shared registry.

Returns:

an absolute path. The file need not exist yet.

Raises:

ValueError – when no project is given and no override is set.

spacr.artifacts.settings_hash(settings: Mapping[str, Any] | None) → str

Return a digest over the material settings of a run.

Two runs with the same digest cannot differ in anything that changes their output, so a downstream artifact whose recorded digest still matches the current settings is not stale on their account.

Parameters:

settings – a settings dict, or None.

Returns:

a lowercase SHA-256 hex digest (of {} when there are none).

spacr.artifacts.ARTIFACTS_DB_ENV = 'SPACR_ARTIFACTS_DB'

spacr.artifacts.ARTIFACTS_DB_NAME = 'artifacts.db'

spacr.artifacts.FULL_HASH_LIMIT_BYTES = 268435456

spacr.artifacts.SCHEMA_VERSION = 1

spacr.artifacts.STATUS_COMPLETE = 'complete'

spacr.artifacts.STATUS_FAILED = 'failed'

spacr.artifacts.STATUS_PARTIAL = 'partial'