starplast.datasets

One dataset the map is built from, and everything known about where it came from.

The unit of provenance for the whole project: the README table, the methods document and the per-dataset scripts are all generated from these, so a source is described once and cannot drift between three prose documents. citation is None where the originating publication has not been confirmed -- a real gap, listed by unresolved(), rather than a guess that could reach a paper.

The explicitly declared organism; dataset names are not species identifiers.

Every dataset, optionally filtered by level or kind.

The dataset with this key. Raises KeyError naming the key if there is none.

Which dataset produced a given node-table column -- of organism's table, when given.

Pass the organism wherever the table is known: without it, a column name both organisms use resolves to whichever dataset was registered first.

The node columns a derived column was computed from, as DECLARED by whoever derived it.

Declaration is needed in addition to measurement, not instead of it. Measured association is pairwise, and a label computed as the argmax of three columns is a JOINT function of them: each source individually explains only about 0.6 of it, under any sensible exclusion threshold, so a pairwise measure cannot see the derivation however good the statistic is. Measurement catches the undeclared leak -- a renamed copy -- and declaration catches the joint one. Neither alone is enough, and this project has already been burnt by trusting one of them.

Transitive derived outputs that depend on any supplied column.

Leakage travels both directions in the dependency graph: holding out a derived target removes its raw inputs, while holding out a raw measurement must also remove every summary or label computed from it. The registry is small, so an explicit fixed-point walk is clearer than a cached graph.

Entries whose originating publication is not recorded. Confirm before citing.

Entries with a direct download URL, in the order the notebook should fetch them.

The registry as a DataFrame, for the methods table and the application's provenance view.

Where this dataset's file is on THIS machine, or None if it is not here.

The registry's path is recorded relative to the dataset root, which differs between a clone, the historical layout and a download cache -- so it is resolved through paths.find rather than opened directly. Opening it directly is what made the build work on exactly one machine.

(can_fetch, how). Says which of the three situations a dataset is in, rather than guessing.

Return a local path for this dataset, downloading it if it is absent and can be fetched.

Returns None rather than raising when the data genuinely cannot be obtained automatically. Several of these datasets exist only inside a paper's supplementary section and one exists only as raw instrument files; pretending otherwise would produce a path to something that is not the dataset.

Registry entries whose data is not on this machine. The honest first-run report.

SHA-256 of a file, streamed -- some of these are gigabytes.

The SHA-256 recorded for each downloaded file, so a reissued supplement is caught.

Pin what was fetched. Returns the digest.

True when the file matches what was pinned, or when nothing was pinned yet.

An unpinned file is not a failure -- most of this tree arrived before checksums existed -- so the honest answer for it is "no claim", and the digest is recorded so the NEXT fetch has something to check against.

The linked dataset catalogue, generated as Markdown from this registry.

Generated rather than written by hand because a hand-written table drifts the moment a dataset is added, and a README that misstates which data is inside is worse than one that omits it. A test asserts the committed README matches what this produces.