starplast.archive
A provenance-first archive of pan-Apicomplexan source files. Instruction 38.
Deliberately NOT wired into the graph. This builds a future-ready store so that extending the leakage-aware slot hierarchy to another parasite does not mean repeating dataset discovery, and integrating it now would quietly make Toxoplasma claims out of Cryptosporidium files.
Three properties, each of which exists because its absence has cost this project time:
Nothing is overwritten. A file already on disk is left exactly as it is and recorded as
present, because the archive is a store of what was retrieved and when, and a re-run that silently
replaced a 2019 download with a 2026 one would destroy the provenance it exists to keep.
A failure is recorded, not omitted. Controlled-access, withdrawn, supplement-less and
plain-404 sources go into the manifest with their status. An archive that lists only what worked is
an archive that sends the next session to rediscover the same dead ends -- which is exactly what the
empty slots' blocked_by verdicts prevent in the slot atlas.
Every retrieved file carries a checksum and a size. "Downloaded" is not a claim anyone can check; a sha256 is.
One downloadable table, described well enough that somebody else could fetch it.
slot_family is a guess about where this would land IF it were integrated, and it is recorded as
a guess: the archive does not fill slots, and a file catalogued as fitness that turns out to be
a growth curve should be discoverable as a mistake rather than believed.
sha256 of a file, read in chunks so a multi-gigabyte archive does not become a memory limit.
Whether the file is a table something can open, and what shape. "" when it is not one.
Instruction 38 asks for verification rather than a size check, and the distinction matters: a
404 page saved as .csv is 4 kB of valid HTML, and every archive that only checked sizes has one.
What happened to one source, which is what the manifest is made of.
Retrieve one source into the archive, or record why it was not retrieved.
opener is injected so the whole path is testable without a network: the interesting behaviour
here is what happens on failure and on a file that already exists, and neither should need a
live server to exercise.
Fetch every source and write the manifest. Returns the records.
The manifest is written after EVERY source rather than at the end, so a run interrupted halfway keeps what it retrieved -- which for a multi-gigabyte archive over a slow link is the difference between a resumable job and a wasted afternoon.
The manifest as CSV and JSON. CSV because a person reads it, JSON because a script does.
The checked-in catalogue of sources, which is what makes discovery reproducible.
Enumerate downloadable gene-level files across the VEuPathDB sites, resolving as it goes.
Reproducible in the sense instruction 38 requires: nothing here is a remembered accession. The
site-to-clade table is checked in, the release is whatever Current_Release points at today, the
organisms come from that release's own index, and the files come from each organism's directory.
Re-run next year and it finds next year's release without an edit.
fetcher is injected so the whole walk can be tested without a network.