Edit on GitHub

starplast.build_graph

Build the Toxoplasma knowledge graph and its 3D embedding, once, into a cache the app loads instantly.

Design notes that matter (see ../HANDOFF.md):

  • Position is a UMAP embedding of a multimodal feature matrix, NOT a force-directed layout. Force-directed position is aesthetic and arbitrary; UMAP over expression, the seven fitness screens, compartment, orthology breadth, domain content and structure confidence gives positions where proximity means biological similarity.
  • Six edge types are stored separately and never merged. Each answers a different question and they are not interchangeable -- the seven CRISPR screens in particular disagree with each other substantially.
  • Co-mention edges are attention-biased. We therefore store BOTH the raw co-mention count and an attention-corrected residual (observed minus expected given each gene's own publication count). The app defaults to the corrected form because the raw form is confidently misleading.

Output: data/graph.npz + data/nodes.parquet

BASE = '/home/runner/.cache/starplast'
DS = '/home/runner/.cache/starplast/datasets'
OUT = '/home/runner/work/starplast/starplast/starplast/data'
FIT = ['fit_invitro_hff', 'fit_invivo_PE', 'fit_invivo_lung', 'fit_invivo_liver', 'fit_invivo_spleen', 'fit_naive_bmdm', 'fit_ifng']
def log(m):

Print a build step with a timestamp, so a long rebuild shows where it is.

def load_nodes() -> pandas.DataFrame:

Assemble the node table: every gene and every measured column, joined and identifier-resolved.

The authoritative assembly for the shipped data. Published supplements cite whatever accession was current when they were written, so everything joins through the identity layer -- without it a table keyed on TGGT1_ matches nothing and looks like a dataset with no coverage.

ABSTRACTS = '/home/runner/.cache/starplast/.claude/skills/toxoplasma-scientist/corpus/pubmed_toxoplasma.jsonl'
FULLTEXTS = '/mnt/wd4tb/skill_corpora/toxoplasma-scientist'
TOXONET = '/home/runner/.cache/starplast/toxonet/data/interim/edges_v3.parquet'
def literature_layer(nodes: pandas.DataFrame):

Scan both literature sources through the identity layer; return co-mention edges and node columns.

This replaces an earlier one-pass scan that matched only current ME49 accessions and ToxoDB symbols. That version could not see the old TGME49_0xxxxx accessions, the GT1/VEG accessions papers use interchangeably, or the Tg- prefixed symbol forms, and it wrote no intermediate anyone could audit. See identity.py, corpus.py and literature.py for the three layers this composes.

Abstracts and full texts stay separate all the way through: they are different populations (all of PubMed versus whichever papers a publisher deposited open access) and support different claims.

def build_edges(nodes: pandas.DataFrame):

Build every edge layer, kept separate and never merged into one score.

Merging would let a co-mention borrow the credibility of a measured crosslink, and no weighting recovers the difference afterwards -- see the Edges menu in the application.

EVIDENCE_FAMILY = {'coexpression': 'expression', 'cofitness': 'fitness', 'orthogroup': 'homology', 'domain': 'homology'}
REQUIRED_FAMILIES = frozenset({'expression', 'fitness'})
def structural_holes(edges: dict, nodes: pandas.DataFrame):

Gene pairs that two independent kinds of biology link and the literature never has.

This is the app's governing question: not what the field says, but where its map has a gap that the data says should be crossed. A hole is a derived relation, not an observed one -- it is the absence of a co-mention edge across a pair the measurements agree about -- and it is labeled as such.

Both co-mention layers count as literature, so a pair discussed anywhere, in any abstract or any open-access paragraph, is not a hole.

def unwritten_interactions(edges: dict, nodes: pandas.DataFrame):

Pairs that were measured to interact and that no paper has ever discussed together.

A stronger claim than structural_hole, and a different one. A hole says two independent phenotype measurements predict a relationship. This says a physical interaction was observed -- two residues covalently joined in a cell lysate, or a replicated pulldown -- and the literature still never put the two proteins in one sentence.

This is a merge of xlms and ip_ms, and it is an explicit, labeled one rather than a silent blending of edge types: both remain separately toggleable.

TG_EMBED_FEATURES = ['expr_tachy', 'expr_cyst', 'expr_max', 'mean_plddt', 'paralog_number', 'n_interpro', 'n_phosphosites', 'has_domain', 'lineage_specific']
def embed_features(nodes: pandas.DataFrame) -> list:

Which columns lay a table out, for whichever species' table this is.

The Toxoplasma list is explicit and stays that way. A second species cannot reuse it -- only six of its sixteen columns exist in pf_nodes.parquet, because the two arms measure different things and their stage names differ on purpose -- so for anything else this takes the numeric columns measured for more than half the genes.

Discovered rather than named for the second arm, and the reason is honest: naming a Plasmodium feature list would be inventing one, and a list invented without looking at what the data supports is worse than a rule that says what it did. The >50% floor is what keeps the layout from being driven by a column measured for forty genes.

def embed(nodes: pandas.DataFrame, return_metadata: bool = False):

Build the packaged display through the interactive embedding recipe system.

def main():

Rebuild the whole cache from the raw datasets. Minutes, and needs the raw tree.