Edit on GitHub

starplast.literature

Mentions: the one auditable intermediate between documents and every literature-derived number.

Before this module, per-gene publication counts and co-mention pairs were accumulated in a single pass inside build_graph, with no intermediate. Nothing could be checked, and no figure could be recomputed at a different confidence threshold without re-scanning the whole corpus. Everything downstream -- coverage, attention correction, co-mention edges -- now derives from one tidy table:

gene_id | doc_id | source | match_kind | section | n_hits

source keeps abstracts and full texts separate (they answer different questions -- see corpus.py) and match_kind keeps confidence tiers separate (see identity.py), so a coverage claim can always name the evidence it rests on.

Co-mention is counted per unit, not per document. For an abstract the unit is the record; for a full text it is the paragraph, because two genes named in the same paragraph plausibly stand in a relation while two genes named anywhere in a 10,000-word paper mostly do not.

TIER = {'accession': 'accession', 'accession_prev': 'accession', 'accession_strain': 'accession', 'symbol': 'symbol', 'alias': 'symbol'}

MAX_GENES_PER_UNIT = 12

SECTION_ORDER = ('title', 'abstract', 'body', 'caption')

def scan( docs, index, max_genes_per_unit: int = 12, log=<built-in function print>):

Scan Documents against a GeneIndex.

Returns (mentions, co, meta) where mentions is the tidy DataFrame described above, co maps source -> Counter of (gene_a, gene_b) -> number of units naming both, and meta carries the unit counts that the attention correction needs as its denominator.

def comention_edges(co: collections.Counter, meta: dict, source: str, min_count: int = 2):

Raw co-mention counts and their attention-corrected residuals for one source.

Attention correction (design decision 3, and a correctness feature rather than decoration): two genes named in n1 and n2 of N units are expected under independence to share n1*n2/N units. What the app shows by default is the residual log2((observed + 0.5) / (expected + 0.5)), because raw co-mention reproduces the field's popularity contest -- GRA16 and ROP18 become bright hubs on attention alone.

The denominator is the number of units, which is what independence is defined over. An earlier version divided by the sum of per-gene counts, a larger and differently-scaled quantity, which inflated every expectation and shrank every residual toward zero.

def coverage(mentions: pandas.DataFrame) -> pandas.DataFrame:

Genes named, broken out by source and confidence tier -- never as one undifferentiated number.

def publication_counts(mentions: pandas.DataFrame, source: str | None = None) -> pandas.Series:

Distinct documents naming each gene. This is the denominator of the attention correction.

DEPTH_OF = {'title': 'focal', 'abstract': 'substantive', 'body': 'incidental', 'caption': 'incidental'}

DEPTH_ORDER = ('focal', 'substantive', 'incidental')

def attention_depth(mentions: pandas.DataFrame) -> pandas.DataFrame:

Per gene: how many papers name it where, and the strongest tier any paper affords it.

Papers are de-duplicated by PMID across the two sources. Note that the abstract corpus carries only title and abstract text, so it can never contribute an incidental paper -- incidental evidence is visible only where a full text is available. That asymmetry limits the counts, not the tiering: a title mention means the same thing whichever source it came from.