Edit on GitHub

starplast.corpus

Documents: one uniform stream over the abstract corpus and the open-access full texts.

Two very different sources have to be read the same way but must never be silently merged, because they support different claims:

abstracts -- 33,924 PubMed records. Complete coverage of the field, but a gene named only in a paper's body is invisible. This is the source behind the "at most 7.4% of the proteome is named" figure, and it is a lower bound on attention.
fulltext -- the open-access XML on local disk. Far deeper per paper, but a biased subset: only papers whose publisher deposited them. Coverage computed over full texts answers a different question and cannot be quoted as if it covered the field.

Every Document therefore carries its source, and every Section carries its kind, so downstream code chooses granularity explicitly instead of inheriting whatever the parser happened to concatenate.

Section kinds: title, abstract, body, caption.

Reference lists are excluded. A reference list reproduces the titles of cited papers, so including it would credit a paper with mentioning every gene named in its bibliography -- inflating both coverage and co-mention with the citation graph rather than the paper's own content.

@dataclass(frozen=True)

class Section:

One titled section of a full text, with its body.

Section(kind: str, text: str)

kind: str

text: str

@dataclass(frozen=True)

class Document:

One article: its identifiers, its sections, and the text they contain.

Document( doc_id: str, source: str, pmid: str | None, year: str | None, sections: tuple)

doc_id: str

source: str

pmid: str | None

year: str | None

sections: tuple

def iter_abstracts(path: str):

Yield one Document per PubMed record in the JSONL corpus.

def parse_jats(path: str) -> Document | None:

Parse one PMC JATS file into a sectioned Document, or None if it is unreadable.

def iter_fulltexts(directory: str, limit: int | None = None):

Yield one Document per PMC XML file in directory.

def count_fulltexts(directory: str) -> int:

How many open-access full texts are present locally. A biased subset, by construction.