Edit on GitHub

starplast.annotations

Annotations: a proposed label for a gene, and the numbers that make it a proposal rather than a claim.

A cluster is mostly one category; the unlabeled genes in it are candidates. Writing those down is a few lines of code and is the single easiest way for this project to start manufacturing exactly what it exists to prevent -- because a candidate list looks identical whether it is 90% right or 6%, and measured on the coarse level-of-detail components of this map it was 1-6%.

So the store refuses a row that does not carry its own error rate. Every annotation records:

gene, the label proposed, the configuration and cluster it came from, that cluster's
composition, the VALIDATED precision and recall for that category, whether the validation
re-embedded per fold, the date, and the reasoning.

Three rules, and each is a refusal rather than a convention:

  • Never written back into the node table. A separate file, always. The node table is measurement; the moment an inference is stored beside it, every downstream reader has to know which columns are which, and one of them eventually will not.
  • Never saved without a validated precision for that category. Not a warning -- an error. A precision from the same clustering the candidate came from is the only honest estimate of how often that annotation will be right, and a stored annotation without one is a guess wearing a record's clothes.
  • Its own color, used for nothing else. Measurement, inference and absence already have one each in this application. An annotation is a fourth thing, and reading as any of the other three is the failure mode -- see ANNOTATION_COLOR.

The file is CSV rather than a database: it is small, a person should be able to read it, and the reasoning field is the part a collaborator will want to argue with.

ANNOTATION_COLOR = (0.95, 0.35, 0.85)
MIN_PRECISION = 0.1
@dataclass
class Annotation:

One proposed label, with everything needed to judge it later.

Every field is here because leaving it out makes the row unfalsifiable. Without the configuration nobody can rebuild the map it came from; without the cluster's composition nobody can see how mixed it was; without the validation numbers nobody can say how often it is wrong.

Annotation( gene_id: str, proposed: str, target: str, cluster: int, blocks: str = '', na_policy: str = '', scaling: str = '', n_neighbors: int = 0, min_dist: float = 0.0, min_cluster_size: int = 0, seed: int = 0, cluster_size: int = 0, cluster_frac_category: float = nan, cluster_frac_contradicting: float = nan, enrichment: float = nan, precision: float = nan, recall: float = nan, folds: int = 0, refit: bool = False, date: str = '', reasoning: str = '')
gene_id: str
proposed: str
target: str
cluster: int
blocks: str = ''
na_policy: str = ''
scaling: str = ''
n_neighbors: int = 0
min_dist: float = 0.0
min_cluster_size: int = 0
seed: int = 0
cluster_size: int = 0
cluster_frac_category: float = nan
cluster_frac_contradicting: float = nan
enrichment: float = nan
precision: float = nan
recall: float = nan
folds: int = 0
refit: bool = False
date: str = ''
reasoning: str = ''
def check(self) -> str:

Why this annotation cannot be saved, or "" if it can.

A sentence rather than an exception because the interface has to show it: refusing is the feature, and a dialog that says "error" teaches people the store is broken rather than that the annotation is unjustified.

COLUMNS = ['gene_id', 'proposed', 'target', 'cluster', 'blocks', 'na_policy', 'scaling', 'n_neighbors', 'min_dist', 'min_cluster_size', 'seed', 'cluster_size', 'cluster_frac_category', 'cluster_frac_contradicting', 'enrichment', 'precision', 'recall', 'folds', 'refit', 'date', 'reasoning']
class AnnotationStore:

Annotations on disk, in their own CSV, never in the node table.

Re-read before every write rather than held in memory: this file is meant to be edited by hand and shared, and a store that cached it would silently overwrite whatever a collaborator added while the window was open.

AnnotationStore(path: str)
path
def load(self) -> pandas.DataFrame:

Everything saved so far, with the columns in their fixed order.

def save(self, annotations, replace: bool = False) -> pandas.DataFrame:

Append annotations, refusing any that cannot justify themselves.

The refusal is per row and it raises: a partial save that silently dropped the unjustified rows would leave the user believing they had saved what they saw.

A gene already annotated for the same target is REPLACED rather than duplicated -- the second opinion is the one someone just formed -- and the replacement is logged, because a changed annotation is exactly the kind of thing that should be traceable afterwards.

def remove(self, gene_ids, target: str | None = None) -> pandas.DataFrame:

Delete annotations for these genes. Withdrawing a proposal has to be as easy as making one.

def mask(self, gene_ids) -> numpy.ndarray:

Which of gene_ids carry an annotation, as a boolean array over that order.

What the map draws its fourth color from. Built here rather than in the window so that "is this gene annotated" has one answer.

def from_candidates( candidates: pandas.DataFrame, target: str, scores, configuration=None, reasoning: str = '', date: str = '') -> list:

Turn validate.candidates rows into annotations, carrying the validation numbers with them.

The join between a candidate and its error rate happens HERE, once, rather than at each call site: a candidate list and a precision that came from a different category, or a different clustering, would be worse than no number at all.