Edit on GitHub

starplast.discovery

Three questions worth optimising a map for, and the instances of each that it finds.

Everything else in this project scores a structure by how well it RECOVERS something already known: hold out the localisation labels, cluster, see how many come back. That is the right way to decide whether a map is trustworthy and the wrong way to decide whether it is useful, because a map that recovers what is known perfectly and says nothing about anything else has produced no work.

These are the two shapes of claim a map like this can actually make.

Guilt by association. A cluster whose labelled members are overwhelmingly one thing, and which also holds genes nobody has labelled. The labelled part is the evidence, the unlabelled part is the prediction, and the strength of the claim is the enrichment: 41 of 58 labelled genes in one cluster being IMC is a statement about the 12 unlabelled ones sitting with them. The same shape works on a measured quantity rather than a category -- a cluster whose screened members are uniformly essential says something about the members of that cluster nobody screened.

Layer disagreement. A cluster that is homogeneous in one layer and split in another. Same compartment, opposite fitness. Same cell-cycle phase, opposite stage expression. Agreement between layers mostly re-derives what is already known; disagreement is where a category is hiding a distinction, and it is the more interesting of the two because nothing about it is predictable from either layer alone.

What makes these honest rather than a slot machine

Three things, all of them enforced here rather than left to the reader.

Multiplicity. A run tests every cluster against every category. At 40 clusters and 12 categories that is 480 tests, and at p < 0.05 two dozen come back by chance. Every p-value here is corrected across the whole family of tests the run performed, and the corrected value is what ranks a finding.

Circularity. A cluster enriched for localisation, in a map built from localisation, is a statement about arithmetic. Findings carry the columns that built the map, and one whose layer was an input is marked rather than dropped -- it is still a fact about the map, it is just not evidence about biology. The same guard as search.excluded_for, applied at the point of interpretation.

Prevalence. A cluster that is 90% "unassigned" in a proteome that is 60% unassigned has found nothing. Enrichment is always against the background rate in the labelled population, never against uniformity.

ABSENT = frozenset({'', 'unlabeled', 'unassigned', 'unlabelled', 'unclear', 'not assigned', 'unknown', 'nan', 'none', 'no data'})
MIN_CLUSTER = 10
MIN_CATEGORY = 5
MIN_PREDICTIONS = 3
MIN_LIFT = 2.0
MIN_PURITY = 0.25
MAX_SHARE = 0.25
MIN_SPLIT = 1.0
MIN_EFFECT = 0.5
def guilt( nodes: pandas.DataFrame, labels: numpy.ndarray, layer: str, kind: str = None, min_cluster: int = 10, min_category: int = 5, min_predictions: int = 3, min_lift: float = 2.0, max_share: float = 0.25, used_columns=()) -> pandas.DataFrame:

Every cluster that says something about the genes in it nobody has measured.

Returns one row per claim: which cluster, which layer, how strong the evidence, how many genes it is about, and which genes those are. Empty where the structure supports no claim at all, which is a real answer and the commonest one for a badly chosen embedding.

def disagreement( nodes: pandas.DataFrame, labels: numpy.ndarray, layer_a: str, layer_b: str, min_cluster: int = 10, min_category: int = 5, purity: float = 0.6, used_columns=()) -> pandas.DataFrame:

Every cluster that agrees about one layer and disagrees about another.

layer_a is the one that has to be homogeneous -- the thing these genes share -- and layer_b is where they part company. The claim is a subdivision: this compartment contains two kinds of gene, and here is the measurement that separates them and the members of each side.

Not symmetric, and deliberately called twice with the layers swapped when both directions are wanted: "same compartment, split fitness" and "same fitness, split compartment" are different claims about different biology.

def conjunction( nodes: pandas.DataFrame, labels: numpy.ndarray, layer_a: str, layer_b: str, min_cluster: int = 10, min_category: int = 5, min_interaction: float = 1.25, max_share: float = 0.25, used_columns=()) -> pandas.DataFrame:

Clusters enriched for a combination of two categorical layers.

joint_lift says how enriched the pair is over its proteome-wide prevalence. The qualifying interaction_ratio divides that by the stronger single-layer lift: values above one mean the pair says more than either margin alone. product_ratio also records the stricter literal joint/(A-lift x B-lift) diagnostic; in a balanced, perfectly resolved A x B grid it is exactly one by construction, so using it as the discovery threshold would paradoxically reject the canonical crossed-factor fixture.

def explains_fragmentation( nodes: pandas.DataFrame, labels: numpy.ndarray, layer_a: str, layer_b: str, permutations: int = 200, seed: int = 42, min_cluster: int = 10, min_category: int = 5) -> pandas.DataFrame:

Do sibling clusters enriched for the same A category separate on categorical layer B?

Returns one row per A category with at least two enriched sibling clusters and a final __run__ row. Distances are total-variation distances between B-composition vectors. The null shuffles B within the A category, preserving both margins while breaking its assignment to sibling clusters.

def yield_score(findings: pandas.DataFrame, alpha: float = 0.05) -> float:

How much a structure found, as one number an optimiser can climb.

The product of strength and reach, summed over surviving findings and logged: a configuration that produces one overwhelming claim about four genes should not beat one that produces six solid claims about ninety. Findings that fail correction contribute nothing rather than a little, because a claim that does not survive multiplicity is not weak evidence -- it is a finding the run did not make.

Circular findings are excluded outright. Their strength is arithmetic, and rewarding it would drive the optimiser straight into the configuration that puts a layer into the map and reads it back out, which is the failure this project has already made once.