starplast.metrics
Scoring a map by how well it would let you ANNOTATE, not by how tidy it looks.
The scores this project already had are agreement scores: F1 between clusters and labels, V-measure,
adjusted Rand. They answer "does this clustering correspond to that labelling", which is a question
about a partition and inherits every weakness of one -- change min_cluster_size and the number
moves, because a different partition of the same map is a different answer.
The ones here answer a different question: if I ranked every gene by how much its neighbours suggest it is a rhoptry protein, how good would that ranking be? That is what guilt by association actually does, and a ranking has its own well-worn statistics.
Why AUPRC is the one to read, and AUROC the one to report
Both summarise a ranking; they disagree about what matters, and on this data the disagreement is large.
AUROC asks: take a random rhoptry gene and a random non-rhoptry gene -- how often is the rhoptry one ranked higher? It is prevalence-independent, which sounds like a virtue and here is a trap. With 120 rhoptry genes among 8,140, a ranking can put four thousand irrelevant genes above the first true one and still score 0.9, because the metric spends most of its mass on comparisons nobody will ever make. Reported because it is what everyone else reports and comparisons need it.
AUPRC (average precision) asks about the top of the list only: of the genes I would actually go
and test, what share are right. It moves with prevalence, so a raw value means nothing on its own --
which is why lift is computed alongside it, dividing by the prevalence a random ranking achieves.
An AUPRC of 0.30 for a class that is 1.5% of the proteome is a twentyfold enrichment over chance and
a genuinely useful shortlist; an AUPRC of 0.30 for a class that is 28% is worse than guessing.
Leave-one-out, which is the whole reason these numbers are honest
A gene's score for class c is the share of its neighbours carrying c -- excluding itself. Without that exclusion every labelled gene votes for its own label, every score is inflated by exactly one vote, and small clusters (where one vote is a large share) are inflated most. The result is a map that scores beautifully and predicts nothing, which is the same failure as reading an enrichment off a layer that built the map.
What leave-one-out costs, stated because it changes how the numbers read
Removing a gene's own vote biases the chance level DOWNWARD, and not by a rounding error. In a cluster of n genes of which m carry the class, a positive gene scores (m-1)/(n-1) and a negative one scores m/(n-1) -- so inside any cluster that carries no real information, every positive sits just below every negative. Measured on four balanced classes in clusters of sixty, a randomly shuffled clustering scores AUROC 0.36 rather than 0.50.
Three consequences worth carrying:
- Do not read 0.5 as the floor.
chance_levelmeasures the floor for a given clustering by permuting the labels, the same permutation nullobjectives.null_scoreuses, and that is what a score should be compared against. - The bias shrinks as clusters grow, which is a mild pressure toward larger clusters among clusterings that are equally uninformative. It does not reach clusterings that carry real signal -- a pure cluster scores near 1 either way -- but it is a reason not to optimise on AUROC alone.
- AUPRC lift is the more robust target, being normalised by prevalence, which is why it is what
optimizeclimbs.
The neighbourhood metrics, and why they matter more than they look
knn_purity and trustworthiness score the EMBEDDING, with no clustering involved at all. That
makes them the right tools for the problem this project actually has -- a clustering that returns two
clusters over eight thousand genes tells you nothing about whether the map underneath it is good.
Separating "is the map good" from "did the clustering find what is in it" is the difference between
tuning two things at once and tuning them one at a time.
Each gene's score for one category: the share of its CLUSTER carrying it, minus itself.
Noise points score zero rather than being dropped. A map that calls half the proteome noise has genuinely failed to rank those genes, and letting them out of the denominator would hide exactly that failure behind a better-looking curve.
The same score from the k nearest genes in the map, with no clustering in the way.
AUROC, AUPRC and the lift of AUPRC over a random ranking, for one class.
Undefined rather than zero where a class has no positives or no negatives: an AUROC over a column of ones is not 0.5, it is a question that was not asked, and averaging a fabricated 0.5 into a summary quietly drags every real number toward the middle.
What this clustering scores for this category when the labels are shuffled.
The floor is not 0.5 and not the prevalence -- leave-one-out moves both, by an amount that depends on how big the clusters are. Measuring it costs twenty cheap re-scorings and is the difference between "AUROC 0.61, promising" and "AUROC 0.61 against a floor of 0.58".
One row per category: how well this map ranks the genes that belong to it.
Both rankings where coordinates are given -- by cluster and by neighbourhood -- because the gap between them is diagnostic. A map with good neighbourhood numbers and poor cluster numbers has the structure and lost it in the clustering, which is a different problem with a different fix.
A ranking table as a handful of numbers, weighted and unweighted.
Both, because they answer different questions and disagree loudly on this data. The plain mean is a statement about CATEGORIES -- a rare compartment counts as much as a common one, which is what a reader looking for somewhere to work wants. The weighted mean is a statement about GENES, and is dominated by whichever two categories are largest.
What share of a labelled gene's k nearest neighbours share its label.
No clustering anywhere in it, which is the point: this is the number to tune an embedding on
before choosing how to cut it up. Chance level is the sum of squared class frequencies, not
zero, so read it against expected_purity below.
What knn_purity would be if the map were shuffled: the sum of squared class frequencies.
How much of the original neighbourhood structure survived the projection, in [0, 1].
The one metric here that needs no labels at all, and the only defence against a map that looks beautifully clustered because the projection invented the clusters. Subsampled, because it is quadratic in the number of genes and the estimate is stable well before 8,140 of them.
The agreement and shape statistics for a clustering, in one call.
Silhouette is on the clustering's own geometry and says nothing about biology -- a map cut into tidy round groups that correspond to nothing scores well. It is here to be read ALONGSIDE the label metrics: high silhouette with low AUPRC is the signature of a clustering that found the shape of the projection rather than the shape of the data.
Every number above, for one configuration, as a flat dict a results table can hold.