Edit on GitHub

starplast.objectives

What "good structure" means, made explicit and selectable.

F1 is one answer to a question that has several, and the right one depends on what the structure is for. Hunting a single compartment is not the same job as organizing the whole proteome, and a search that only ever maximises one blended number cannot tell you it found the other thing.

The four objectives a user actually asks for, in their own words, and what each one is:

"most clusters are one label"                  PRECISION, averaged over clusters
"most labels are in one cluster"               RECALL, averaged over labels
"one or more clusters is mostly one label"     MAX PRECISION over (cluster, label) pairs
"one or more labels is in mostly its own"      MAX F1 over pairs

Precision is |c and l| / |c| -- of this cluster's members, what fraction share a label. Recall is |c and l| / |l| -- of this label's genes, what fraction are in this cluster. Everything below is a different way of aggregating those two numbers.

EVERY OBJECTIVE HERE HAS A DEGENERATE MAXIMISER, and the guards matter more than the metrics:

mean precision over clusters is maximised by many TINY clusters; a singleton is perfectly pure.
mean recall over labels is maximised by ONE GIANT cluster; recall is 1.0 for every label. This is not hypothetical. On the full proteome the positive control scored 0.675 -- the highest of four targets -- from a two-cluster solution whose per-label recalls were 1.000, 1.000, 1.000.
max precision is maximised by one tiny pure cluster: three co-located genes, precision 1.0, and nothing to annotate from.

So min_cluster_size is enforced here rather than left to the caller, and every objective reports the coverage and cluster count beside its score. A number without those is not interpretable.

WEIGHTING IS A CHOICE, not a detail. search.score_recovery weights by label size, so on this proteome nucleus-chromatin at 769 genes dominates and dense granules at 167 barely registers -- and someone hunting dense granules is optimizing against themselves. macro weights every label equally and is the right default for finding a rare class.

NOISE = -1

EXPLANATION = 'PRECISION and RECALL answer different questions, and which one you want depends on what you are\nlooking for.\n\nFor one cluster c and one label l:\n\n precision = |c and l| / |c| of THIS CLUSTER\'s members, what fraction share the label\n recall = |c and l| / |l| of THIS LABEL\'s genes, what fraction are in the cluster\n F1 = harmonic mean both at once; high only when neither is low\n\nSo:\n\n "most clusters are one label" precision, averaged over CLUSTERS\n "most labels are in one cluster" recall, averaged over LABELS\n "one or more clusters is mostly one label" the single best precision\n "one or more labels has mostly its own" the single best F1 -- "its own cluster"\n claims both directions at once\n\nEVERY ONE OF THESE HAS A DEGENERATE SOLUTION THAT WINS IT OUTRIGHT. Measured on a synthetic case\nwith three labels of sixty genes:\n\n perfect one big cluster all singletons\n mean precision 1.00 0.33 1.00\n mean recall 1.00 1.00 0.02\n mean F1 1.00 0.50 0.03\n best precision 1.00 0.33 1.00\n V-measure 1.00 0.00 0.35\n ARI / AMI 1.00 0.00 0.00\n\nAny precision objective is won by shattering the map into singletons, because a cluster of one is\nperfectly pure. Any recall objective is won by one giant cluster, because everything is then in its\nbest cluster. Neither is hypothetical: on the full proteome the positive control scored 0.675 -- the\nhighest of four targets -- from a two-cluster solution whose per-label recalls were 1.000, 1.000 and\n1.000. A hyperparameter walk is an optimizer, and an optimizer finds exactly what you reward.\n\nThree things follow, and they are enforced rather than advised:\n\n - a minimum cluster size, which is what stops the singleton exploit;\n - coverage and cluster count reported beside every score, because the attention control\'s winning\n configuration was 52% noise;\n - chance-corrected agreement (ARI, AMI) available alongside, since it is the one family immune to\n BOTH degenerate cases. A high objective next to an ARI near zero means the objective was gamed.\n\nWEIGHTING IS ALSO A CHOICE. Averaging over labels can weight every label equally (macro) or every\ngene equally (size). On this proteome nucleus-chromatin has 769 genes and dense granules 167, so\nsize weighting means a search for dense granules is dominated by the nucleus. Macro is the default\nfor that reason.\n\nFINALLY, precision_at_recall is usually the objective an annotation actually wants: the purest\ncluster that still holds enough of the label to be worth annotating from. Best precision alone\nselects three co-located genes at precision 1.00 and gives you nothing to work with.'

OBJECTIVES = {'mean_precision': 'most clusters are one label (purity, averaged over clusters)', 'mean_recall': 'most labels are in one cluster (completeness, averaged over labels)', 'mean_f1': 'labels and clusters correspond, on average (recommended)', 'best_precision': 'at least one cluster is mostly one label', 'best_f1': 'at least one label has mostly its own cluster', 'n_recovered': 'how many labels clear a threshold (immune to class prevalence)', 'v_measure': 'the whole clustering agrees with the whole labeling', 'precision_at_recall': 'the purest cluster that still holds enough of a label to annotate from (recommended for annotation)'}

NULL_PERMUTATIONS = 20

def pairs( labels: numpy.ndarray, truth: pandas.Series, *, min_label: int = 15, min_cluster: int = 10, exclude_labels=frozenset({'', 'unlabeled', 'unassigned', 'unlabelled', 'nan', 'none', 'unknown'})) -> pandas.DataFrame:

Precision, recall and F1 for every (cluster, label) pair worth considering.

One table that every objective below is computed from, so they cannot disagree about the underlying numbers. Noise is dropped: "it is in the noise" is not a structure.

def score( labels: numpy.ndarray, truth: pandas.Series, *, objective: str = 'mean_f1', weighting: str = 'macro', category=None, min_recall: float = 0.25, threshold: float = 0.5, **kw) -> dict:

Score a clustering under one named objective.

category restricts scoring to one label or to several -- a string, or any sequence of them. "Precision for dense granules" and "recall for dense granules and rhoptries" are both ordinary questions, and they are different questions, so the objective and the label set are independent choices rather than one combined mode.

With several, the aggregation follows the objective: the mean_* objectives average over the chosen labels and the best_* objectives take the best among them. Nothing else changes, so a score over two labels is comparable with a score over all of them.

weighting is macro (every label equal) or size (every gene equal); macro is the default because the alternative buries exactly the rare classes worth hunting.

def v_measure(labels: numpy.ndarray, truth: pandas.Series) -> float:

Homogeneity and completeness together, as their harmonic mean.

The whole-clustering version of the user's first two objectives: homogeneity IS "most clusters are one label" and completeness IS "most labels are in one cluster", so V-measure is what asking for both at once means. Unlike either alone it cannot be won by one giant cluster (completeness 1, homogeneity 0) or by singletons (homogeneity 1, completeness 0).

def agreement(labels: numpy.ndarray, truth: pandas.Series) -> dict:

Chance-corrected agreement between the clustering and the labeling.

Reported alongside any objective because it is the one family immune to BOTH degenerate cases: one giant cluster and all-singletons each score about zero, since neither beats chance. A high objective with an ARI near zero means the objective was gamed.

def null_score( labels: numpy.ndarray, truth: pandas.Series, *, n_permutations: int = 20, seed: int = 0, **kw) -> float:

What this objective scores on the SAME clustering with the labels shuffled.

The baseline moves with the number of classes and with how uneven they are, and none of the raw precision/recall/F1 objectives know that. Mean F1 of 0.30 across three balanced classes is worse than guessing; the same 0.30 across twenty-four compartments of wildly different sizes is remarkable. A search that compares configurations across targets is comparing against different nulls without saying so.

Permutation rather than a formula: shuffling the labels preserves exactly how many classes there are, how many genes are in each, and the sizes of the clusters they are being matched against, so the null answers "what would this objective give me for nothing, on this data". That is the same idea that makes ARI and AMI trustworthy, applied to whichever objective was chosen.

def adjusted( labels: numpy.ndarray, truth: pandas.Series, *, n_permutations: int = 20, seed: int = 0, **kw) -> dict:

A score, its permutation null, and the score corrected for it.

adjusted = (score - null) / (1 - null), so 0 is "no better than shuffled labels" and 1 is perfect, whatever the objective and however many classes there are. Negative means the structure is worse than chance, which is a real result and should not be clipped away.

n_labels counts the classes the score was computed over -- the number the null is there to make sense of. It was every distinct string in the column, which counts unassigned as a compartment and a class too small to score as a class, and so said "24 classes" for a score computed over eleven.