starplast.clustering
Clustering an embedding, and asking what — if anything — the clusters correspond to.
Two parts. walk_* searches hyperparameters and scores each setting. battery then asks, for every
feature, whether it separates the clusters, and reports it in the form a biologist actually wants:
of all known tachyzoite genes, what fraction landed in cluster 1, and of cluster 1, what fraction are
tachyzoite?
The circularity guard is the point of this module. If a feature fed the embedding, then finding that
it separates the resulting clusters is guaranteed and means nothing — the clusters were built to separate
it. Every result is therefore labeled used or held_out, and only held_out features are evidence.
Reporting a used feature as a discovery is the single easiest way to produce a confident artefact here,
so battery refuses to sort the two together.
Scores are chosen so features of different types can be ranked against each other:
- categorical feature x cluster -> Cramer's V (0-1) from the contingency table, plus per-cluster precision and recall and a Fisher exact test
- continuous feature x cluster -> epsilon-squared (0-1) from Kruskal-Wallis, plus per-cluster medians
Both are bounded, both mean "share of this feature's variation explained by cluster membership", and both get Benjamini-Hochberg correction across the whole battery, which for ~90 features and k clusters is a lot of tests.
Grid over eps x min_samples. eps defaults to a spread around the median 4-NN distance.
HDBSCAN handles the variable density UMAP produces far better than a single global eps.
Cluster an embedding, returning a label per point with -1 for noise.
HDBSCAN by default: it allows clusters of differing density and calls the rest noise, which suits a proteome where most genes belong to no tight group. Calling that noise is the honest answer.
Return (summary_row, per_cluster_rows) for one categorical feature.
Return (summary_row, per_cluster_rows) for one continuous feature.
Test every feature against the clustering.
used_features are the node-table columns that fed the embedding. Those, and anything that restates
them (see _derived_from), are reported but marked used / derived, because a feature the clusters
were built from separates them by construction. Only held_out results are evidence.
The best cluster for each CATEGORY of each held-out feature: precision, recall, F1.
The battery reports one number per feature -- Cramer's V over the whole contingency table -- and
a single number over 27 compartments hides the thing worth knowing. V = 0.2 is the same value
whether every compartment is weakly smeared across every cluster or one falls out cleanly and
the rest are noise, and those are entirely different findings. Only the per-category table
separates them, and it uses the same "best single cluster per label" convention as
search.score_recovery, so the two cannot disagree about what recovery means.
F1 is computed here rather than read off the battery, whose rows are per (cluster, category) pair: the row that matters for a category is its best one.
min_in_cluster is the singleton guard in its per-category form, and it is not optional.
Ranked by lift without it, the top of this table was a cluster holding ONE gene of a rare
category at 5x enrichment -- true, meaningless, and indistinguishable at a glance from a real
result. The same floor objectives.score applies for the same reason.
Turn the battery into the sentences a reader wants, held-out features only.
Why this clustering cannot support a recovery claim, or "" when it can.
Returned as a REASON rather than a boolean so a caller can report which guard fired -- the three fail in different ways and a reader deciding what to change needs to know which.