starplast.validate
Does a cluster-based annotation actually work? Measured by hiding labels we already have.
The workflow this exists to check: hold out a label, build a map, find a cluster that is mostly one category, and propose the unlabeled members of that cluster as candidates. That produces a list of genes and no error rate, and a candidate list with no error rate is a list of guesses.
The check is to make the same inference where the answer is already known. Hide a fraction of the genes that DO carry a category, rebuild nothing -- the embedding is fixed and was built without the label -- cluster it, and ask: of the hidden genes, how many land in the cluster the visible ones picked out? That is precision and recall for exactly the inference being made, on genes whose true label we withheld from ourselves.
Two things this is careful about.
The embedding must not have seen the label. If compartment fed the map, a cluster matching
compartment is circular and this measures nothing -- so the caller passes the columns the embedding
used and the run is refused when the target column is among them. The check is on the COLUMN: it
used to be on the category value, comparing "dense granules" against a list of column names, which
is never true -- and the application passed block names, so the guard never fired at all.
The hidden genes are hidden from the SCORING, not from the embedding. Re-embedding per fold is the
stricter test and costs a full embedding per fold; refit does it, and requires a rebuild
callable to do it with. Asking for it without one is an error rather than a flag, because refit
used to change only the sentence the result prints -- so a run that re-fit nothing described itself
as "re-embedded per fold". Whether it re-fit is recorded on every row either way, since the
difference is invisible in the numbers.
A candidate list is the output all of this exists to qualify, so candidates attaches the
composition of the cluster it came from, and orthogonal_support counts how many genes already
carrying the category share an orthogroup or a domain with each candidate -- agreement from
evidence the map never saw. A column that fed the embedding is refused there too: its agreement
would be the map agreeing with itself.
One fold: what was hidden, and how much of it came back.
The answer, with the caveats that have to travel with it.
Why this label cannot be scored against this embedding, or "" if it can.
Returned as a sentence rather than raised from three call sites, because the interface has to SHOW this: a tab that pops an exception dialog teaches people that validation is broken, when what happened is that validation correctly refused to produce a meaningless number.
The test is on the label COLUMN, not on the category value. It used to be the value -- category
in used_columns -- which compares "dense granules" against a list of column names and is
therefore never true. In the application it was worse than never true: the panel passed the
embedding's BLOCK names, so the guard was comparing a compartment against "expression_summary"
and the tab would happily score a target the map was built from.
Hide some of a category's genes and see whether the clustering puts them back.
labels is a clustering of a FIXED embedding. truth is the held-out label column. For each
fold, hold_frac of the genes carrying category are hidden; the cluster is chosen by which one
holds most of the still-visible members -- exactly as a user would pick it by eye -- and then
scored against the hidden ones, who had no say in the choice.
Precision is over the cluster's members that were not visibly of this category, which is the set a user would actually annotate from. Recall is over the hidden genes.
refit requires rebuild, a callable taking the fold number and returning that fold's labels
-- a fresh embedding and clustering, so the estimate is not conditional on one particular map.
Asking for it without supplying one is an error rather than a flag: refit=True used to change
only the sentence the result prints, so a run that had re-fit nothing reported itself as
"re-embedded per fold". A number that overstates how it was obtained is worse than a weaker
number that describes itself accurately.
Run masked_recovery for every category with enough genes, and rank them.
Per-category rather than one global number, deliberately. A method that recovers hidden dense granules but not hidden rhoptries is not "60% accurate"; it works for one compartment and not the other, and only the per-category table says so.
log is called once per category, which is what makes the run reportable and, in the
application, stoppable: the cancellation is raised from the reporting callable, so a run with
refit on -- a full embedding per fold per category -- can be stopped between categories
instead of at the end.
The cluster holding most of a category's labeled genes, or None if none does.
The same choice masked_recovery makes inside each fold, and the same one a user makes by eye,
so the cluster a candidate list comes from is the cluster the error rate was measured on. Noise
is never chosen: "it is in the noise" is not an annotation.
For each candidate, how many genes already labeled category it shares a group with.
A candidate's whole claim is that it sits near genes of one category in a map. That is one piece of evidence, and the honest question about it is whether anything the map never saw agrees. Sharing an orthogroup or a domain with the labeled members is such a thing: not proof -- paralogs of a dense-granule protein need not be dense-granule proteins -- but independent, and countable.
A column that fed the embedding is refused rather than counted, and the refusal is logged. Its agreement would be the map agreeing with itself, which is the failure this whole tab exists to put a number on.
Zero support is the common answer and is reported as zero, never as a blank: most of this proteome has no orthogroup neighbour with a measured label, and a candidate with no independent support is exactly the one to be careful about.
The genes a cluster would have you annotate: its members with no label of their own.
Returned with the cluster's composition attached, so a candidate never travels without the number that says how much to believe it.