starplast.track_record
The track record: hold every labelled gene out in turn, and keep what each strategy said.
A self-test hides a quarter of a label once and reports one number. That answers "does this strategy beat chance on this label", which is what a verdict needs, and it leaves the question a biologist actually asks unanswered: could this software have told me what I already know about THIS gene?
So this module runs the hold-out to completion. The labelled genes are split into folds by
ORTHOGROUP, every fold is hidden in turn, and the strategy is asked for the genes it cannot see --
so each labelled gene is held out exactly once, is never predicted by a model that saw it or its
paralog, and ends with a row of its own saying what was predicted, whether that was right, and how
sure the strategy was. Those rows are the ledger; summary pools them at the level being asked
about, from one gene up to one strategy, and every rate carries the count it rests on and a Wilson
interval, because a rate without those is a number nobody can act on.
ledger = evaluate(ctx, "feature_knn", "compartment") # one row per labelled gene
summary(ledger, "gene") # how each gene fared
summary(ledger, "class") # and each class, with what it is confused with
Abstentions are kept apart from errors throughout: a strategy that declines to answer is not wrong, and averaging the two together would hide exactly the difference the scorecard exists to show.
A fold number per position, whole orthogroups kept together.
Groups are dealt round-robin from a shuffled list rather than cut into blocks, so one enormous family cannot put a quarter of the genes into one fold.
Hold out every labelled gene in turn and record what key said about each.
One row per labelled gene: the fold it was hidden in, the label it carries, what the strategy predicted without seeing it or its orthogroup, whether that was right, and the strategy's own support for the call. Returns an empty frame, rather than raising, when the strategy cannot speak about this target at all -- a refusal is a fact about the pairing, not an error.
The strategies this can hold genes out for, in catalogue order.
Those that call a label for named genes, and whose prediction can therefore be attributed to the gene it is about. A map walk or a module search answers about structure, not about one gene, and keeps its existing self-test instead.
Pool the ledger at one level: "gene", "class", "target" or "strategy".
Every row carries the counts it rests on, the rate over ANSWERED genes, a Wilson 95% interval and
enough, which is False where too few genes were answered for a percentage to mean anything.
Rows are never pooled across settings: a gene called right at one setting and wrong at another is
not half right, it is a gene whose answer depends on the setting, and that is worth seeing.
Every strategy's verdict on one gene: what it said, whether it was right, how sure it was.
One line for the gene card: the condensed form, with the detail a click away.
Hide each named set of genes TOGETHER, and record what the strategy said about its members.
A different question from the fold-by-fold run. There, a gene is hidden while its neighbours keep their labels, so the answer may rest on one close relative; here the whole set goes at once, which asks whether the evidence still reaches those genes when the obvious witnesses are gone too. Hiding a whole compartment is the strongest form: nothing of that class is left to copy from, so a label-caller cannot name it at all, and the honest result is that it abstains or places the genes somewhere else -- which is worth seeing stated rather than assumed.
sets maps a name to gene positions. Returns the same rows as evaluate, with mode "set",
the set's name and its size, so the two can be concatenated and read side by side.
One set per label value: every gene of that class, hidden at once.
The hardest hold-out there is. Nothing of the class remains to copy, so this measures whether a class is recoverable from the evidence itself or only by resemblance to its own members.
Random sets of several sizes: how fast does a strategy lose a gene as its company is hidden?
The degradation curve. One gene hidden alone is the easiest case there is; a hundred hidden with it is the case a real screen presents, where a whole list of genes is unknown at once.
Per held-out set: how many members came back, and whether they stayed together.
together is the share of the set given the SAME label as each other, whatever that label was:
a set that is placed consistently but wrongly has still been recognised as one thing, and that
is a different finding from a set scattered across the map.
Where the built record lives in the package.
The built record, loaded once and kept. Empty when it has not been built here.
Read rather than recomputed: the folds are a one-off build (scripts/build_track_record.py),
so asking what happened to a gene costs a lookup and not a model fit.
The labels worth recording for this table, the default first.
The labels the record holds for one organism, the default first.
The label a view means when it names none: the organism's first declared target.
The gene card's line and its table, as HTML: the condensed form first, the detail under it.
One sentence a reader can take in -- what is known, and how many strategies recovered it when it
was hidden -- then a row per strategy saying what each one actually said. Empty string when the
record says nothing about this gene, so the card simply does not grow a section. mode "alone"
reads the rows alone computed on demand, for a label the shipped record does not cover.
One class: how well each strategy recovers it, and what it is mistaken for.
The level above the gene. A class every strategy misses is not a failure of any one of them -- it is a statement that this distinction is not in the measurements, which is worth knowing before a screen is designed around it.
Every class of one information category on a line: how recoverable each one is.
The level above the class. Each class names its best strategy and that strategy's rate, and links down to the class page; classes no strategy answers often enough to judge are said to be so rather than ranked, and classes no strategy recovers are listed as such -- they are the distinctions the measurements do not carry.
One line for a strategy card: where this strategy is weakest, by class.
Only classes with enough answered genes to judge; a class it answered twice says nothing about it, and saying so would be worse than saying nothing.
Hide YOUR genes together and ask every strategy what they are: would it have found them?
genes is any list of identifiers -- a screen's hits, a complex, a pull-down -- resolved the way
the gene-list strategies resolve them. Each strategy is asked about all of them at once, with
their labels hidden, so the answer says whether the evidence reaches your genes or only reaches
genes whose neighbours are already known. Returns the per-gene rows; set_summary pools them.
rows = track_record.my_list(ctx, ["TGME49_294550", "TGME49_244470"], "compartment")
track_record.set_summary(rows)[["strategy", "right", "answered", "together"]]
False only when the strategy loses to always naming the commonest class on BOTH accuracy and mean per-class recall; None where there is no record to judge from.
A clause for a recommendation: how often this strategy was right with labels hidden.
Only from the shipped record and only for its label: if target is given and is not the label
the record holds, nothing is said rather than quoting a different label's rate as this one's.
What my_list found, condensed first: a line per strategy, then a gene-by-strategy grid.
rows_of maps a gene id to its row in the window's table, so each gene links to its card.
Hide ONE gene, with its orthogroup, and ask each fast strategy what it is.
For any label, not only the shipped one: the shipped record covers each organism's default
label, and this answers the same question for the others on demand. The orthogroup goes too,
exactly as in the folds, so a paralogue cannot give its sibling away. gene is a position or an
identifier. Results are cached per release, organism, label and gene.
T.alone(ctx, "TGME49_294550", "function")