Edit on GitHub

starplast.scorecard

Scorecards: the same metrics, in the same order, for every strategy that does the same kind of task.

A self-test's verdict rests on one number judged against its null. That number is chosen to be the honest one for the strategy, but on its own it cannot say how a strategy is good or bad: whether it is precise but reaches few genes, whether it finds the common classes and misses the rare ones, whether a ranking is good at the top or only on average. The scorecard adds that, and standardises it: every strategy states which of six tasks it performs, and every strategy performing a task reports that task's metrics in the same order, computed the same way on the same hidden genes.

The six tasks, and what is hidden in each:

label calls -- a share of a label is hidden and called back (localization, stage, ...);
ranking -- hidden positives (set members, edges, corrupted labels) are ranked among negatives;
set retrieval -- a set of genes is returned and compared with the hidden members;
cluster recovery -- clusters chosen on visible labels are scored on hidden ones;
values -- hidden measurements (fitness, abundance) are predicted;
replication -- findings made on half the genes are checked on the other half.

Every metric is defined once, in :data:METRICS, with its range, its chance level and how to read it; :func:glossary returns that as a table, and the application shows it wherever a metric appears. Metrics that need something a strategy does not produce -- per-class scores for a macro AUROC, say -- are reported as missing rather than guessed, and the glossary says why.

NAN = nan

@dataclass(frozen=True)

class Metric:

One metric: what it measures, its range, what chance gives, and how to read it.

Metric( key: str, label: str, task: str, definition: str, range: str, chance: str, reading: str, higher_is_better: bool = True)

key: str

label: str

task: str

definition: str

range: str

chance: str

reading: str

higher_is_better: bool = True

@dataclass(frozen=True)

class Task:

A kind of inference, with the metrics every strategy performing it reports, in order.

Task(key: str, description: str, hidden: str, metrics: tuple)

key: str

description: str

hidden: str

metrics: tuple

T_LABEL = 'label calls'

T_RANK = 'ranking'

T_SET = 'set retrieval'

T_CLUSTER = 'cluster recovery'

T_VALUES = 'values'

T_REPL = 'replication'

METRICS: dict = {'accuracy': Metric(key='accuracy', label='Accuracy', task='label calls', definition='Hidden genes called with their true label, divided by ALL hidden genes. A gene the strategy declined to call counts as wrong.', range='0 to 1', chance="about the sum of squared class shares (Cohen's chance term)", reading='The headline for label calls. Counting abstentions as errors stops a strategy looking accurate by calling only the easy genes; coverage and precision of calls split it apart.', higher_is_better=True), 'coverage': Metric(key='coverage', label='Coverage', task='label calls', definition='Share of hidden genes that received any call.', range='0 to 1', chance='not applicable (a property of the strategy, not of luck)', reading='How far the strategy reaches. A network strategy cannot call a gene with no edges; a low coverage with a high precision of calls is a precise but narrow tool.', higher_is_better=True), 'precision_of_calls': Metric(key='precision_of_calls', label='Precision of calls', task='label calls', definition='Correct calls divided by calls made (also called selective accuracy).', range='0 to 1', chance='as accuracy, among the genes called', reading='How far to trust one call. Equals accuracy when coverage is 1.', higher_is_better=True), 'macro_precision': Metric(key='macro_precision', label='Macro precision', task='label calls', definition='For each true class: of the genes called that class, the share that truly are; averaged over classes with equal weight. A class never called scores 0.', range='0 to 1', chance='about the average class share', reading='Whether calls of the RARE classes can be trusted too; a strategy that only ever calls the commonest class scores low.', higher_is_better=True), 'macro_recall': Metric(key='macro_recall', label='Macro recall (balanced accuracy)', task='label calls', definition='For each true class: the share of its hidden genes called correctly; averaged over classes with equal weight. Identical to balanced accuracy.', range='0 to 1', chance='1 / number of classes, for a caller that ignores the data', reading='Whether every class is found, not just the large ones. Compare with accuracy: a large gap means the strategy lives on the big classes.', higher_is_better=True), 'macro_f1': Metric(key='macro_f1', label='Macro F1', task='label calls', definition='Per class, the harmonic mean of precision and recall; averaged over classes with equal weight.', range='0 to 1', chance='low; roughly the average class share', reading='One number balancing finding each class and being right when calling it, with rare classes counted as much as common ones.', higher_is_better=True), 'weighted_f1': Metric(key='weighted_f1', label='Weighted F1', task='label calls', definition='Per-class F1 averaged with each class weighted by its number of hidden genes.', range='0 to 1', chance='about the sum of squared class shares', reading="Macro F1's counterpart that follows the class sizes; close to accuracy when coverage is high.", higher_is_better=True), 'kappa': Metric(key='kappa', label="Cohen's kappa", task='label calls', definition='Agreement between calls and truth corrected for the agreement their class frequencies alone would give: (observed - expected) / (1 - expected). No call is its own category.', range='-1 to 1', chance='0', reading='Accuracy with chance removed: 0 is no better than matching class frequencies, 1 is perfect. Comparable across labels with different numbers and sizes of classes.', higher_is_better=True), 'mcc': Metric(key='mcc', label='Matthews correlation (MCC)', task='label calls', definition="The multiclass Matthews correlation coefficient (Gorodkin's R_K) between calls and truth, with no call as its own category.", range='-1 to 1', chance='0', reading='A correlation between the call and the truth that stays honest under strong class imbalance; often the single most informative number for an unbalanced label.', higher_is_better=True), 'macro_auroc': Metric(key='macro_auroc', label='Macro AUROC', task='label calls', definition='For each class, the probability that a random hidden gene of that class gets a higher score for the class than a random hidden gene of another class; averaged over classes. Needs per-class scores.', range='0 to 1', chance='0.5', reading="How well the strategy's scores separate each class from the rest, before any threshold is chosen. Missing for strategies that call labels without scoring every class.", higher_is_better=True), 'macro_auprc': Metric(key='macro_auprc', label='Macro AUPRC', task='label calls', definition='For each class, the area under the precision-recall curve of its one-vs-rest scores (average precision); averaged over classes. Needs per-class scores.', range='0 to 1', chance="the class's share among hidden genes, averaged", reading='Like macro AUROC but dominated by the top of each ranking, which is where calls are made; far below AUROC means good separation overall but a noisy top.', higher_is_better=True), 'auroc': Metric(key='auroc', label='AUROC', task='ranking', definition='Probability that a random hidden positive is ranked above a random negative (area under the ROC curve); ties count half.', range='0 to 1', chance='0.5', reading='Separation over the whole ranking. Insensitive to how rare positives are, so a high AUROC can coexist with a poor top of the list -- read it with AUPRC.', higher_is_better=True), 'auprc': Metric(key='auprc', label='AUPRC', task='ranking', definition='Average precision: the mean, over the hidden positives, of the precision of the ranking down to that positive (area under the precision-recall curve).', range='0 to 1', chance='the prevalence (share of positives among everything ranked)', reading='How clean the top of the ranking is. Its chance level is the prevalence, so compare it with that (AUPRC lift), never with 0.5.', higher_is_better=True), 'auprc_lift': Metric(key='auprc_lift', label='AUPRC lift', task='ranking', definition='AUPRC divided by the prevalence, its value for a random ordering.', range='0 to 1/prevalence', chance='1', reading='How many times better than a random ordering the ranking is, where it matters. Comparable across tests with different prevalences, unlike AUPRC itself.', higher_is_better=True), 'prevalence': Metric(key='prevalence', label='Prevalence', task='ranking', definition='Hidden positives divided by everything ranked.', range='0 to 1', chance='not applicable (a property of the test)', reading='The chance level of AUPRC and of precision at any depth; context for every other number.', higher_is_better=True), 'pauroc_10': Metric(key='pauroc_10', label='Partial AUROC (FPR <= 10%)', task='ranking', definition='AUROC restricted to false-positive rates up to 10%, rescaled (McClish) so 0.5 is chance and 1 perfect.', range='0.5 to 1 (after rescaling)', chance='0.5', reading='Separation in the part of the ranking anyone would act on; what matters for screening.', higher_is_better=True), 'r_precision': Metric(key='r_precision', label='R-precision', task='ranking', definition='Precision among the top R items, where R is the number of hidden positives. Equal to recall at that depth.', range='0 to 1', chance='the prevalence', reading='If you took as many candidates as there are true positives, the share that would be right.', higher_is_better=True), 'precision_at_1pct': Metric(key='precision_at_1pct', label='Precision @ top 1%', task='ranking', definition='Share of positives among the top 1% of the ranking (at least one item).', range='0 to 1', chance='the prevalence', reading='How good the very first candidates are -- the list anyone would test first.', higher_is_better=True), 'enrichment_at_1pct': Metric(key='enrichment_at_1pct', label='Enrichment @ top 1%', task='ranking', definition='Precision at the top 1% divided by the prevalence.', range='0 to 1/prevalence', chance='1', reading='How many times more positives the first 1% holds than a random 1% would.', higher_is_better=True), 'recall_at_10pct': Metric(key='recall_at_10pct', label='Recall @ top 10%', task='ranking', definition='Share of all hidden positives found in the top 10% of the ranking.', range='0 to 1', chance='0.1', reading='How much of the answer a short list contains.', higher_is_better=True), 'max_f1': Metric(key='max_f1', label='Best F1', task='ranking', definition='The highest F1 (harmonic mean of precision and recall) over every cut-off of the ranking.', range='0 to 1', chance='about 2 x prevalence / (1 + prevalence) (taking everything)', reading='The best single-threshold trade-off the ranking allows; optimistic, since the cut-off is chosen on the answer.', higher_is_better=True), 'ndcg': Metric(key='ndcg', label='nDCG', task='ranking', definition='Normalised discounted cumulative gain: positives count 1 / log2(rank + 1), divided by the value of a perfect ranking.', range='0 to 1', chance='depends on prevalence; low when positives are rare', reading='A whole-ranking score that weights the top most, smoothly rather than at one cut-off.', higher_is_better=True), 'precision': Metric(key='precision', label='Precision', task='set retrieval', definition='Share of the returned genes that are hidden members.', range='0 to 1', chance="the members' share of the candidate genes", reading='How many of the candidates are real.', higher_is_better=True), 'recall': Metric(key='recall', label='Recall', task='set retrieval', definition='Share of the hidden members that were returned.', range='0 to 1', chance='the returned share of the candidate genes', reading='How much of the set was found.', higher_is_better=True), 'f1': Metric(key='f1', label='F1', task='set retrieval', definition='Harmonic mean of precision and recall.', range='0 to 1', chance="low; about the members' share for a random return of the same size", reading='One number that is only high when both are.', higher_is_better=True), 'jaccard': Metric(key='jaccard', label='Jaccard index', task='set retrieval', definition='Overlap of returned and hidden sets divided by their union.', range='0 to 1', chance='near 0', reading='How close the returned set is to being exactly the hidden set; stricter than F1.', higher_is_better=True), 'set_mcc': Metric(key='set_mcc', label='Matthews correlation (MCC)', task='set retrieval', definition="Correlation between 'returned' and 'is a hidden member' over all candidate genes.", range='-1 to 1', chance='0', reading='Accounts for the genes correctly NOT returned as well; honest when the set is small.', higher_is_better=True), 'fold_enrichment': Metric(key='fold_enrichment', label='Fold enrichment', task='set retrieval', definition="Precision divided by the members' share of the candidate genes.", range='0 to 1/share', chance='1', reading='How many times more members the returned set holds than a random set of its size.', higher_is_better=True), 'returned': Metric(key='returned', label='Genes returned', task='set retrieval', definition='How many genes the strategy returned as the set.', range='0 to all', chance='not applicable', reading='Context: a high recall from returning half the genome is not a finding.', higher_is_better=True), 'weighted_f1_clusters': Metric(key='weighted_f1_clusters', label='Weighted F1', task='cluster recovery', definition='For each label, F1 of its hidden genes against the ONE cluster chosen for it on visible genes; averaged with labels weighted by their hidden genes. A label with no cluster scores 0.', range='0 to 1', chance="set by permuting hidden labels over the same clusters (the verdict's null)", reading='Whether a label that nobody showed the clustering falls out as a cluster, scored on genes the choice never saw.', higher_is_better=True), 'weighted_precision_clusters': Metric(key='weighted_precision_clusters', label='Weighted precision', task='cluster recovery', definition='For each label, the share of hidden genes in its chosen cluster that carry it; label-size weighted.', range='0 to 1', chance="about the label's share", reading='How pure the chosen clusters are.', higher_is_better=True), 'weighted_recall_clusters': Metric(key='weighted_recall_clusters', label='Weighted recall', task='cluster recovery', definition='For each label, the share of its hidden genes that landed in its chosen cluster; label-size weighted.', range='0 to 1', chance="about the cluster's share of the genes", reading='How completely each label is gathered into one cluster.', higher_is_better=True), 'ari': Metric(key='ari', label='Adjusted Rand index (ARI)', task='cluster recovery', definition='Agreement between the clustering and the hidden labels over all pairs of hidden genes, corrected so random partitions of the same sizes score 0.', range='-0.5 to 1', chance='0', reading='Whole-partition agreement, independent of which cluster was chosen for which label.', higher_is_better=True), 'nmi': Metric(key='nmi', label='Normalised mutual information (NMI)', task='cluster recovery', definition='Mutual information between clusters and hidden labels, divided by the mean of their entropies.', range='0 to 1', chance='small but above 0 for many small clusters', reading="How much knowing a gene's cluster tells you about its label.", higher_is_better=True), 'homogeneity': Metric(key='homogeneity', label='Homogeneity', task='cluster recovery', definition="1 minus the uncertainty about the label left once the cluster is known, relative to the label's own entropy.", range='0 to 1', chance='near 0', reading='Whether each cluster holds one label.', higher_is_better=True), 'completeness': Metric(key='completeness', label='Completeness', task='cluster recovery', definition="1 minus the uncertainty about the cluster left once the label is known, relative to the clustering's entropy.", range='0 to 1', chance='near 0', reading='Whether each label sits in one cluster. Its harmonic mean with homogeneity is the V-measure, which equals NMI.', higher_is_better=True), 'noise_share': Metric(key='noise_share', label='Unclustered share', task='cluster recovery', definition='Share of hidden genes HDBSCAN left as noise (in no cluster).', range='0 to 1', chance='not applicable', reading='Context: noise genes cannot be recovered by any cluster.', higher_is_better=False), 'spearman': Metric(key='spearman', label='Spearman rho', task='values', definition='Rank correlation between predicted and hidden measured values.', range='-1 to 1', chance='0 (standard deviation 1/sqrt(n - 1))', reading='Whether the ORDER of the genes is predicted, whatever the scale. The headline for values, robust to outliers and to a prediction on a different scale.', higher_is_better=True), 'pearson': Metric(key='pearson', label='Pearson r', task='values', definition='Linear correlation between predicted and hidden values.', range='-1 to 1', chance='0', reading='Like Spearman but on the values themselves, so a few extreme genes can dominate it.', higher_is_better=True), 'kendall': Metric(key='kendall', label='Kendall tau-b', task='values', definition='Share of gene pairs ordered the same way by prediction and truth, minus the share ordered oppositely, corrected for ties.', range='-1 to 1', chance='0', reading='The most directly interpretable rank agreement: (1 + tau) / 2 is the chance a random pair is ordered correctly.', higher_is_better=True), 'r2': Metric(key='r2', label='R-squared (out of sample)', task='values', definition='1 minus the squared prediction error over the variance of the hidden values around their own mean.', range='below 0 to 1', chance='0 or below (predicting the mean scores 0)', reading="The share of the hidden values' variance the prediction explains. Negative when the prediction is further off than the mean -- common for a good ranking on the wrong scale.", higher_is_better=True), 'nrmse': Metric(key='nrmse', label='Normalised RMSE', task='values', definition='Root-mean-square error divided by the standard deviation of the hidden values.', range='0 upward', chance='1 (predicting the mean)', reading="Typical error in units of the measurement's own spread: below 1 beats the mean.", higher_is_better=False), 'mae': Metric(key='mae', label='Mean absolute error', task='values', definition="Mean absolute difference between predicted and hidden values, in the measurement's units.", range='0 upward', chance='the mean absolute deviation of the hidden values', reading='Typical size of an error, in the same units as the data.', higher_is_better=False), 'top_decile_recall': Metric(key='top_decile_recall', label='Top-decile recall', task='values', definition='Share of the genes in the true top 10% of hidden values that are also in the predicted top 10%.', range='0 to 1', chance='0.1', reading='Whether the extreme genes -- usually the interesting ones -- are predicted extreme.', higher_is_better=True), 'bottom_decile_recall': Metric(key='bottom_decile_recall', label='Bottom-decile recall', task='values', definition='The same for the bottom 10% -- for fitness scores, the most essential genes.', range='0 to 1', chance='0.1', reading='Whether the genes at the other extreme are found.', higher_is_better=True), 'value_coverage': Metric(key='value_coverage', label='Coverage', task='values', definition='Share of hidden values that received a prediction.', range='0 to 1', chance='not applicable', reading='Every other value metric is computed on these genes.', higher_is_better=True), 'replication_rate': Metric(key='replication_rate', label='Replication rate', task='replication', definition='Share of the findings made on one half of the genes that hold on the other half.', range='0 to 1', chance="the rate with the second half's evidence scrambled", reading='Whether the findings are properties of the genes or of the sample they were found in.', higher_is_better=True), 'findings': Metric(key='findings', label='Findings made', task='replication', definition='Number of findings made on the first half.', range='0 upward', chance='not applicable', reading='Context: how much there was to replicate.', higher_is_better=True), 'replicated': Metric(key='replicated', label='Findings replicated', task='replication', definition='Number of those findings that held on the second half.', range='0 to findings', chance='findings x the null rate', reading='The findings worth reading first; each is a claim that held twice.', higher_is_better=True), 'null_rate': Metric(key='null_rate', label='Replication by chance', task='replication', definition="The replication rate with the second half's evidence scrambled, averaged over runs.", range='0 to 1', chance='is itself the chance level', reading='What replication looks like for findings with nothing behind them.', higher_is_better=False), 'replication_lift': Metric(key='replication_lift', label='Replication lift', task='replication', definition='Replication rate divided by the replication rate by chance.', range='0 upward', chance='1', reading='How many times more often real findings replicate than chance ones.', higher_is_better=True)}

TASKS: dict = {'label calls': Task(key='label calls', description='Call a label for genes that lack it.', hidden="a share of a label's genes, whole orthogroups at a time", metrics=('accuracy', 'coverage', 'precision_of_calls', 'macro_precision', 'macro_recall', 'macro_f1', 'weighted_f1', 'kappa', 'mcc', 'macro_auroc', 'macro_auprc')), 'ranking': Task(key='ranking', description='Rank candidates so the true ones come first.', hidden='set members, edges or corrupted labels, ranked among negatives', metrics=('auroc', 'auprc', 'auprc_lift', 'prevalence', 'pauroc_10', 'r_precision', 'precision_at_1pct', 'enrichment_at_1pct', 'recall_at_10pct', 'max_f1', 'ndcg')), 'set retrieval': Task(key='set retrieval', description='Return a set of genes that belong with a query.', hidden='part of a gene set, to be returned among all other genes', metrics=('precision', 'recall', 'f1', 'jaccard', 'set_mcc', 'fold_enrichment', 'returned')), 'cluster recovery': Task(key='cluster recovery', description='Find clusters that correspond to a label nobody showed them.', hidden='a share of a label, scored against clusters chosen on the rest', metrics=('weighted_f1_clusters', 'weighted_precision_clusters', 'weighted_recall_clusters', 'ari', 'nmi', 'homogeneity', 'completeness', 'noise_share')), 'values': Task(key='values', description='Predict a measured value for genes without one.', hidden="a share of a measurement's values", metrics=('spearman', 'pearson', 'kendall', 'r2', 'nrmse', 'mae', 'top_decile_recall', 'bottom_decile_recall', 'value_coverage')), 'replication': Task(key='replication', description='Make findings that hold beyond the genes they were made on.', hidden='half of the genes, on which first-half findings are checked', metrics=('replication_rate', 'findings', 'replicated', 'null_rate', 'replication_lift'))}

VERDICT = (('verdict', 'Verdict', 'PASS: above the 95th percentile of the null AND by the stated margin. FAIL otherwise. INCONCLUSIVE when too little could be hidden to score.'), ('metric', 'Judged on', 'The one metric the verdict rests on, chosen per strategy as the honest test of its claim.'), ('observed', 'Observed', 'That metric on the hidden genes.'), ('chance', 'Chance', 'The same metric for the same procedure on shuffled labels, random sets or permuted identities -- measured, not assumed.'), ('bar', 'Bar', 'The 95th percentile of the null runs: what luck reaches one time in twenty.'), ('p_value', 'p', 'Share of null runs at least as good as the observed, (1 + k) / (1 + runs).'), ('skill', 'Skill', '(observed - chance) / (1 - chance): 0 is chance, 1 is perfect, negative is worse than chance. Puts every metric on one scale.'), ('n_hidden', 'Hidden', 'How many genes, pairs or findings the test scored.'))

def glossary(task: str | None = None) -> pandas.DataFrame:

Every metric (or one task's), in scorecard order, with its definition and how to read it.

def explain(key: str) -> str:

One metric's definition, range, chance level and reading, as a paragraph.

def frame(card: dict, task: str) -> pandas.DataFrame:

A scorecard as a table in the task's order: metric, value, and its one-line reading.

def ranking(score, positive) -> dict:

The ranking scorecard for score (higher = more likely positive) against positive.

Genes or pairs with a non-finite score are ranked last, tied: a strategy that cannot score an item has not ranked it high.

def label_calls( pred, truth, positions, class_scores: pandas.DataFrame | None = None) -> dict:

The label-call scorecard on positions: pred may abstain (NaN); truth must be labelled.

class_scores (genes x classes, higher = more likely) adds macro AUROC and AUPRC; without it those two are missing, not estimated.

def set_retrieval(returned, members, universe) -> dict:

Returned genes against hidden members, over a candidate universe (both subsets of it).

def cluster_recovery( cluster_of_hidden, truth_of_hidden, chosen: dict, noise: int = -1) -> dict:

Clusters (one per hidden gene) against hidden labels, with each label's chosen cluster.

def values(pred, truth) -> dict:

Predicted against hidden measured values; NaN predictions lower coverage, nothing else.

def replication(replicated: int, findings: int, null_rates) -> dict:

Findings made on one half, checked on the other, against the rate with the evidence scrambled.

@dataclass(frozen=True)

class Headline:

One bar of the headline card: which number, in what words, on what scale, against what.

scale is how the bar is drawn: unit is 0 to 1 (a negative value draws empty and says so), lift is logarithmic from 0.5 to 32 with 1 at chance, count is logarithmic from 1 to 1,000. chance is a fixed chance level; chance_key names the card metric that holds a measured one (replication's scrambled-evidence rate). invert shows 1 minus the metric (the unclustered share becomes the clustered share). note says why an analogue stands in.

Headline( key: str, label: str, technical: str, reading: str, scale: str = 'unit', chance: float | None = None, chance_key: str | None = None, invert: bool = False, note: str = '')

key: str

label: str

technical: str

reading: str

scale: str = 'unit'

chance: float | None = None

chance_key: str | None = None

invert: bool = False

note: str = ''

SKILL = Headline(key='skill', label='Better than chance', technical='skill', reading='0 is what the same procedure scores on shuffled data, 1 is perfect: how far the strategy got from luck toward the right answer.', scale='unit', chance=0.0, chance_key=None, invert=False, note='')

REACH = {'label calls': Headline(key='coverage', label='Reach', technical='coverage', reading='Share of the hidden genes it made any call for. A gene it cannot call is counted wrong under Right calls.', scale='unit', chance=None, chance_key=None, invert=False, note=''), 'values': Headline(key='value_coverage', label='Reach', technical='coverage', reading='Share of the hidden values it made a prediction for.', scale='unit', chance=None, chance_key=None, invert=False, note=''), 'cluster recovery': Headline(key='noise_share', label='Reach', technical='1 - unclustered share', reading='Share of the hidden genes placed in any cluster; a gene left as noise cannot be recovered by any cluster.', scale='unit', chance=None, chance_key=None, invert=True, note='Clustering has no coverage; its analogue is the share of genes it placed in a cluster at all.'), 'ranking': Headline(key='recall_at_10pct', label='Reach', technical='recall @ top 10%', reading='Share of the true ones that land in the top 10% of the list: how much of the answer a short list holds.', scale='unit', chance=0.1, chance_key=None, invert=False, note='A ranking scores every candidate, so coverage is always complete; its analogue is how many of the true ones a short list reaches.'), 'set retrieval': Headline(key='returned', label='Reach', technical='genes returned', reading='How many genes it returned as the set. Context, not merit: a longer list finds more members by being longer.', scale='count', chance=None, chance_key=None, invert=False, note='A set strategy speaks about the genes it returns, so its reach is how many it returned.'), 'replication': Headline(key='findings', label='Reach', technical='findings made', reading='How many findings it made on the first half of the genes, each then checked on the second half.', scale='count', chance=None, chance_key=None, invert=False, note='A replication test has no coverage; its reach is how many findings it made to check.')}

HEADLINE = {'label calls': (Headline(key='accuracy', label='Right calls', technical='accuracy', reading='Share of hidden genes given their true label; a gene left uncalled counts as wrong.', scale='unit', chance=None, chance_key=None, invert=False, note=''), Headline(key='macro_f1', label='Fair across classes', technical='macro F1', reading='Whether rare classes are found and called correctly as well as common ones; low when it lives on the big classes.', scale='unit', chance=None, chance_key=None, invert=False, note='')), 'ranking': (Headline(key='auroc', label='True ones ranked first', technical='AUROC', reading='Chance that a true one is ranked above a random other: 0.5 is a coin flip, 1 is every true one on top.', scale='unit', chance=0.5, chance_key=None, invert=False, note=''), Headline(key='auprc_lift', label='Clean top of the list', technical='AUPRC lift', reading='How many times cleaner the top of the list is than a random order; 1 is no better than random.', scale='lift', chance=1.0, chance_key=None, invert=False, note='')), 'values': (Headline(key='spearman', label='Order predicted', technical='Spearman rho', reading='Whether genes predicted high really are high, whatever the scale; 0 is no relation, 1 a perfect order.', scale='unit', chance=0.0, chance_key=None, invert=False, note=''), Headline(key='r2', label='Variance explained', technical='R-squared, out of sample', reading='Share of the spread in the hidden values the prediction explains; 0 is no better than guessing the average.', scale='unit', chance=0.0, chance_key=None, invert=False, note='')), 'cluster recovery': (Headline(key='weighted_f1_clusters', label='Label falls out as a cluster', technical='weighted F1', reading='Whether the genes of a label nobody showed the clustering end up together in one cluster.', scale='unit', chance=None, chance_key=None, invert=False, note=''), Headline(key='ari', label='Partition agreement', technical='adjusted Rand index', reading='Agreement between the whole clustering and the hidden labels; 0 is what random clusters of the same sizes give.', scale='unit', chance=0.0, chance_key=None, invert=False, note='')), 'set retrieval': (Headline(key='precision', label='Returned genes that are real', technical='precision', reading='Share of the genes it returned that are true members of the set.', scale='unit', chance=None, chance_key=None, invert=False, note=''), Headline(key='recall', label='Members found', technical='recall', reading='Share of the hidden members it returned.', scale='unit', chance=None, chance_key=None, invert=False, note='')), 'replication': (Headline(key='replication_rate', label='Findings that hold', technical='replication rate', reading='Share of the findings made on half the genes that hold on the other half.', scale='unit', chance=None, chance_key='null_rate', invert=False, note=''), Headline(key='replication_lift', label='Beyond chance', technical='replication lift', reading='How many times more often its findings hold than findings made on scrambled evidence.', scale='lift', chance=1.0, chance_key=None, invert=False, note=''))}

def headline(task: str) -> tuple:

The four headline bars of a task, in card order: skill, reach, and two task metrics.

SCALES = {'unit': (0.0, 1.0, False), 'lift': (0.5, 32.0, True), 'count': (1.0, 1000.0, True)}

def position(scale: str, value) -> float:

Where value sits along a bar of scale, 0 (left) to 1 (right); NaN for no value.

def fmt_value(scale: str, value) -> str:

A bar's value as text: 0.62, x1.4 for a lift, 63 for a count, -- for none.

def headline_bars(task: str, card: dict, verdict: dict | None = None) -> list:

The four headline bars filled in: one dict per bar, in card order.

card maps metric -> value, (mean, low, high) or {"mean", "low", "high"} (the calibration's form). verdict carries skill, skill_low, skill_high and, where known, the verdict's observed value and measured chance; that chance becomes a bar's tick when the bar shows the very metric the verdict rests on. Each dict has key, label, technical, value, low, high, chance, scale, position, reading (the plain sentence shown on hover) and note.

def bar_reading( h: Headline, value, low=nan, high=nan, chance=nan) -> str:

One bar in plain words: the number, its interval, chance, and what the number means.