Test before you trust

A number without its chance level is not a result. Every self-test hides known information, asks the strategy for it back, and compares the answer with the SAME procedure run without the information it claims to use. The five patterns are: hide labels, hide members of a set, hide edges, hide values, and replicate findings on the other half of the genes.

In [1]:

from starplast import strategies as St = S.test('physical_partners', target='compartment')t

Out[1]:

PASS -- correct calls per hidden gene 0.520 against 0.069 ± 0.017 under 20 runs on shuffled labels (bar 0.098, p 0.0476); effect +0.451, needed +0.050; 300 hidden

Hidden: 25% of the compartment labels, whole orthogroups at a time. Null: 20 runs on shuffled labels.

Scorecard (label calls)

Accuracy	0.520
Coverage	0.860
Precision of calls	0.605
Macro precision	0.600
Macro recall (balanced accuracy)	0.559
Macro F1	0.533
Weighted F1	0.555
Cohen's kappa	0.470
Matthews correlation (MCC)	0.477
Macro AUROC	0.836
Macro AUPRC	0.428

In [2]:

print('observed', round(t.observed, 3), '| chance', round(t.null_mean, 3),      '+/-', round(t.null_sd, 3), '| bar', round(t.null_high, 3),      '| effect', round(t.effect, 3), '| p', round(t.p_value, 4))

Out[2]:

observed 0.52 | chance 0.069 +/- 0.017 | bar 0.098 | effect 0.451 | p 0.0476

In [3]:

t.details

Out[3]:

class	hidden	recovered	recall	called	precision
19S proteasome	4	4	1	5	0.8
40S ribosome	7	2	0.286	4	0.5
60S ribosome	5	5	1	16	0.312
ER	8	4	0.5	4	1
ER 2	1	1	1	7	0.143
Golgi	4	2	0.5	2	1
IMC	12	8	0.667	11	0.727
PM - integral	5	1	0.2	2	0.5
PM - peripheral 1	2	2	1	4	0.5
apical 1	7	2	0.286	4	0.5
apical 2	3	1	0.333	3	0.333
apicoplast	6	2	0.333	3	0.667

22 rows; first 12 shown

The scorecard: in what way a strategy is good

The verdict rests on one number. The scorecard reports, on the same hidden genes, every standard metric for the kind of task the strategy performs -- for label calls: accuracy, coverage, precision of calls, macro precision and recall, macro and weighted F1, Cohen's kappa, MCC, macro AUROC and AUPRC -- always in the same order, so strategies doing the same task can be compared number by number.

In [4]:

t.card()

Out[4]:

section	metric	key	value	reading
verdict	Verdict	verdict	PASS	PASS: above the 95th percentile of the null AND by the stated margin. FAIL otherwise. INCONCLUSIVE when too little could be hidden to score.
verdict	Judged on	metric	correct calls per hidden gene	The one metric the verdict rests on, chosen per strategy as the honest test of its claim.
verdict	Observed	observed	0.52	That metric on the hidden genes.
verdict	Chance	chance	0.0695	The same metric for the same procedure on shuffled labels, random sets or permuted identities -- measured, not assumed.
verdict	Bar	bar	0.0975	The 95th percentile of the null runs: what luck reaches one time in twenty.
verdict	p	p_value	0.0476	Share of null runs at least as good as the observed, (1 + k) / (1 + runs).
verdict	Skill	skill	0.484	(observed - chance) / (1 - chance): 0 is chance, 1 is perfect, negative is worse than chance. Puts every metric on one scale.
verdict	Hidden	n_hidden	300	How many genes, pairs or findings the test scored.
label calls	Accuracy	accuracy	0.52	The headline for label calls. Counting abstentions as errors stops a strategy looking accurate by calling only the easy genes; coverage and precision of calls split it apart.
label calls	Coverage	coverage	0.86	How far the strategy reaches. A network strategy cannot call a gene with no edges; a low coverage with a high precision of calls is a precise but narrow tool.
label calls	Precision of calls	precision_of_calls	0.605	How far to trust one call. Equals accuracy when coverage is 1.
label calls	Macro precision	macro_precision	0.6	Whether calls of the RARE classes can be trusted too; a strategy that only ever calls the commonest class scores low.

19 rows; first 12 shown

In [5]:

S.metrics('label calls')[['metric', 'chance', 'reading']]

Out[5]:

metric	chance	reading
Accuracy	about the sum of squared class shares (Cohen's chance term)	The headline for label calls. Counting abstentions as errors stops a strategy looking accurate by calling only the easy genes; coverage and precision of calls split it apart.
Coverage	not applicable (a property of the strategy, not of luck)	How far the strategy reaches. A network strategy cannot call a gene with no edges; a low coverage with a high precision of calls is a precise but narrow tool.
Precision of calls	as accuracy, among the genes called	How far to trust one call. Equals accuracy when coverage is 1.
Macro precision	about the average class share	Whether calls of the RARE classes can be trusted too; a strategy that only ever calls the commonest class scores low.
Macro recall (balanced accuracy)	1 / number of classes, for a caller that ignores the data	Whether every class is found, not just the large ones. Compare with accuracy: a large gap means the strategy lives on the big classes.
Macro F1	low; roughly the average class share	One number balancing finding each class and being right when calling it, with rare classes counted as much as common ones.
Weighted F1	about the sum of squared class shares	Macro F1's counterpart that follows the class sizes; close to accuracy when coverage is high.
Cohen's kappa	0	Accuracy with chance removed: 0 is no better than matching class frequencies, 1 is perfect. Comparable across labels with different numbers and sizes of classes.
Matthews correlation (MCC)	0	A correlation between the call and the truth that stays honest under strong class imbalance; often the single most informative number for an unbalanced label.
Macro AUROC	0.5	How well the strategy's scores separate each class from the rest, before any threshold is chosen. Missing for strategies that call labels without scoring every class.
Macro AUPRC	the class's share among hidden genes, averaged	Like macro AUROC but dominated by the top of each ranking, which is where calls are made; far below AUROC means good separation overall but a noisy top.

In [6]:

import pandas as pdrows = []for key in ('feature_knn', 'supervised_classifier', 'graph_convolution', 'random_forest'):    card = S.test(key, target='compartment').scorecard    rows.append({'strategy': S.get(key).name, **{m: round(card[m], 3) for m in ('accuracy', 'macro_f1', 'kappa', 'mcc', 'macro_auroc')}})pd.DataFrame(rows)

Out[6]:

strategy	accuracy	macro_f1	kappa	mcc	macro_auroc
Call a gene by the genes that behave like it (kNN)	0.331	0.276	0.267	0.284	0.822
Train a classifier on the known genes and call the rest (logistic regression)	0.461	0.417	0.414	0.416	0.919
Smooth the measurements along the networks, then classify (graph convolution + logistic regression)	0.491	0.431	0.445	0.446	0.923
Let a random forest find what defines a label (random forest + permutation importance)	0.497	0.435	0.433	0.438	0.911

The same test on a table with nothing in it

A test that passes on noise tests nothing. planted_context(null=True) deals every column and edge out at random.

In [7]:

noise = S.planted_context(null=True)S.get('physical_partners').test(noise)

Out[7]:

FAIL -- correct calls per hidden gene 0.125 against 0.147 ± 0.046 under 20 runs on shuffled labels (bar 0.227, p 0.667); effect -0.022, needed +0.050; 88 hidden

Hidden: 25% of the compartment labels, whole orthogroups at a time. Null: 20 runs on shuffled labels.

Scorecard (label calls)

Accuracy	0.125
Coverage	0.875
Precision of calls	0.143
Macro precision	0.144
Macro recall (balanced accuracy)	0.110
Macro F1	0.124
Weighted F1	0.140
Cohen's kappa	-0.033
Matthews correlation (MCC)	-0.033
Macro AUROC	0.493
Macro AUPRC	0.202

Calibration: where each strategy works

scripts/calibrate_strategies.py runs every strategy's self-test over a grid of its settings, several held-out labels and five seeds, and summarises each configuration with a mean and a 95% interval. Skill puts every metric on one scale: 0 is chance, 1 is perfect.

In [8]:

cal = S.overview('Tg')cal[['number', 'name', 'grade', 'skill_default', 'skill_tuned']].head(12)

Out[8]:

number	name	grade	skill_default	skill_tuned
1	Hold out a category and search for a map that finds it (UMAP + HDBSCAN)	weak	0.0711	0.0701
2	Find the map where your gene list is one cluster (UMAP + HDBSCAN)	weak	0.0251	0.0283
3	Ask which categories the data can rediscover (UMAP + neighbour AUROC)	reliable	0.499	0.502
4	Keep only the modules that survive the whole walk (UMAP + HDBSCAN co-clustering)	weak	0.073	0.0451
5	Tune a map without labels, then read what it encodes (UMAP + HDBSCAN, chi-square / Kruskal-Wallis)	reliable	0.949	0.965
6	Find which kind of evidence carries a label (kNN ablation)	reliable	0.14	0.159
7	Call a gene by the genes that behave like it (kNN)	reliable	0.225	0.292
8	Call a gene by its neighbours on the map (UMAP + kNN)	weak	0.125	0.114
9	Name a cluster by the label it is enriched for (UMAP + HDBSCAN, hypergeometric)	reliable	0.332	0.409
10	Find genes whose label their neighbours contradict (kNN + network neighbours)	reliable	0.638	0.65
11	Diffuse a label across one measured network (random walk with restart)	reliable	0.125	0.136
12	Let every network vote, weighted by what it has earned (chance-weighted ensemble vote)	weak	0.3	0.246

In [9]:

S.calibration('physical_partners')['tuned']

Out[9]:

{'setting': {}, 'runs': 10, 'conclusive': 10, 'skill': 0.2566, 'skill_low': 0.0755, 'skill_high': 0.426, 'pass_rate': 0.6, 'pass_low': 0.3127, 'pass_high': 0.8318, 'observed': 0.3896, 'chance': 0.1885, 'targets': ['compartment', 'dtm_class', 'lopit_unified', 'screen_any_phenotype', 'stage_enriched_derived'], 'not_run': 0, 'chosen_on_seeds': [1, 2, 3], 'reported_on_seeds': [4, 5], 'scorecard': {'accuracy': {'mean': 0.3896, 'low': 0.1678, 'high': 0.5907, 'runs': 10}, 'coverage': {'mean': 0.5563, 'low': 0.2732, 'high': 0.8352, 'runs': 10}, 'precision_of_calls': {'mean': 0.6802, 'low': 0.5868, 'high': 0.7685, 'runs': 10}, 'macro_precision': {'mean': 0.4852, 'low': 0.343, 'high': 0.6222, 'runs': 10}, 'macro_recall': {'mean': 0.3272, 'low': 0.1332, 'high': 0.4963, 'runs': 10}, 'macro_f1': {'mean': 0.3585, 'low': 0.1713, 'high': 0.5208, 'runs': 10}, 'weighted_f1': {'mean': 0.4387, 'low': 0.2175, 'high': 0.6392, 'runs': 10}, 'kappa': {'mean': 0.247, 'low': 0.0603, 'high': 0.4078, 'runs': 10}, 'mcc': {'mean': 0.2662, 'low': 0.082, 'high': 0.4264, 'runs': 10}, 'macro_auroc': {'mean': 0.7229, 'low': 0.5984, 'high': 0.8231, 'runs': 10}, 'macro_auprc': {'mean': 0.4378, 'low': 0.3681, 'high': 0.5111, 'runs': 10}}}

In [10]:

S.tuned('triangulation')

Out[10]:

{'min_agree': 3}

Self-tests, nulls and calibration, from Python

Test before you trust

The scorecard: in what way a strategy is good

The same test on a table with nothing in it

Calibration: where each strategy works