Edit on GitHub

starplast.calibration

How good each strategy is, measured: the calibration sweep turned into verdicts.

scripts/calibrate_strategies.py runs every strategy's self-test over a grid of its settings, over several held-out labels ("targets") and five seeds. This module turns those runs into what a person choosing a strategy needs, and what the Strategies tab and the README show:

skill        (observed - chance) / (1 - chance): 0 is the shuffled-data null, 1 is perfect,
             so a correct-call rate, an AUROC, an F1 and a correlation read on one scale.
at defaults  the strategy as it opens, averaged over every target and seed.
tuned        the best setting -- CHOSEN on seeds 1-3 and REPORTED on seeds 4-5. Choosing and
             reporting on the same runs would report the luckiest of many settings; splitting
             by seed keeps the reported number an honest estimate of what the choice buys.
95% CI       a two-stage bootstrap: targets resampled, then runs within each target. Runs on
             one target share its labels and are not independent; treating them as such gives
             intervals several times too narrow.
pass rate    the share of conclusive runs that beat the null's 95th percentile by the
             strategy's minimum effect, with a Wilson interval.

A grade summarises the lot (grade): reliable, works when tuned, weak, no skill, or untestable when too few runs were conclusive to say.

DATA = '/home/runner/work/starplast/starplast/starplast/data'

CALIBRATION = '/home/runner/work/starplast/starplast/starplast/data/strategy_calibration.json'

TARGET_PARAMS = ('target', 'exclude')

CHOOSE_SEEDS = (1, 2, 3)

REPORT_SEEDS = (4, 5)

GRADES = ('reliable', 'works when tuned', 'weak', 'no skill', 'untestable')

def wilson(k: int, n: int, z: float = 1.96) -> tuple:

Wilson score interval for k successes in n.

def cluster_bootstrap(values, clusters, n_boot: int = 2000, seed: int = 0) -> tuple:

(mean, low, high): a two-stage bootstrap -- clusters, then values within each cluster.

def runs_frame(rows) -> pandas.DataFrame:

runs.jsonl records -> one row per run, with skill, target and setting split apart.

def scorecard_cell(g: pandas.DataFrame, n_boot: int = 1000) -> dict:

metric -> {mean, low, high, runs}: each scorecard metric over the conclusive runs of g, with the same two-stage bootstrap over held-out targets as skill. Empty for runs recorded before scorecards existed.

def grade(entry: dict) -> str:

One word for how far to trust a strategy, from its default and tuned calibration.

def summarise(runs: pandas.DataFrame, defaults: dict | None = None) -> dict:

{organism: {strategy: entry}} -- the calibration the application ships.

defaults maps (organism, strategy) -> the default setting's JSON string, so "at defaults" means the strategy as it opens; without it the most common setting stands in.

def write( calibration: dict, path: str = '/home/runner/work/starplast/starplast/starplast/data/strategy_calibration.json', meta: dict | None = None) -> str:

Atomically publish measured strategies, preserving every unswept result and its provenance.

Updates merge by organism and strategy. meta describes the latest publication; provenance[organism][strategy] records the sweep behind each individual result. Existing malformed JSON is an error, never an excuse to discard earlier measurements.

def load( path: str = '/home/runner/work/starplast/starplast/starplast/data/strategy_calibration.json') -> dict:

The shipped calibration, {"meta": ..., "organisms": {org: {key: entry}}}; empty if none.

def entry( key: str, organism: str = 'Tg', path: str = '/home/runner/work/starplast/starplast/starplast/data/strategy_calibration.json') -> dict | None:

What calibration measured for one strategy on one organism, or None if it was not measured.

def tuned_settings( key: str, organism: str = 'Tg', path: str = '/home/runner/work/starplast/starplast/starplast/data/strategy_calibration.json') -> dict:

The setting calibration found best, without the held-out target (the user picks that).

def setting_text(setting: dict) -> str:

A setting as text a reader cannot misparse: a grid value is bracketed, parameters are split by semicolons. min_cluster_size=20, 50, n_neighbors=10, 30 read as four parameters.

def sentence( key: str, organism: str = 'Tg', path: str = '/home/runner/work/starplast/starplast/starplast/data/strategy_calibration.json') -> str:

One paragraph of plain text: what calibration measured for this strategy.

def overview( organism: str = 'Tg', path: str = '/home/runner/work/starplast/starplast/starplast/data/strategy_calibration.json') -> pandas.DataFrame:

One row per calibrated strategy -- the README table as a frame.

def markdown_table( organism: str = 'Tg', path: str = '/home/runner/work/starplast/starplast/starplast/data/strategy_calibration.json') -> str:

The README table: every strategy's measured skill on held-out known labels.

def scorecard_markdown( organism: str = 'Tg', path: str = '/home/runner/work/starplast/starplast/starplast/data/strategy_calibration.json', setting: str = 'default', intervals: bool = True) -> str:

One table per scorecard task: every strategy doing that task, its task's metrics in their standard order, as mean [95% CI] over held-out targets and seeds at setting ("default" or "tuned"). Strategies without recorded scorecards are listed with dashes, not left out.