Edit on GitHub

starplast.holdout_cv

Nested, label-locked validation for cluster-to-annotation inference.

Embeddings and clusters are unsupervised, but choosing their hyperparameters by the same labels later reported as validation is still leakage. This module separates those uses: an inner validation split chooses a structure, an untouched outer fold estimates its performance, and a cluster-to-category map is learned only from the training labels. Genes without a sufficiently supported/pure cluster abstain.

ABSTAIN = '__abstain__'
@dataclass(frozen=True)
class ClusterCall:

One cluster's training-only category call and its empirical support.

ClusterCall(category: str, support: int, probability: float)
category: str
support: int
probability: float
def fit_cluster_calls( labels, truth, train_index, *, min_support=5, min_probability=0.55) -> dict[int, ClusterCall]:

Learn cluster-to-category calls from training indices only; noise never receives a call.

def predict_cluster_calls(labels, calls, index) -> numpy.ndarray:

Predict selected indices, using an explicit abstention where the training map has no call.

def score_predictions(truth, predicted, categories) -> tuple[dict, pandas.DataFrame]:

Macro F1 plus per-category precision/recall; abstentions count as false negatives.

def nested_structure_cv( structures: dict[str, numpy.ndarray], truth: pandas.Series, *, folds=5, seed=42, min_support=5, min_probability=0.55, split_labels: list[tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]] | None = None):

Select structures on inner validation labels and score once on each untouched outer fold.

Returns (outer, per_category, validation_scores, splits). splits can be reused with permuted labels so the null repeats the complete selection procedure on identical partitions.

def choose_final_structure(validation_scores: pandas.DataFrame) -> str:

Choose the structure with the best mean inner-fold score, never the outer test score.

def pooled_category_scores(per_category: pandas.DataFrame) -> pandas.DataFrame:

Mean and spread of locked outer-fold category metrics.

def inference_table( labels, truth: pandas.Series, gene_ids, category_cv: pandas.DataFrame, *, min_support=5, min_probability=0.55) -> pandas.DataFrame:

Call unlabeled genes from the final structure, carrying cluster and locked-CV confidence.