starplast.search

One scored configuration of a search: its numbers, its map, and its clustering.

The automated walk's output is a table of (configuration x category) scores, and a table of scores is not a result -- the structure it scored is. A step carries the coordinates and the labels so the configuration can be shown as it finishes, which is the difference between a walk that can be watched and one that reports at the end. Same contract as tuning.WalkStep, plus the clustering, because here the clustering is half of what was scored.

Which rows are on the Pareto frontier of two objectives: nothing beats them on both.

The task asked for the two targets "offered rather than chosen", and ranked on each. Ranking on each is two sorts; the frontier is the thing neither sort shows -- a configuration that is second on both is often the one to use, and it appears at the top of neither list. Rows not on the frontier are beaten outright by some other row, which is a fact about them worth having.

The columns of a same-quantity family present in columns. An entry starting with "^" is a pattern, for families defined by a whole series rather than a list of names.

Every declared column below a hierarchy branch.

This is the class-level operation the slot catalogue previously promised but did not implement. hierarchy='evidence' omits an assay/data class, 'biology' omits everything about a biological subject, and 'context' omits a system or stage. Dependencies are closed in both directions by :func:excluded_for when a target is supplied.

The catalogue slots a holdout closes over, at the given scope.

Shared by the column guard and the edge guard, because they must ban the SAME slots. Two copies of this selection would eventually disagree, and the way it would show up is a graph method quietly eating the layer whose label it is being scored against.

{edge layer: why} -- the graph layers a method scored against target must not traverse.

The guard this closes did not exist until instruction 47, and the gap was not small. compartment is the most-used holdout in this project AND a 118,712-edge layer built by joining genes that share a compartment. A propagation or graph-learning method handed that layer while holding out that label would recover it perfectly, and every column-based guard would pass, because none of them can see an edge.

The family is ALWAYS closed, whatever scope is asked for, and that was measured rather than assumed. Asked at class scope -- which is what every recipe uses -- this banned NOTHING for compartment, while target_family banned the layer: Tg_shared_compartment sits under molecular relationships > parasite-parasite in the biology tree, because an edge joining two genes IS a relational measurement, while the compartment label sits under cell organization > localization and topology. The class of the target therefore does not contain the edge layer built from the target, and a class-scoped guard would have left the biggest leak in the graph wide open.

For columns this does not arise: measured association, shared provenance and same-quantity all catch the family by other routes. An edge layer has none of those mechanisms -- there is no column to correlate -- so the family membership is the only thing that can catch it, and it is taken unconditionally.

{layer: why} -- the layers computed from any of columns.

The measurement columns a held-out LAYER must take with it: its sources and their experiments.

The inverse of layers_built_from. A correlation layer is its source columns' correlation structure drawn as edges, so a model that hides the layer but keeps the columns in a similarity is handed the layer back. The calibration sweep caught it twice -- held-out co-fitness "predicted" at AUROC 0.99 from the fitness screens it was built from -- and dropping the source columns alone is not enough: translation efficiency is footprints over RNA from the same runs, so every column of the source EXPERIMENTS goes too, the rule the closure applies to a held-out column.

{layer: why} -- the declared family's layers, and every layer built from an excluded column. The edge counterpart of :func:excluded_detail, and what anything that walks a graph against a held-out target should use.

Every edge layer the catalogue declares, which is the vocabulary a recipe may draw from.

{column: why} -- everything a map scored against target must not see, and what caught it.

:func:excluded_for is this with the reasons dropped, so there is one implementation rather than two. That matters more than it sounds: a guard whose explanation is computed separately from its decision can confidently explain a column it did not actually exclude, and the explanation is the part a reader checks.

Five mechanisms, because each is blind to what the others catch and this guard has now leaked three separate ways:

• measured association, for the undeclared copy -- a renamed column, a second inference over the same data. The threshold is stricter than the 0.95 used to flag derived columns after the fact, because here we are choosing what an embedding may see, and a 0.85-associated column leaks nearly as much as an identical one: lopit_mcmc is a second inference over the same experiment at 0.74;
• declared derivation, for the joint function -- a label that is the argmax of three columns shows 0.56-0.66 against each of them separately, so no pairwise statistic can see it;
• shared provenance, for the same experiment's OTHER outputs, which is neither of the above;
• the declared family, the catalogue's own statement of which slots estimate one quantity;
• the same quantity measured another way, which provenance cannot see because the second estimate came from a different experiment -- sometimes a different species.

The target, everything that restates it, and everything the same experiment produced.

The set of :func:excluded_detail. A caller that only has to BAN columns wants this; one that has to say what it banned and why -- a recipe reporting its closure before it builds anything -- wants the detail.

How well does the clustering recover truth? Per-label precision, recall and F1.

For each label the best single cluster is taken -- the question is whether the structure isolates the label somewhere, not whether the clustering happens to use the same number of groups.

Walk combinations of datasets and hyperparameters, scoring recovery of a held-out target.

objective selects what "good structure" means, as the keyword arguments objectives.score takes -- which of precision, recall or both, averaged or best-case, and over every category or one. Omitted, the per-label F1 summary is used, which is what this always did.

Whichever is chosen, the per-label table is computed regardless and returned alongside, because no single number survives contact with a real result: the winning configuration is chosen by the objective, and then read per label to see whether it earned it.

should_stop is asked before every configuration, and a search that is stopped RETURNS what it has -- ranked, with its per-category table -- rather than raising. A stop is a decision that enough has been seen, not an error, and throwing away twenty minutes of finished configurations because the user pressed stop on the twenty-first would make the button unusable. Asked per configuration rather than per log line, because the log is throttled to every fortieth run and a stop that takes four minutes to land reads as a button that does not work.

on_run is called with a RunStep as each EMBEDDING finishes -- carrying its best clustering among the min_cluster_sizes tried, its per-category scores, its coordinates and its labels. Per embedding rather than per row, because the rows for one embedding differ only in the clustering and a gallery of the same map five times is not a gallery. Every row still reaches the returned table: what is streamed is what can be looked at, not a subset of what was scored.

Rebuild one row of a search result: the same map, on the same genes, with its clustering.

The inverse of what search records, and it lives here so it cannot drift from the loop that wrote the row. Every step is taken the way the search took it, because a rebuild that differs in any of them puts a different map on screen from the one the row's numbers describe:

• the feature matrix is built over the WHOLE table and then subset, not built over the subsample -- rank scaling over 3,000 genes is not rank scaling over 8,140 restricted to them;
• the subsample is redrawn from the recorded seed and sample_size, then sorted;
• the columns the run excluded are excluded again, from the row rather than recomputed, since the exclusion is what makes the score mean anything;
• the clustering uses the row's own min_cluster_size.

Returns (coords, genes, labels, features), where genes is a boolean mask over the node table saying which genes have a position -- the rest are not in this map and must not be drawn in it.

Turn a recovered structure into predictions for the unlabeled genes in each cluster.

Only clusters that are already purely one label are used, and the precision achieved on the labeled members is carried through as the prediction's confidence -- it is the only honest estimate of how often the prediction will be right.

Every category address at one LEVEL of a hierarchy, with the blocks beneath it.

Returns [(path, [block, ...]), ...]. A level rather than the whole tree, because the question "does this class of evidence recover on a map built without it" is asked of a class, and which level counts as a class is the reader's choice: level 1 is "molecular measurements", level 3 is "transcript abundance". Both are real questions and they have different answers.

Hold out each category in turn, rebuild the map without it, and ask whether it comes back.

This is the project's whole argument, run as a procedure instead of by hand. For each category: build the embedding from every OTHER block, cluster it, then test the held-out category's columns against those clusters with clustering.battery -- which marks anything the map actually saw as used or derived, so only genuinely held-out evidence is scored.

A category that recovers strongly is structure the rest of the evidence already implies. One that does not is either independent information or noise, and the two are told apart by looking, not by this function -- which is why the per-feature table comes back rather than only a score.

A row whose clustering is degenerate is not scored at all. best_score comes back NaN with usable=False and why_not saying which guard fired. Measured on the shipped table, every block set and every parameter setting tried produces a degenerate clustering today, so this is the common case rather than a corner: without the guard the sweep would report a column of plausible numbers computed on arbitrary bisections.

should_stop is asked between categories and a stopped sweep RETURNS what it has, for the same reason search does: a stop is a decision that enough has been seen, not an error.

How much structure a clustering found, and whether it may be believed at all.

usable is clustering.degenerate and it gates everything: a configuration that produced a bisection scores nothing, however good its other numbers look. Tuning toward a score without that gate would simply find the settings that bisect most confidently.

Find UMAP settings that give a map worth clustering, cheaply first and fully only for survivors.

Successive halving, because the cost is the embedding and most settings are not worth paying it twice for: every configuration is built on a SAMPLE, the top keep fraction survives, and only those are rebuilt on the full table. On this catalogue that is four settings evaluated at 1,500 genes and two at 8,140, rather than six full builds.

Returns every configuration with its quality, ranked -- including the ones that failed, and why, because "these four settings all bisected the cloud" is the useful half of the answer when nothing works.

Search clustering settings on one finished embedding, ranked by map_quality.

Separate from tune_umap because the costs differ by orders of magnitude: an embedding of 8,140 genes takes tens of seconds and a reclustering of it takes a fraction of one. Tuning them jointly would rebuild the map for every clustering setting and spend the entire budget re-deriving the same coordinates.

The best clustering settings for one map, or the least bad with a reason when none is usable.

Hold out every category at every level of every hierarchy, and say what was skipped and why.

sweep_categories asks the question of ONE level of ONE facet, and which level counts as a class is the reader's choice rather than a fact -- level 1 is "molecular measurements", level 3 is "transcript abundance", and both are real questions with different answers. Running one level and reporting it as the answer quietly picks one of those for the reader.

Returns (table, skipped). The skips are returned rather than logged away because a sweep that covered nine of thirty categories and said "nine categories" reads exactly like one that covered everything -- and the commonest reason for a skip here is the circularity guard removing every block a category names, which is the fact a reader most needs.

The sweep, plus the genes each recovered category would name. Step 4a of instruction 43.

The sweep answers "does this class of evidence come back", which is a property of the catalogue. This adds the part a biologist reads: for every category that recovered, the UNLABELLED genes in clusters where that category's own labels are concentrated, with the statistics that produced them and a relevance ranking beside -- never instead of -- the raw numbers.

A gene named here is a PREDICTION and never goes back into the node table as though it were a measurement. It comes out as its own table, carrying the category, the cluster and the enrichment that produced it.