starplast.importer
Bringing a user's own table into the map, with every choice made along the way visible.
Everything here already exists somewhere in this project, applied to published data:
tuning.import_table resolves identifiers through the identity layer, sources.normalize applies
the transform a quantification type implies, embedding supplies the scalings and the missing-value
policies. What was missing is that none of it was reachable for a file of your own, so importing
meant either editing the node table by hand or not importing.
The one rule that governs the whole module, learned the expensive way and written on the wall of
build_graph:
**THE RANGE DECIDES, NOT THE FILENAME.**
GSE108740_FPKM.xlsx reaches 16,520 and is real FPKM; the columns derived from it in the node table
are still called rna108740_*_FPKM and top out at 9.7, because they were logged upstream. Take the
log twice and real variation compresses to nothing; skip it and one gene dominates every distance.
Neither is visible in the output. So describe_columns reports each column's range and
suggest_quantification reads the numbers rather than the name -- and the interface shows both and
lets a person say which is right.
Nothing here writes to the node table on disk. An imported column lives in the session, is prefixed so it cannot be mistaken for a measurement that shipped with the cache, and carries a record of every choice that produced it.
Read a table in whatever format it arrived in.
The separator is sniffed rather than assumed: half the supplementary tables in this field are
.txt that are really TSV, and a .csv exported from Excel in a European locale is
semicolon-separated. Guessing wrongly gives one column with everything in it, which looks like a
file with no data rather than a file read badly.
The sheet names of a workbook, or [] for anything else.
Published supplements routinely put the table on sheet 3 behind a legend and a blank sheet, so which sheet is a choice the user has to be able to make.
The columns worth importing: the numbers, minus the identifier column.
Each column's range, missingness and shape -- what the quantification decision rests on.
Shown before anything is transformed, because this is the table on which a person decides
whether they are looking at FPKM or at something that has already been logged. max is the
number that gives it away: a column that reaches 16,520 has not been logged, one that stops at
9.7 has.
A guess at the quantification type, and the reason for it, read off the NUMBERS.
Never off the filename. The guess is a starting point for a human, which is why it comes back with its reason attached: "values reach 16,520, so this has not been logged" is something a person can agree or disagree with, and "fpkm" alone is not.
A user's table to gene-keyed, transformed columns -- returning the record of what was done.
Returns (frame, record). The record is not decoration: an imported column whose provenance is
a memory of which dropdowns were set is a column nobody can defend three weeks later, and this
project's whole discipline is that a number carries what produced it.
The order is the order these things have to happen in. Identifiers first, because a table that resolves to nothing should say so before anything is computed. Then duplicates, because two rows for one gene must become one before a rank means anything. Then the sign, then the quantification transform, then the missing-value policy, then the scaling -- each one operating on what the previous produced.
Attach imported columns to the node table, in memory, keyed by gene id.
A copy, never the file. The cache is measurement that shipped with the program; an imported column is the user's own and must not become indistinguishable from it by being written into the same table on disk. The prefix does the same job for anyone reading a column name.