starplast.sources
Downloading expression data from its source repositories, and normalizing it comparably.
Two repositories, two very different situations, and the difference decides what can be automated.
GEO deposits processed matrices. GSE108740_FPKM.xlsx, GSE206344_Normalized_data...xlsx — the
per-gene quantification is right there, so transcriptomics can be fetched and normalized end to end.
PRIDE deposits raw instrument files. PXD019729 holds 24 .raw files, two search outputs and no
per-protein table at all. Turning those into quantification means running MaxQuant or Proteome Discoverer,
which is a different project. So for proteomics this module fetches what PRIDE labels RESULT where it
exists, and otherwise says plainly that only raw was deposited and the numbers must come from the paper's
supplementary tables. Pretending otherwise would be the useful-looking lie here.
Why normalization needs the quantification type recorded
Different quantifications are not interchangeable, and the failure is silent in both directions. The
same series demonstrates each. GEO's GSE108740_FPKM.xlsx reaches 16,520 and is genuine FPKM: skip
the log and one gene dominates every distance in the matrix. The columns derived from it in this
project's node table, still named rna108740_*_FPKM, top out at 9.7 because they were logged
upstream and the name was kept: log them again and real variation compresses into nothing.
Neither error is visible in the output. So the range decides, not the filename, and infer_quant
reports what it concluded.
So every dataset carries a quant label, and the transform follows from it:
counts log2(x + 1), then per-sample median centering
fpkm / tpm log2(x + 1), then per-sample median centering
log_intensity already logged: center only, never log twice
ratio / lfc already relative: leave alone, centering would destroy the reference
ibaq / lfq log2(x + 1), then per-sample median centering
Cross-dataset comparison uses ranks, because that is the only defensible way to put an FPKM column and an iBAQ column on one axis.
Download a GEO series' processed supplementary matrices.
Per-SAMPLE processed files from a GEO series, chosen by suffix.
geo_supplementary reads the series-level directory, which for many deposits holds only a
_RAW.tar of everything -- 139 MB for the ten 25 kB tables that are actually wanted. The
processed per-gene files sit under each sample instead, and the series' own filelist.txt names
every one of them, so which files exist is READ rather than guessed. The one thing derived is the
directory a named file lives in, which GEO forms from the sample accession the file name starts
with; a derivation that is wrong 404s immediately, which is the failure mode to prefer.
Skipped names are logged and counted. A fetcher that silently returns fewer files than the deposit holds is the shape of bug that cost this project a Cryptosporidium corpus.
What a PRIDE project actually contains, by category.
The point is usually to discover that it contains no processed table. Raw files are not silently downloaded: they are tens of gigabytes and useless without a search engine.
Guess a quantification type when it was not recorded, and say so where it is a guess.
Deliberately conservative. The one inference worth making is spotting data that is already logged
despite a name claiming otherwise -- the rna108740_*_FPKM columns top out at 9.13, and logging them
a second time would flatten real differences into noise.
Standardise one dataset's numeric matrix according to its quantification type.
Per-column ranks scaled to [-0.5, 0.5].
The only defensible way to put an FPKM column and an iBAQ column on one axis: it keeps the ordering, which both quantifications support, and discards the units, which are not comparable.
Normalize several datasets and join them on gene id, ranked onto a common scale.
datasets maps name -> (DataFrame indexed by gene_id, quant type).