Edit on GitHub

starplast.hf_publish

Publishing the study corpus to HuggingFace, license-gated.

The 97 proximity-labeling and pulldown studies come to ~400 MB, which is past what belongs in git but unremarkable for a HuggingFace dataset repo. The obstacle is not size, it is redistribution rights.

Measured from the JATS license blocks of the local full texts:

52  CC-BY               redistributable with attribution
 1  non-commercial      must be excluded
 6  vague               "open access", "other" -- needs a human decision
38  no local full text  unknown

So this module does two things and refuses to do a third. It builds a derived release — the parsed gene-membership table plus a manifest of source URLs — which sidesteps most of the ambiguity, is what is actually useful downstream, and is roughly a hundredth the size. And it can mirror the raw supplementary files, but only for studies whose license it has positively confirmed as permissive; anything unknown or non-commercial is skipped and listed, never silently included.

Nothing here uploads without an explicit call. plan() shows what would go, and is the intended first step.

PERMISSIVE = re.compile('creativecommons\\.org/licenses/by/|/publicdomain/|^by$|^by \\(text\\)$', re.IGNORECASE)
RESTRICTED = re.compile('nc|nd', re.IGNORECASE)
def licenses_from_fulltexts(catalogue: pandas.DataFrame, xml_dir: str) -> pandas.DataFrame:

Read each study's license out of its local JATS, where we have it.

def plan( members: pandas.DataFrame, studies: pandas.DataFrame, licenses: pandas.DataFrame) -> dict:

What a release would contain. Call this before build_release.

def build_release( out_dir: str, members: pandas.DataFrame, studies: pandas.DataFrame, licenses: pandas.DataFrame, log=<built-in function print>) -> str:

Write the derived release: membership table, study manifest, licenses, and a dataset card.

def upload( repo_id: str, out_dir: str, private: bool = True, log=<built-in function print>):

Push a staged release. Requires huggingface_hub and a token; never called automatically.