Metadata-Version: 2.4
Name: multiverse_cache
Version: 0.2.0
Summary: Download, cache, and read genome info, datasets, and ML models
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: requests
Requires-Dist: tqdm
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: intervalframe
Provides-Extra: motif
Requires-Dist: MOODS-python; extra == "motif"
Provides-Extra: models
Requires-Dist: torch; extra == "models"
Provides-Extra: sturgeon
Requires-Dist: onnxruntime; extra == "sturgeon"
Provides-Extra: marlin
Requires-Dist: onnxruntime; extra == "marlin"
Requires-Dist: pyreadr; extra == "marlin"
Requires-Dist: openpyxl; extra == "marlin"

# multiverse_cache

Download, cache, and read the reference resources used by the M-PACT pipeline.
Three kinds of resource are managed through one machine-level cache:

- **genomes** — genome "info" packages (from `kylessmith/<genome>_info` on GitHub)
- **datasets** — Zenodo data records (e.g. methylation annotation parquet tables)
- **models** — Zenodo model records (e.g. M-PACT `.pth` checkpoints)
- **sturgeon** — Sturgeon CNS classifier models (zip archives, from the Sturgeon project)

Nothing is written into the installed package directory. Everything lands in a
per-machine cache created on first use and reused thereafter.

## Cache location

Resolved in priority order:

1. an explicit `cache_dir=` argument
2. the `MULTIVERSE_CACHE` environment variable
3. `$XDG_CACHE_HOME/multiverse_cache`
4. `~/.cache/multiverse_cache`

On a cluster, point it at scratch to avoid `$HOME` quotas:

```bash
export MULTIVERSE_CACHE=/scratch/$USER/multiverse_cache
```

Layout:

```
<root>/
  base/<genome>.pickle           # genome info manifest
  data/<genome>/...              # genome info data + external/
  datasets/<name>/...            # Zenodo datasets (e.g. methyl_anno)
  models/<name>/...              # Zenodo models (e.g. MPACT)
  sturgeon/<name>/<model>.zip    # Sturgeon models (kept zipped, never unpacked)
  <name>/.complete               # per-resource completion marker
  <key>.lock                     # advisory download lock
```

## Usage

### Genome info (unchanged API)

```python
from multiverse_cache import InfoReader
info = InfoReader("hg38")        # downloads once, cached after
frame = info["some_key"]
```

### Datasets (replaces import_data / download_data)

```python
from multiverse_cache import get_data_file, download_dataset, list_data_files

download_dataset("methyl_anno")              # explicit prefetch (optional)
path = get_data_file("some.parquet")         # auto-downloads methyl_anno, returns path
files = list_data_files("methyl_anno")
```

`get_data_file(filename)` is the cache-aware drop-in for the old function: it
ensures the dataset is present, then locates the file inside the cache. Pass
`dataset=None` to search across all cached datasets without downloading.

### Models (new)

```python
from multiverse_cache import download_models, list_models, get_model_file, load_model

download_models("MPACT")                     # explicit prefetch (optional)
list_models("MPACT")                         # -> [".../MPACT_classifier.pth", ...]
ckpt = load_model("MPACT_classifier.pth")    # torch.load(map_location="cpu") -> object
path = get_model_file("MPACT*.pth")          # just the path, no load
```

`load_model` lazily imports torch and defaults to `map_location="cpu"` so a
checkpoint trained on H100s loads on a login node before you move it to a
device. `weights_only` defaults to `False` (trusted first-party checkpoints
that bundle config/objects); set `True` for untrusted files.

### Sturgeon CNS models (new)

[Sturgeon](https://github.com/UMCUGenetics/sturgeon) (Vermeulen, Pagès-Gallego,
Kester et al., *Nature* 2023) ships each CNS classifier as a **zip** holding an
ONNX network plus probe/decoding/calibration tables. The zip is the unit of
distribution and is opened as-is by Sturgeon's own loader, so it is cached
**intact and never unpacked**.

> **Licensing.** Sturgeon is distributed under an *Evaluation Software License
> Agreement* (Oncode/Cyclomics/UMCU), **not** an open-source licence: academic
> research only, **no derivative works / reimplementation**, and
> clinical/diagnostic use needs a separate commercial licence
> (`software@cyclomics.com`); publishing results requires referral to the
> source. This package therefore **drives the genuine Sturgeon package** — it
> does not vendor or reimplement Sturgeon's prediction code.

```python
from multiverse_cache import (
    list_sturgeon, download_sturgeon, get_sturgeon_model_file, predict_sturgeon,
)

list_sturgeon()                              # -> ['brainstem', 'general']

# Path-only access (no extra deps): hand this to the sturgeon CLI / loader
zip_path = get_sturgeon_model_file("general")
```

#### Predicting (recommended: isolated, zero env impact)

Sturgeon's released code needs `pandas<2.2`, `numpy<2`, an old `onnxruntime`,
and matplotlib — versions that usually clash with a modern torch/Mamba stack.
`predict_sturgeon` sidesteps that by building a **dedicated venv under the cache**
(once), installing the genuine Sturgeon + compatible deps into it, and running
Sturgeon's own `predict` CLI there. Your main environment is never modified.

```python
# bed = output of `sturgeon inputtobed` (T2T/CHM13v2-aligned methylation calls)
scores = predict_sturgeon("sample.bed", model="general")   # first call builds the venv
# -> {"sample": <per-class score DataFrame>}
```

> **Interpreter requirement.** Those pinned deps only ship wheels for **Python
> 3.9–3.12**. The venv is built from such an interpreter, resolved in order:
> `python_executable=` / `$MULTIVERSE_STURGEON_PYTHON` → the current interpreter
> if it qualifies → the newest `python3.X` on `PATH` → **a conda/mamba env**.
>
> So if your session is Python 3.13/3.14 (e.g. a bleeding-edge mamba env) and no
> `python3.X` is on `PATH`, `predict_sturgeon` will, by default, reuse a conda
> env named `sturgeon` or create one automatically:
>
> ```
> mamba create -y -n sturgeon python=3.10      # run for you (mamba/micromamba/conda)
> ```
>
> This assumes a conda/mamba tool is installed (it picks `mamba` → `micromamba`
> → `conda`). Controls:
> - `predict_sturgeon(..., auto_conda=False)` disables the auto-create (you'll
>   get a clear error instead);
> - `sturgeon_env_python(conda_env=..., conda_python=...)` change the env name /
>   Python version;
> - `python_executable=...` or `export MULTIVERSE_STURGEON_PYTHON=...` pin a
>   specific interpreter and skip discovery entirely (recommended for
>   reproducible cluster runs).
>
> (Without a compatible interpreter, pip would try to compile pandas 2.1.x from
> source and fail against the newer CPython C-API — which is the error you hit
> on a 3.14 env.)

`sturgeon_env_python(python_executable=...)` builds/returns that venv's
interpreter directly, and `predict_sturgeon(..., env_python=...)` lets you point
at a conda env where you already have a compatible Sturgeon.

#### In-process loading (only on an already-compatible env)

If *this* environment already satisfies Sturgeon's pins, you can load the ONNX
session in-process:

```python
from multiverse_cache import load_sturgeon, install_sturgeon

install_sturgeon()                  # pip-installs genuine sturgeon into THIS env
model  = load_sturgeon("general")   # or load_sturgeon("general", auto_install=True)
df     = model.predict("sample.bed")
```

`install_sturgeon()` installs the package unmodified with
`--no-deps --ignore-requires-python` (so it won't downgrade your numpy/pandas),
but prediction will still fail here if this env has `pandas>=2.2` / `numpy>=2`
— hence the isolated route above is preferred. `download_sturgeon(..., install=True)`
fetches the model and installs Sturgeon in one call.

The models are hosted on Dropbox; `?dl=0` share links are normalised to
`?dl=1` automatically. If those links rot, re-register the resource with new
`files` (see below).

To add a Sturgeon model from your own URL:

```python
from multiverse_cache import register, Resource, STURGEON
register(Resource(name="sturgeon_custom", category=STURGEON, source="url",
                  extract=False,
                  files=(("custom.zip", "https://host/custom.zip"),)))
predict_sturgeon("sample.bed", model="custom")
```

### MARLIN acute-leukemia classifier (Python/ONNX, no R)

[MARLIN](https://github.com/hovestadt/MARLIN) (Steinicke, Benfatto et al.,
*Nature Genetics* 2025, MIT-licensed) classifies acute leukemia from sparse
methylation profiles with a Keras network. MARLIN ships an R prediction script,
but this plugin needs **no R**: the trained model is converted from Keras HDF5
to **ONNX once** (in a throwaway TensorFlow env that is then discarded) and run
with `onnxruntime`, exactly like the Sturgeon runtime. The reference probe order
(an `.RData`) is read with `pyreadr` and the class table (`.xlsx`) with `pandas`.

```python
from multiverse_cache import predict_marlin

# bed: chrom  start  end  methylation(0-1 or NA)  probe   (e.g. modkit pileup
#      intersected with the MARLIN probe coordinates)
scores = predict_marlin("sample.bed")     # first call: download + one-time convert
# -> DataFrame: samples (rows) x 42 methylation classes (cols), softmax probs
```

Runtime dependencies are pure Python: `pip install "multiverse_cache[marlin]"`
(`onnxruntime`, `pyreadr`, `openpyxl`). The **one-time** HDF5→ONNX conversion
needs TensorFlow + tf2onnx; rather than touch your environment, it builds a
temporary venv from a **3.9–3.11** interpreter (TF 2.13's wheel range), converts,
caches `marlin_v1.model.onnx`, and deletes the temp env. If your session is
outside 3.9–3.11 it uses the same conda/mamba fallback as Sturgeon
(`predict_marlin(..., auto_conda=...)`, `python_executable=...`, or
`$MULTIVERSE_MARLIN_PYTHON`). Pass `keep_convert_env=True` to keep the TF env
for re-conversion.

Acquisition splits across sources automatically: the model HDF5 from Zenodo
(`marlin_model`) and the probe/annotation files from GitHub (`marlin_refs`).

```python
from multiverse_cache import download_marlin, load_marlin
download_marlin()                 # prefetch model + refs (optional)
model = load_marlin()             # convert-if-needed + onnxruntime session
df = model.predict("beds/")       # a directory of beds, one sample each
```

### Registering more resources

```python
from multiverse_cache import register, Resource, MODELS, DATASETS

register(Resource(name="MPACT_v2", category=MODELS, record_id="12345678",
                  description="next M-PACT checkpoints"))
load_model("MPACT_v2_classifier.pth", name="MPACT_v2")
```

### Managing the cache

```python
from multiverse_cache import MultiverseCache
cache = MultiverseCache("/scratch/me/multiverse_cache")
cache.list_genomes()                 # downloaded genome-info packages
cache.list_resources("models")       # downloaded models
cache.list_resources("datasets")     # downloaded datasets
cache.remove_resource("models", "MPACT")
cache.clear()
```

## Behavior notes

- **Atomic downloads** — every file streams to a `.part` temp and is renamed
  into place only on success; checksums from Zenodo are verified before rename.
- **Completion markers** — a resource is only "ready" once `.complete` exists,
  so a half-finished download is re-fetched rather than read.
- **Concurrency** — downloads take a per-resource `flock`, so an LSF/SLURM array
  all requesting the same genome/dataset/model fetches it once; the rest wait.
- **Extension-aware extraction** — archives are unpacked by extension
  (`.zip`, `.tar`, `.tar.gz`/`.tgz`, `.gz`); everything else (`.parquet`,
  `.pth`, `.npz`, ...) is left intact. This fixes a bug in the original loader,
  which force-extracted every download as a zip and would corrupt a torch
  `.pth` checkpoint (those are themselves zip archives). A resource can also
  set `extract=False` to keep its archives whole even when they end in `.zip`
  — used for Sturgeon model zips, which their own loader opens with
  `zipfile.ZipFile`.
- **Multiple sources** — a resource's `source` is either `"zenodo"` (files,
  sizes and checksums discovered from a record) or `"url"` (explicit
  `(filename, url)` pairs, e.g. Dropbox-hosted Sturgeon models).
- Source accounts/records: genome info from `REPO_USER` in `download.py`;
  dataset/model records in `registry.py`.
