Metadata-Version: 2.4
Name: zipoop
Version: 0.2.1
Summary: Decoder and reader for Zipoop teacher shards (per-window neural compression of ENCODE signal tracks), v3 and v4 archives
Author: zatumout
License-Expression: MIT
Project-URL: Dataset, https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4
Project-URL: DatasetV3, https://huggingface.co/datasets/zatumout/zipoop_32k1885_010
Keywords: genomics,ENCODE,neural compression,distillation
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.26
Requires-Dist: torch>=2.2
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Dynamic: license-file

# zipoop

Decoder and reader for the Zipoop teacher shards: 89,438 windows of 32 kb, 1,885 ENCODE tracks (GRCh38), stored as one `<chrom>.npz` per chromosome plus `MANIFEST.tsv`.

Two equivalent releases are published:

- [zatumout/zipoop_32k1885_010_v4](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010_v4) — `pooper-dna-v4`, 89.3 GiB, **requires `zipoop>=0.2`**;
- [zatumout/zipoop_32k1885_010](https://huggingface.co/datasets/zatumout/zipoop_32k1885_010) — the older `pooper-dna-v3` encoding, 128.4 GiB, read by 0.1.x as well.

Both decode to the same signals; v4 only changes how the arrays are stored (byte-shuffled members, LZMA, and a float16 sparse FFT with one float32 scale per track).

## Data layout

`<root>/<chrom>.npz` (25 files) plus `MANIFEST.tsv` (chrom, windows, members, bytes, sha256). Each file is a zip with members `<window>/<key>.npy` (161 arrays per window in v3, 164 in v4), windows in genome order. Window names are `<chrom>_<start>_<end>`.

## Usage

```python
from zipoop import Dataset
with Dataset("/path/to/weights_npz") as ds:
    w = ds.load_window(ds.windows("chr21")[0])   # TeacherWindow: key, dna (L,), signal (L, T), valid (L, T)
```

```
zipoop verify /path/to/weights_npz      # size + sha256 against MANIFEST.tsv
zipoop show   /path/to/weights_npz chr21_10027008_10059776
```

Lower level: `Dataset.read(window)` gives the parsed `Shard`, `Decoder(shard).decode(start, end, tracks)` decodes any slice or track subset. Evaluation helpers (`window_pearson`, `summarize`, `calibrate`, `restore_real`) are in `zipoop.evaluation`.

`zipoop.evaluation.subwindow_metrics(pred, target, valid, subwindow=32768)` is the evaluator frozen in the D5 pipeline: it splits the length axis into contiguous sub-windows (4 x 32,768 bp inside a 131,072 bp segment) and returns `pearson` / `proj` / `nmse` / `total_err` / `eligible` arrays of shape (sub-windows, tracks), with the same clamps and the same eligibility rule (at least 2 observed positions and a non-constant target). The aggregation used by the frozen pipeline is the mean Pearson over eligible pairs and the median `proj` / `nmse`.

Rules baked into `load_window` (see `zipoop/teacher.py`): RNA-seq tracks are 0 and valid where unrecorded; other tracks stay missing there. Opening a chromosome file parses its whole zip directory (chr1 about 3.6 s and 0.9 GB), so keep the `Dataset` open; forked DataLoader workers reopen their own handles.

## Supported formats

| archive | `format` | `hybrid_format` | storage |
|---|---|---|---|
| v3 | `pooper-dna-v3` | `fft-plus-dna-residual-v1` | deflate, `fft_coefficients` complex64 |
| v4 | `pooper-dna-v4` | `fft-plus-dna-residual-v2` | LZMA, byte-shuffled floats, `fft_packed` float16 + `fft_scale` float32 |

`read_shard` accepts both and rejects anything else with a clear error. The v4 decode path is verified against v3: parameters, masks and calibrations are bit-identical, the FFT payload differs only by float16 rounding (max relative error about 4.7e-4), and decoded windows reach a per-track Pearson of at least 0.9999997 against the v3 decode.

Tests: `ZIPOOP_DATA=<weights_npz> pytest tests` (the pooper-equivalence test also needs the original shards and the pooper package). The v4 storage encoding has dataset-free unit tests in `tests/test_v4_format.py`.

## License and data terms

The code is MIT-licensed (see `LICENSE`; keep the copyright notice "zatumout"). The shard dataset is released separately under CC BY 4.0 (credit: zatumout), commercial use and model distillation allowed. The shards and `zipoop/data/encode-panel_1885.json` are derived from public ENCODE tracks: when you use them, also cite and acknowledge the ENCODE Consortium as described at https://www.encodeproject.org/help/citing-encode.
