Metadata-Version: 2.4
Name: npcomp
Version: 0.1.0
Summary: Minimal CLI wrapper around mtscomp for SpikeGLX Neuropixels data compression and backup
License-Expression: BSD-3-Clause
Keywords: neuropixels,spikeglx,ephys,mtscomp,compression,backup,neuroscience
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mtscomp==1.0.2
Requires-Dist: numpy>=1.20
Requires-Dist: tqdm>=4.60
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: build>=1; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"
Dynamic: license-file

# npcomp — lossless compression and backup for SpikeGLX Neuropixels data

`npcomp` is a minimal CLI wrapper around [mtscomp](https://github.com/int-brain-lab/mtscomp), the IBL lossless compressor for raw Neuropixels data (about 3x compression):

- **compress**: losslessly compress SpikeGLX `.bin` files into `.cbin + .ch` (mtscomp's built-in byte-per-byte verification runs automatically);
- **restore**: decompress `.cbin + .ch` back to `.bin` (also automatically verified);
- **status**: report compression state, space already saved, estimated savings and time;
- **backup mode (`--dest`)**: mirror a whole folder into a destination; `.bin` files are compressed, everything else is copied and verified.

Chinese version: [README_ZH.md](README_ZH.md).

## Installation

From PyPI:

```powershell
pip install npcomp
```

From source (current development workflow):

```powershell
conda create -n mtscomp python=3.10 -y
conda activate mtscomp
cd <path-to-repo>
pip install -e .
```

Add the `Scripts` folder of the conda environment (e.g. `<conda-env>\Scripts`, where `npcomp.exe` lives) to your user PATH, then open a new terminal and run `npcomp` directly.

## Quick start

```powershell
# Report compression status
npcomp status <path-to-session>

# Compress in place (.bin -> .cbin + .ch, originals kept)
npcomp compress <path-to-session>

# Mirror backup to a destination: .bin compressed, everything else copied & verified
npcomp compress <path-to-session> --dest <backup-dir>

# Same backup, but run .bin compression through a fast scratch disk (e.g. NVMe)
npcomp compress <path-to-session> --dest <backup-dir> --scratch <fast-disk-dir>

# Delete source .bin only after a successful, verified compression (this run only)
npcomp compress <path-to-session> --dest <backup-dir> --delete

# Restore (in place by default; --dest redirects the output)
npcomp restore <backup-dir>
npcomp restore <backup-dir> --dest <restore-dir>
```

## Command reference

**`npcomp compress <folder> [--dest DIR] [--scratch DIR] [--dry-run] [--delete] [--cpus N] [--chunk SEC] [--force] [--yes] [--quiet]`**

- Recursively scans `<folder>` and only processes `.bin` files that have a valid SpikeGLX `.meta` sibling (containing `nSavedChans` plus `imSampRate`/`niSampRate`); covers ap/lf/nidq and leaves every other `.bin` untouched.
- Parameters are read from the `.meta`: channel count from `nSavedChans`; sample rate from `imSampRate` (ap), `imSampRate/12` (lf), or `niSampRate` (nidq); data type is fixed to `int16`.
- Defaults: 1-second chunks, all CPU cores, verification always on.
- **Without `--dest`**: writes `X.cbin + X.ch` next to the source; existing pairs are skipped, `--force` recompresses.
- **With `--dest`**: mirror-backup mode — `.bin` files are compressed into the destination, every other file (`.meta`, logs, video sidecars, ...) is copied and byte-verified. Existing destination files with identical content are skipped; same-name/different-content files trigger an overwrite prompt (`--yes` answers yes everywhere, `--force` redoes everything).
- **With `--scratch` (requires `--dest`)**: raw `.bin` files are first copied to a fast scratch directory (e.g. an NVMe SSD), compressed there, and the verified `.cbin/.ch` are copied back to the destination. This can be much faster when the destination is a slow disk. Scratch files are removed after each file; the scratch directory must be empty before a run (npcomp refuses otherwise) and should stay dedicated to npcomp.
- `--delete`: delete the source `.bin` only for files successfully compressed and verified in this run.
- `--dry-run`: print the plan (compress/copy/delete) without touching anything.
- If the destination already contains data (e.g. a previous run interrupted by power loss), you are prompted first; non-interactive runs require `--yes`.

**`npcomp restore <folder> [--dest DIR] [--overwrite] [--delete] [--dry-run] [--quiet]`**

- Recursively finds every `X.cbin + X.ch` pair and decompresses it to `X.bin` (mtscomp verifies each decompression byte-for-byte).
- Restores in place by default; existing `.bin` files are skipped unless `--overwrite` is given.
- The compressed `X.cbin + X.ch` pair is **kept by default**; only an explicit `--delete` removes it after a successful restore (and only pairs successfully restored in this run; skipped pairs are kept).
- A missing `.meta` only produces a warning — it is not needed for byte-level decompression, but downstream tools need it.

**`npcomp status <folder> [--ratio F] [--throughput MBS]`**

- Reports per-file state (`raw-only / both / compressed-only / incomplete`, plus `+tmp` for leftovers) and sizes.
- Compressed files use actual sizes for the saved-space figure; uncompressed files are estimated with the default compression ratio 3.0 (`--ratio`) and throughput 88 MB/s (`--throughput`).
- Status is based on file existence and size only; it does not validate file contents.

## Safety design

- **Lossless + automatic verification**: compression and decompression both run a byte-per-byte check; failures raise errors and clean up temp files, so no half-written output survives.
- **Atomic rename**: data is written to `*.tmp` first and renamed into place via `os.replace` only after verification; a power loss leaves only `.tmp` files, which are cleaned automatically on the next run. Source `.bin` files are never touched by a failed run.
- **Conservative deletion**: only an explicit `--delete` removes source `.bin` files, and only those successfully compressed and verified in this run.
- **Backup verification**: in `--dest` mode every copied file is byte-verified after copying; `.npcomp_manifest.json` in the destination records the SHA-256 of every source file (plus the `.cbin`/`.ch` hashes for recordings), so re-runs skip unchanged content and detect same-name/different-content or tampered files.
- **Space check before writing**: `compress` estimates the output size (raw size / 3 plus a small per-file overhead) and refuses to start when the target drive — or the scratch drive, against the largest single file — lacks room. If a disk still fills up mid-run, the run aborts immediately instead of failing file by file.
- **Two-step rename caveat**: `.cbin` and `.ch` are each renamed into place in separate steps; a power loss in between can leave a mismatched pair. The next `compress` run detects this via the manifest and prompts to recompress (self-healing).
- **Keep the trio**: after backup, keep `X.cbin + X.ch + X.meta` together. Losing `.ch` makes decompression impossible; losing `.meta` makes the restored data unusable by downstream tools.

## License

This project is licensed under the [BSD 3-Clause License](LICENSE).
