Metadata-Version: 2.4
Name: crispyx
Version: 0.0.10
Summary: Memory-efficient streaming analysis of large-scale CRISPR and Perturb-seq screens on disk-backed AnnData files
Author: Jin-Hong Du
License: Modified MIT License
        
        Copyright (c) 2025-2026 Jin-Hong Du
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        --------------------------------------------------------------------------------
        ADDITIONAL ATTRIBUTION CONDITIONS
        --------------------------------------------------------------------------------
        
        The only modifications to the MIT License are the two attribution conditions
        below. They impose no fee, no royalty, and no reporting obligation, and they do
        not restrict who may use the Software or for what purpose.
        
        For the purposes of these conditions, "Commercial Use" means use of the
        Software, or of any derivative work thereof, in the internal operations of a
        for-profit entity, or in the research, development, production, or delivery of
        any product or service that is sold, licensed, or offered for a fee.
        
        1.  ACKNOWLEDGEMENT IN COMMERCIAL USE
        
            Should the Software or any derivative work thereof be put to Commercial
            Use, you shall acknowledge crispyx and its author in the documentation,
            release notes, "About" screen, methods section, or equivalent accompanying
            materials of the product or service concerned. The following form is
            sufficient:
        
                This product uses crispyx (https://github.com/jinhongdu-lab/crispyx),
                copyright (c) Jin-Hong Du.
        
        2.  PROMINENT ATTRIBUTION AT SCALE
        
            Should the Software or any derivative work thereof be put to Commercial
            Use by an organisation that meets any one of the following criteria, you
            shall additionally display "crispyx" prominently on the user interface,
            product documentation, public product description, or public research and
            development pages of the product, service, or programme concerned:
        
              (a) more than 1 million US dollars (or the equivalent in other
                  currencies) in annual revenue; or
        
              (b) more than 5 million US dollars (or the equivalent in other
                  currencies) in cumulative capital raised, whether by equity, grant,
                  or other funding; or
        
              (c) more than 25 employees.
        
            Criteria (b) and (c) are included because organisations engaged in
            pre-clinical research, which are the principal users of this Software,
            commonly operate for many years with substantial resources and no product
            revenue. Revenue alone is therefore not a meaningful measure of scale in
            this field.
        
        --------------------------------------------------------------------------------
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
        --------------------------------------------------------------------------------
        NOTE ON EARLIER VERSIONS
        --------------------------------------------------------------------------------
        
        Versions of crispyx up to and including 0.0.8 were released under the
        unmodified MIT License. That grant is perpetual and irrevocable and is not
        withdrawn by this document. The attribution conditions above apply to crispyx
        version 0.0.9 and later.
        
Project-URL: Homepage, https://github.com/jinhongdu-lab/crispyx
Project-URL: Documentation, https://crispyx.readthedocs.io
Project-URL: Repository, https://github.com/jinhongdu-lab/crispyx
Project-URL: Bug Tracker, https://github.com/jinhongdu-lab/crispyx/issues
Keywords: CRISPR,Perturb-seq,single-cell RNA-seq,AnnData,Scanpy,differential expression,negative binomial GLM,pseudobulk,bioinformatics,functional genomics
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: anndata>=0.9
Requires-Dist: numpy>=1.23
Requires-Dist: numba>=0.59
Requires-Dist: pandas>=1.5
Requires-Dist: scipy>=1.10
Requires-Dist: h5py>=3.0
Requires-Dist: joblib>=1.0
Requires-Dist: scikit-learn>=1.0
Requires-Dist: scanpy>=1.9.2
Requires-Dist: seaborn>=0.12
Requires-Dist: matplotlib>=3.5
Requires-Dist: tqdm>=4.50
Provides-Extra: test
Requires-Dist: filelock; extra == "test"
Requires-Dist: pytest; extra == "test"
Requires-Dist: statsmodels>=0.14; extra == "test"
Requires-Dist: pydeseq2>=0.4; extra == "test"
Provides-Extra: benchmark
Requires-Dist: pertpy>=0.4; extra == "benchmark"
Requires-Dist: pyyaml>=6.0; extra == "benchmark"
Requires-Dist: tqdm>=4.65; extra == "benchmark"
Requires-Dist: psutil>=5.9; extra == "benchmark"
Provides-Extra: docs
Requires-Dist: sphinx>=6.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.2; extra == "docs"
Requires-Dist: sphinx-copybutton>=0.5; extra == "docs"
Requires-Dist: nbsphinx>=0.9; extra == "docs"
Requires-Dist: ipykernel>=6.0; extra == "docs"
Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "docs"
Dynamic: license-file

# crispyx

[![License: Modified MIT](https://img.shields.io/badge/License-Modified%20MIT-yellow.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![PyPI](https://img.shields.io/pypi/v/crispyx?label=pypi&color=orange)](https://pypi.org/project/crispyx)
[![PyPI Downloads](https://static.pepy.tech/personalized-badge/crispyx?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=BRIGHTGREEN&left_text=downloads)](https://pepy.tech/project/crispyx)
[![Tests](https://github.com/jinhongdu-lab/crispyx/actions/workflows/tests.yml/badge.svg)](https://github.com/jinhongdu-lab/crispyx/actions/workflows/tests.yml)

## Motivation

Genome-wide CRISPR screens routinely produce datasets with hundreds of thousands of cells and tens of thousands of genes. Standard single-cell analysis toolkits (Scanpy, Pertpy) load the entire count matrix into memory, requiring large RAM allocations and often making routine workflows impractical on laptops or shared compute nodes.

**crispyx** solves this by streaming data directly from on-disk AnnData (`.h5ad`) files. Quality control, normalisation, pseudo-bulk aggregation, and differential expression all operate without materialising the full matrix.

## Features

- **Streaming QC & preprocessing** – Filter cells, perturbations, and genes; normalise and log-transform; CSC-aware streaming with `format_mismatch_policy`; all without loading the full matrix into memory
- **Pseudo-bulk aggregation** – Absolute profiles over multiple grouping columns (for example, perturbation × batch), strict count sums or mean log1p expression, optional deterministic bootstrap sampling, and explicit within-batch effect calculation
- **Differential expression** – t-test, Wilcoxon rank-sum (including batch-stratified / van Elteren test via `batch_column`), and negative binomial GLM with apeGLM LFC shrinkage; multi-core support and adaptive memory management; per-condition low-expression filtering to exclude genes that are near-zero in both groups
- **Dimension reduction** – Memory-efficient PCA and KNN graph construction on backed data
- **Scanpy-compatible API & plotting** – Familiar `cx.pp`, `cx.pb`, `cx.tl`, and `cx.pl` namespaces; Scanpy-style rank genes plots, volcano, MA, PCA, UMAP, QC summaries, and overlap heatmaps
- **Data preparation utilities** – Edit backed metadata without loading X; standardise gene names; normalise perturbation labels; auto-detect metadata columns
- **HPC-ready** – Resume/checkpoint for long-running jobs; configurable `memory_limit_gb`; Docker and Singularity support
- **Disk-aware** – Estimates and warns about scratch-disk usage before large writes or CSC/CSR conversions, and `cx.estimate_disk_usage(...)` answers "how much disk will this need?" up front; the memory savings above assume the machine has enough free disk for streaming intermediates and output files

## Quick Start

```python
import crispyx as cx

# Open dataset without loading into memory
adata = cx.read_h5ad_ondisk("data/demo_benchmark.h5ad")

# Quality control with adaptive thresholds
adata = cx.pp.qc_summary(
    adata,
    perturbation_column="perturbation",
    min_genes=5,
    min_cells_per_perturbation=5,
)

# Differential expression
adata = cx.tl.rank_genes_groups(
    adata,
    perturbation_column="perturbation",
    method="wilcoxon",  # or "t-test", "nb_glm"
)

# Access results
print(adata.uns["rank_genes_groups"])
de_results = adata.uns["rank_genes_groups"].load()
```

For the full workflow (normalisation, PCA, pseudo-bulk, NB-GLM, LFC shrinkage, plotting, data preparation utilities), see the [Usage Guide](docs/usage.rst) and the [tutorial notebook](docs/notebooks/crispyx_tutorial.ipynb).

## Performance

Benchmarked across 12 CRISPR screen datasets (21k–1.97M cells), crispyx (v0.0.1) consistently outperforms Scanpy, Pertpy/PyDESeq2, and edgeR in both speed and memory:

| Metric | crispyx vs Scanpy | crispyx vs Pertpy/PyDESeq2 |
|---|---|---|
| **t-test** | **1.6–3.9× faster** (median 2.5×) | — |
| **Wilcoxon** | **5–43× faster** (median 8.5×) | — |
| **NB-GLM** | — | **1.6–3.6× faster** (median 2.0×) |
| **Peak memory** | **2–59× lower** for t-test (median 6×); comparable for Wilcoxon | **1.4–10× lower** (median 2.5×) |
| **Accuracy** (effect sizes) | Pearson *r* > 0.999 | Pearson *r* > 0.97 |

Speed and memory ratios are computed only over runs where both tools completed, so they exclude the largest screens — precisely the cases where crispyx's advantage is greatest.

crispyx completed **all 89 runs across all 12 datasets**. Pertpy/PyDESeq2 completed on 4 of 12 datasets and edgeR on none, both failing by timeout or memory limit. Scanpy completed at least one task on every dataset, but timed out or exceeded the memory cap on 12 of its 36 runs.

<p align="center">
  <img src="benchmarking/figures/fig2.png" width="800" alt="Benchmark results across 12 CRISPR screens: (a) dataset sizes, (b) completion status by method, (c) concordance with Scanpy, (d) runtime scaling, (e) peak memory scaling">
</p>

See [benchmarking/](benchmarking/) for full results and reproduction scripts.

## Installation

```bash
pip install crispyx
```

For development (editable install with all extras):

```bash
git clone https://github.com/jinhongdu-lab/crispyx.git
cd crispyx
pip install -e ".[test,benchmark,docs]"
```

crispyx supports Python 3.10–3.12 and is compatible with recent releases of the
scientific stack, including `anndata >= 0.13` and `pandas >= 3.0` (where string
metadata is stored on disk using the nullable-string encoding).

## Benchmarking

```bash
cd benchmarking
./run_benchmark.sh config/Adamson.yaml       # single dataset
./run_benchmark.sh config/*.yaml             # all datasets
```

See [benchmarking/README.md](benchmarking/README.md) for configuration options and output structure.

## Testing

```bash
pytest
```

## Documentation

```bash
sphinx-build docs docs/_build
```

## Acknowledgements

crispyx builds on the foundational work of [Scanpy](https://scanpy.readthedocs.io/) (Wolf *et al.*, 2018), [Pertpy](https://pertpy.readthedocs.io/), [PyDESeq2](https://pydeseq2.readthedocs.io/) (Muzellec *et al.*, 2023), and [AnnData](https://anndata.readthedocs.io/) (Virshup *et al.*, 2024). We gratefully acknowledge these projects for establishing the single-cell analysis ecosystem in Python; crispyx extends their APIs and algorithmic designs to enable memory-efficient, streaming computation for large-scale CRISPR screen datasets.

## Contributing

Suggestions, bug reports, and contributions are welcome! Please open an [issue](https://github.com/jaydu1/crispyx/issues) or submit a pull request.

## License

crispyx **0.0.9 and later** is released under a [Modified MIT License](LICENSE) — every MIT freedom, with no fee or royalty, plus two attribution conditions that apply to commercial use. See the [LICENSE](LICENSE) for the terms. crispyx **0.0.8 and earlier** remains under the unmodified MIT License.
If you use crispyx in research, please cite it — see [CITATION.cff](CITATION.cff).
