Metadata-Version: 2.4
Name: statetrace-sc
Version: 0.2.0
Summary: Donor-aware framework for building, freezing, validating and transporting cellular-state references across single-cell datasets.
License: Apache License
                                    Version 2.0, January 2004
                                 http://www.apache.org/licenses/
         
            TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
         
            1. Definitions.
         
               "License" shall mean the terms and conditions for use, reproduction,
               and distribution as defined by Sections 1 through 9 of this document.
         
               "Licensor" shall mean the copyright owner or entity authorized by
               the copyright owner that is granting the License.
         
               "Legal Entity" shall mean the union of the acting entity and all
               other entities that control, are controlled by, or are under common
               control with that entity. For the purposes of this definition,
               "control" means (i) the power, direct or indirect, to cause the
               direction or management of such entity, whether by contract or
               otherwise, or (ii) ownership of fifty percent (50%) or more of the
               outstanding shares, or (iii) beneficial ownership of such entity.
         
               "You" (or "Your") shall mean an individual or Legal Entity
               exercising permissions granted by this License.
         
               "Source" form shall mean the preferred form for making modifications,
               including but not limited to software source code, documentation
               source, and configuration files.
         
               "Object" form shall mean any form resulting from mechanical
               transformation or translation of a Source form, including but
               not limited to compiled object code, generated documentation,
               and conversions to other media types.
         
               "Work" shall mean the work of authorship made available under
               the License, as indicated by a copyright notice that is included in
               or attached to the work (an example is provided in the Appendix below).
         
               "Derivative Works" shall mean any work, whether in Source or Object
               form, that is based on (or derived from) the Work and for which the
               editorial revisions, annotations, elaborations, or other modifications
               represent, as a whole, an original work of authorship. For the purposes
               of this License, Derivative Works shall not include works that remain
               separable from, or merely link (or bind by name) to the interfaces of,
               the Work and Derivative Works thereof.
         
               "Contribution" shall mean, as submitted to the Licensor for inclusion
               in the Work by the copyright owner or by an individual or Legal Entity
               authorized to submit on behalf to the Licensor, the copyright owner.
         
               "Contributor" shall mean Licensor and any Legal Entity on behalf of
               whom a Contribution has been received by the Licensor and included
               within the Work.
         
            2. Grant of Copyright License. Subject to the terms and conditions of
               this License, each Contributor hereby grants to You a perpetual,
               worldwide, non-exclusive, no-charge, royalty-free, irrevocable
               copyright license to reproduce, prepare Derivative Works of,
               publicly display, publicly perform, sublicense, and distribute the
               Work and such Derivative Works in Source or Object form.
         
            3. Grant of Patent License. Subject to the terms and conditions of
               this License, each Contributor hereby grants to You a perpetual,
               worldwide, non-exclusive, no-charge, royalty-free, irrevocable
               (except as stated in this section) patent license to make, have made,
               use, offer to sell, sell, import, and otherwise transfer the Work,
               where such license applies only to those patent claims licensable
               by such Contributor that are necessarily infringed by their
               Contribution(s) alone or by the combination of their Contributions
               with the Work to which such Contributions were submitted. If You
               institute patent litigation against any entity (including a
               cross-claim or counterclaim in a lawsuit) alleging that the Work
               or a Contribution incorporated within the Work constitutes direct
               or contributory patent infringement, then any patent licenses
               granted to You under this License for that Work shall terminate
               as of the date such litigation is filed.
         
            4. Redistribution. You may reproduce and distribute copies of the
               Work or Derivative Works thereof in any medium, with or without
               modifications, and in Source or Object form, provided that You
               meet the following conditions:
         
               (a) You must give any other recipients of the Work or Derivative
                   Works a copy of this License; and
         
               (b) You must cause any modified files to carry prominent notices
                   stating that You changed the files; and
         
               (c) You must retain, in the Source form of any Derivative Works
                   that You distribute, all copyright, patent, trademark, and
                   attribution notices from the Source form of the Work,
                   excluding those notices that do not pertain to any part of
                   the Derivative Works; and
         
               (d) If the Work includes a "NOTICE" text file, as part of the
                   distribution, You must include a readable copy of the
                   attribution notices contained within such NOTICE file, in
                   at least one of the following places: within a NOTICE text
                   file distributed as part of the Derivative Works; within
                   the Source form or documentation, if provided along with the
                   Derivative Works; or, within a display generated by the
                   Derivative Works, if and wherever such third-party notices
                   normally appear. The contents of the NOTICE file are for
                   informational purposes only and do not modify the License.
                   You may add Your own attribution notices within Derivative
                   Works that You distribute, alongside or in addition to the
                   NOTICE text from the Work, provided that such additional
                   attribution notices cannot be construed as modifying the License.
         
               You may add Your own license statement for Your modifications and
               may provide additional grant of rights to use, copy, modify, merge,
               publish, distribute, sublicense, and/or sell copies of the
               Contribution.
         
            5. Submission of Contributions. Unless You explicitly state otherwise,
               any Contribution intentionally submitted for inclusion in the Work
               by You to the Licensor shall be under the terms and conditions of
               this License, without any additional terms or conditions.
               Notwithstanding the above, nothing herein shall supersede or modify
               the terms of any separate license agreement you may have executed
               with Licensor regarding such Contributions.
         
            6. Trademarks. This License does not grant permission to use the trade
               names, trademarks, service marks, or product names of the Licensor,
               except as required for reasonable and customary use in describing the
               origin of the Work and reproducing the content of the NOTICE file.
         
            7. Disclaimer of Warranty. Unless required by applicable law or
               agreed to in writing, Licensor provides the Work (and each
               Contributor provides its Contributions) on an "AS IS" BASIS,
               WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
               implied, including, without limitation, any conditions of TITLE,
               NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR
               PURPOSE. You are solely responsible for determining the
               appropriateness of using or reproducing the Work and assume any
               risks associated with Your exercise of permissions under this License.
         
            8. Limitation of Liability. In no event and under no legal theory,
               whether in tort (including negligence), contract, or otherwise,
               unless required by applicable law (such as deliberate and grossly
               negligent acts) or agreed to in writing, shall any Contributor be
               liable to You for damages, including any direct, indirect, special,
               incidental, or exemplary damages of any character arising as a
               result of this License or out of the use or inability to use the
               Work (even if such Contributor has been advised of the possibility
               of such damages).
         
            9. Accepting Warranty or Additional Liability. While redistributing
               the Work or Derivative Works thereof, You may choose to offer,
               and charge a fee for, acceptance of support, warranty, indemnity,
               or other liability obligations and/or rights consistent with this
               License. However, in accepting such obligations, You may offer only
               conditions consistent with this License.
         
            END OF TERMS AND CONDITIONS
         
            Copyright 2024-2026 Kynon J Benjamin
         
            Licensed under the Apache License, Version 2.0 (the "License");
            you may not use this file except in compliance with the License.
            You may obtain a copy of the License at
         
                http://www.apache.org/licenses/LICENSE-2.0
         
            Unless required by applicable law or agreed to in writing, software
            distributed under the License is distributed on an "AS IS" BASIS,
            WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
            See the License for the specific language governing permissions and
            limitations under the License.
License-File: LICENSE
Keywords: single-cell,snRNA-seq,reference mapping,label transfer,cell state,donor,validation,calibration,anndata,scanpy,bioinformatics
Author: Kynon J Benjamin
Author-email: kj.benjamin90@gmail.com
Requires-Python: >=3.11,<3.15
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Provides-Extra: all
Provides-Extra: dev
Provides-Extra: docs
Provides-Extra: plot
Provides-Extra: scvi
Requires-Dist: anndata (>=0.10,<1)
Requires-Dist: hypothesis (>=6.112,<7) ; extra == "dev"
Requires-Dist: ipykernel (>=6,<8) ; extra == "docs"
Requires-Dist: matplotlib (>=3.8,<4) ; extra == "plot"
Requires-Dist: myst-nb (>=1.1,<2) ; extra == "docs"
Requires-Dist: numpy (>=1.26,<3)
Requires-Dist: pandas (>=2.2,<3)
Requires-Dist: pytest (>=8.3,<9) ; extra == "dev"
Requires-Dist: pytest-cov (>=5,<8) ; extra == "dev"
Requires-Dist: ruff (==0.6.9) ; extra == "dev"
Requires-Dist: scanpy (>=1.10,<2)
Requires-Dist: scikit-learn (>=1.5,<2)
Requires-Dist: scipy (>=1.13,<2)
Requires-Dist: scvi-tools (>=1.2,<2) ; extra == "scvi"
Requires-Dist: sphinx (>=7,<9) ; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints (>=2,<4) ; extra == "docs"
Requires-Dist: sphinx-copybutton (>=0.5,<1) ; extra == "docs"
Requires-Dist: sphinx-rtd-theme (>=3,<4) ; extra == "docs"
Requires-Dist: statetrace-sc[dev,docs,plot,scvi] ; extra == "all"
Project-URL: Bug Tracker, https://github.com/heart-gen/StateTrace/issues
Project-URL: Documentation, https://statetrace.readthedocs.io/en/latest/
Project-URL: Homepage, https://github.com/heart-gen/StateTrace
Project-URL: Repository, https://github.com/heart-gen/StateTrace
Description-Content-Type: text/markdown

# StateTrace

[![CI](https://github.com/heart-gen/StateTrace/actions/workflows/ci.yml/badge.svg)](https://github.com/heart-gen/StateTrace/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/statetrace-sc.svg)](https://pypi.org/project/statetrace-sc/)
[![Python](https://img.shields.io/pypi/pyversions/statetrace-sc.svg)](https://pypi.org/project/statetrace-sc/)
[![Docs](https://readthedocs.org/projects/statetrace/badge/?version=latest)](https://statetrace.readthedocs.io/en/latest/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

StateTrace is a donor- and cohort-aware framework for determining whether cellular states
discovered in one dataset remain reproducible, supported, phenotypically preserved and suitable
for downstream inference in another. Mapping tools (scPoli, scANVI/scArches, Symphony, Azimuth,
CellTypist, or a user classifier) decide which state a cell belongs to; two-sample tests such as
[scXMatch](https://doi.org/10.1093/bioinformatics/btag706) and the E-distance decide whether two
sets of profiles differ. StateTrace takes that evidence and answers what neither was designed to
answer: **is a cellular state defined in one human cohort reproducible and transportable enough
to support inference in another?** It treats donors, not cells, as the replication unit; it
reports reproducibility across reference donors, recovery in an independent cohort, robustness
to the mapping backend, and whether the query version of a state is the same, a shifted version
of it or out of reference; and it gates which states may enter differential abundance,
pseudobulk expression or genetic analyses. It does not map cells and it is not a two-sample
test. It is tissue-, disease- and atlas-agnostic.

## Core capabilities

- **Grouped validation.** Leave-one-donor-out and leave-one-study-out holdouts with group x
  state recall, per-group balanced accuracy, and a rule-based `reproducibility` category
  (stable / moderate / fragile / insufficient); the conventional random cell split is one call
  away so the optimism of standard practice is visible as a gap.
- **Backend-agnostic.** `Predictions.from_scpoli / from_scanvi / from_symphony / from_azimuth /
  from_celltypist / from_obs` normalise any tool's labels and confidence; external mappers can
  also be refitted inside every holdout fold through `AnnDataBackend`. A kNN baseline is built in.
- **Cross-cohort transport validation.** Per state: reference reproducibility, query donors
  recovering it, uncertain and out-of-reference fractions, centroid shift, and a transport status.
- **Backend robustness.** `compare_backends` reports cell agreement, per-state concordance and
  whether the transport status agrees across mappers.
- **Shift and novelty diagnostics.** Absent, shifted and novel populations are distinguished
  by a donor x state pseudobulk distance in a frozen expression space built from the reference
  alone, calibrated against held-out reference donors (it decides the status), with
  reference-distance z-scores in the mapping latent as secondary evidence.
- **Two-sample engines (optional, reported).** `TransportConfig(two_sample=TwoSampleConfig(...))`
  runs the E-distance permutation test and, when installed, scXMatch's cross-match test for every
  state, at cell level and on donor profiles, inside the frozen space. Their p-values sit next to
  the status and never decide it: a cell-level test calls genuinely new, unshifted donors
  "different", which is a statement about cells, not about transporting the state.
- **Inference gating.** `supported`, `supported_with_caution`, `shifted`, `insufficient`,
  `absent`; `route.pseudobulk(states="supported")` and `route.state_abundance(states="supported")`.
- **Transport report.** Standardised CSV tables plus a Markdown summary of the evidence for every
  state, suitable to accompany a publication.
- **Frozen references** (checksummed, pickle-free `.statetrace` bundles) and **seeded
  simulations** of reference/query cohort pairs with planted stable, shifted, minority, absent and
  novel states.

## Installation

```bash
pip install statetrace-sc
```

The distribution is `statetrace-sc` on PyPI (PyPI reserves `statetrace` against the unrelated
`state-trace` project); the import name is `statetrace`.

| Extra | Adds | Needed for |
|---|---|---|
| `dev` | pytest, pytest-cov, hypothesis, ruff | tests and linting |
| `docs` | sphinx, myst-nb, sphinx-rtd-theme | building the docs |
| `plot` | matplotlib | `statetrace.plotting` (planned: the module raises `NotImplementedError` in 0.2.0; the extra reserves the dependency) |
| `scvi` | scvi-tools (pulls in torch) | the scANVI refit-per-fold backend (planned: `backends/scvi.py` raises `NotImplementedError` in 0.2.0; the extra reserves the dependency) |
| `all` | all of the above | |

torch is never a direct dependency; install the build that matches your machine before adding
the `scvi` extra. Python 3.11 to 3.14.

The optional scXMatch engine needs the `scxmatch` package, which installs only from conda
because it depends on graph-tool: `conda install scxmatch -c conda-forge -c bioconda`.

## Quickstart

Four tutorials, from a donor holdout to a transport report from two `.h5ad` files, are in the
[documentation](https://statetrace.readthedocs.io/en/latest/) and under `docs/tutorials/`.

```python
import statetrace as st
from statetrace.datasets import simulate_transport

ref_adata, query_adata, truth = simulate_transport(
    n_ref_groups=8, n_query_groups=6, n_states=5, n_latent=10,
    shifted_states={"S1": 9.0}, absent_states=["S4"], minority_states={"S3": 0.25}, seed=5,
)

# 1. reference: validate by donor, then freeze
ref = st.Reference(ref_adata, group_key="donor_id", state_key="cell_state", latent_key="X_latent").fit()
res = ref.leave_one_group_out()
res.state_metrics[["n_groups_reference", "median_group_recall", "group_recovery_rate", "reproducibility"]]
ref.freeze("my_reference.statetrace")

# 2. transport into an independent cohort (built-in kNN, or predictions from any tool)
route = st.validate_transport(ref, query_adata, group_key="donor_id")
route.states[["ref_reproducibility", "groups_recovered", "query_groups", "shift_level", "ood_fraction", "status"]]

preds = st.Predictions.from_scpoli(query_adata)          # after running scPoli yourself
route_scpoli = st.validate_transport(ref, query_adata, group_key="donor_id", predictions=preds)
st.compare_backends({"knn": route, "scpoli": route_scpoli}).summary()

# 3. gate downstream inference
route.supported_states
pb = route.pseudobulk(states="supported")                # donor x state pseudobulk AnnData
route.report("results/statetrace_report")
```

The transport table reads, for example:

| state | ref_reproducibility | groups_recovered | shift_level | ood_fraction | status |
|---|---|---|---|---|---|
| S0 | stable | 6/6 | low | 0.00 | supported |
| S1 | stable | 6/6 | high | 0.01 | shifted |
| S3 | insufficient | 6/6 | low | 0.00 | insufficient |
| S4 | stable | 0/6 | unknown | | absent |

With real data, point `latent_key` at a shared embedding (`"X_scVI"`) and `group_key` /
`state_key` at your `obs` columns; `study_key` enables leave-one-study-out.

## Documentation

[statetrace.readthedocs.io](https://statetrace.readthedocs.io/en/latest/): concepts, grouped
validation, transport validation and gating rules, the frozen reference format, API.

## Development

```bash
git clone https://github.com/heart-gen/StateTrace.git && cd StateTrace
uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -e ".[dev,docs]"
make lint PY=.venv/bin/python
make test PY=.venv/bin/python          # add test-slow for the larger synthetic run
make docs PY=.venv/bin/python
```

Benchmarks (scPoli, scANVI/scArches and Symphony as mapping backends, scConform as the
uncertainty benchmark, scXMatch and the E-distance as the distribution-shift benchmark, with
gated release criteria) and the manuscript live in separate repositories (`StateTrace-benchmarking`,
`statetrace-manuscript`).

## Citation

See [`CITATION.cff`](CITATION.cff). A preprint will be linked here once available.

## Funding

This work is supported by the National Institute on Minority Health and Health Disparities
(R00 MD016964).

## Limitations

- With the built-in kNN the latent space is fixed across holdout folds (fitted with every donor
  present), an optimistic bound on embedding transfer; per-fold refitting needs an
  `AnnDataBackend` (scANVI planned).
- Shift and novelty diagnostics need the query in the reference latent space; with labels-only
  predictions from an external tool, gating uses group support and confidence alone.
- The `reproducibility` and `status` rules are descriptive defaults to be predeclared, not
  biological claims.
- A population that the mapper places *between* two reference states, at distances no larger
  than those states' own held-out cells, is not seen by any distance score (mapper latent or
  frozen expression space); only its mixed reference neighbourhood is. Benchmark gate G1
  (withheld CD4 T cells among Tregs and CD8) failed on this case and the failure is carried as a
  stated limitation; the support-deficit score (`ood_score="support"`) is a sensitivity
  diagnostic, not the default. Read "no novel population" together with the unknown fraction.

