Metadata-Version: 2.4
Name: zidstats
Version: 0.2.0
Summary: Two-sample testing, ranking, and distributional diagnostics with ZID
Author-email: Hao Chen <hxchen@ucdavis.edu>
License-Expression: MIT
Project-URL: Homepage, https://github.com/haochen6/zidstats
Project-URL: Paper, https://arxiv.org/abs/2608.24881
Project-URL: Source, https://github.com/haochen6/zidstats
Project-URL: Issues, https://github.com/haochen6/zidstats/issues
Keywords: two-sample-test,nonparametric,permutation-test,kernel-test,graph-based-test,distributional-diagnostics
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Mathematics
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: scipy>=1.9
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Provides-Extra: embed
Requires-Dist: torch>=1.13; extra == "embed"
Requires-Dist: torchvision>=0.14; extra == "embed"
Dynamic: license-file

# zidstats — two-sample testing, ranking, and distributional diagnostics

**ZID** compares two samples through their within-sample similarity structure and
returns three complementary outputs: a permutation-calibrated **p-value** for equality
testing, a **score** for ranking departures under a fixed protocol, and **diagnostic
readouts** that report separate location-sensitive and dispersion-sensitive component
magnitudes and, when direction is identifiable, distinguish under- from over-dispersion.
It is designed to
remain responsive to distributional differences—including scale, shape, dependence, and
matched-moment structure—that can be compressed or missed by a single moment-based
distance.

**Paper:** Hao Chen (2026), [*What FID Hides: Detecting, Ranking, and
Diagnosing Deviations in Generative Evaluation*](https://arxiv.org/abs/2608.24881),
arXiv:2608.24881.

**Dissimilarity-native:** ZID accepts a finite, symmetric, nonnegative
pairwise dissimilarity with zero self-distance—no coordinates, metric, or triangle
inequality required. The off-diagonal entries must include a positive value and must not
all be constant. These format conditions do not guarantee a nondegenerate
permutation-null covariance: highly symmetric geometries can still produce a singular
or numerically ill-conditioned covariance, in which case the test raises an error rather
than reporting an unstable statistic. Object-valued observations such as networks,
trees, shapes, and distributions can therefore be compared once their pooled pairwise
dissimilarity matrix is supplied to `zid_test_from_dissimilarity`. Point matrices with a
`metric` are the convenience case.

The core package is application-agnostic; generative-model evaluation helpers
are provided separately in [`zidstats.generative`](https://github.com/haochen6/zidstats/blob/main/src/zidstats/generative/README.md).

## Install
```bash
pip install zidstats        # numpy, scipy only
```

## Quick start — one call
```python
import numpy as np
from zidstats import zid_test

rng = np.random.default_rng(20260718)
centers = np.zeros((3, 64))
centers[:, :2] = 1.5 * np.array([
    [1.0, 0.0], [-0.5, np.sqrt(3) / 2], [-0.5, -np.sqrt(3) / 2]
])

# Reference: three balanced feature-space modes.
x_mode = rng.choice(3, 300, p=[1/3, 1/3, 1/3])
# Comparison sample: the third mode has almost disappeared.
y_mode = rng.choice(3, 300, p=[0.475, 0.475, 0.05])
X = centers[x_mode] + 0.7 * rng.standard_normal((300, 64))
Y = centers[y_mode] + 0.7 * rng.standard_normal((300, 64))

res = zid_test(X, Y, seed=17)
print(res)
# ZID: score=25.68 | p=0.002 | statistically distinguishable (alpha=0.05).
# Significant distribution difference. Component magnitudes: W=26.37;
# D=1.01, direction not assigned (D-component p=0.314).
# These are sensitivity readouts, not a causal decomposition.
res.pvalue, res.statistic, res.reject, res.verdict
res.statistic, res.extra["score"]           # same protocol-dependent ranking score
res.extra["W_component_magnitude"]          # nonnegative S_W magnitude
res.extra["D_component_magnitude"]          # nonnegative S_D magnitude
res.extra["W_component_pvalue"]             # component-level permutation tail p_W
res.extra["D_component_pvalue"]             # p_D used by the diagnostic gate
res.extra["D_ZID"]                           # gated signed diagnostic, or None
res.extra["dispersion_net_score"]           # continuous signed visualization score
res.extra["dispersion_diagnosis"]           # formal diagnosis label
res.extra["arms"]                           # signed member-level Z_W and Z_D values
```
`zid_test` runs the complete ZID procedure on two sample matrices and returns
its permutation-calibrated p-value, ranking score, and diagnostic readouts in
one result.

The public `zid_score(X, Y)` and `zid_arms(X, Y)` utilities return only the
observed score or six observed member coordinates. They do not run the outer
label permutations and therefore do not return a calibrated p-value or test
decision.

The ZID `extra` schema uses component names only for aggregated quantities;
member-level coordinates are contained exclusively in `extra["arms"]`.
Protocol metadata are recorded alongside the readouts, including
`calibration="score_permutation"`, `n_perm`, bandwidths, and the RISE graph
settings.

### Main function inputs

```python
zid_test(
    X,
    Y,
    alpha=0.05,
    n_perm=499,
    seed=None,
    metric="euclidean",
    **kwargs,
)
```

| input | meaning | default |
|-------|---------|---------|
| `X` | Real-valued reference sample matrix with shape `(m, d)`; rows are finite observations and `m >= 4`. | required |
| `Y` | Real-valued comparison sample matrix with shape `(n, d)`; it must have the same number of columns as `X`, contain only finite values, and have `n >= 4`. | required |
| `alpha` | Nominal level for the equality decision; the same numerical threshold is used by the prespecified dispersion-diagnosis gate. | `0.05` |
| `n_perm` | Number of label permutations used to calibrate the complete six-arm ZID score. | `499` |
| `seed` | `None` or a nonnegative integer seed for reproducible label permutations; booleans are rejected. | `None` |
| `metric` | Pairwise dissimilarity: a SciPy `cdist` metric name or a callable `f(u, v) -> float`. Callable metadata store `module.qualified_name`; parameters captured by partials, closures, or callable-object state must be recorded separately with the evaluation protocol. | `"euclidean"` |
| `small_bandwidth_ratio` | GPK-small bandwidth as a multiple of the GPK-med bandwidth. | `0.175` |
| `sigma` | Optional positive absolute bandwidth for GPK-med; if omitted, ZID uses the median positive pooled pairwise distance. | pooled median |
| `k` | Optional RISE neighborhood parameter, with `N = m + n`. | `floor(N**0.65)` |
| `rank_type` | RISE graph construction: `"RgNN"` or `"RoNN"`. | `"RgNN"` |

The final four entries are optional expert settings passed through `**kwargs`.
For ordinary use, supply `X` and `Y` and leave the construction defaults fixed.
With `B = n_perm`, the smallest attainable Monte Carlo p-value is
`1 / (B + 1)`. Choose `B` so that this value is no larger than the intended
significance level; for example, testing at `alpha=0.001` requires at least
`n_perm=999`.

### Computational scope

ZID forms dense pooled dissimilarity, rank, and kernel matrices, so memory use is
quadratic in `N = m + n`. With `B = n_perm`, the current outer-permutation
implementation performs approximately `O(B * N**2)` work and therefore scales
roughly linearly with `B`. For pooled samples in the several-thousand range, assess
both memory and runtime on the intended hardware before committing to a large
permutation budget.

### Input forms: vectors, custom dissimilarities, or object data

For vector features, Euclidean dissimilarity is the default, but any SciPy `cdist`
metric name can be substituted without changing the ZID construction:

```python
res = zid_test(X, Y, metric="cosine")
```

An advanced callable metric receives two numeric rows `u` and `v`, each with
shape `(d,)`, and must produce finite, symmetric, nonnegative pairwise
dissimilarities with zero self-distance. It is for custom numeric-vector geometry,
not for passing arbitrary Python objects directly; the resulting pooled matrix is
validated before the test runs.

Networks, trees, shapes, distributions, and other object-valued observations do
not need vector coordinates. Compute a pooled pairwise dissimilarity matrix
`D`, place the reference objects first, and run ZID directly:

```python
from zidstats import zid_test_from_dissimilarity

# D[i, j] may be network distance, tree distance, shape distance, or another
# domain-specific dissimilarity; it need not be Euclidean or satisfy the triangle inequality.
res = zid_test_from_dissimilarity(D, n_x=len(reference_objects))
```

The same route supports complex-valued observations: first compute an
application-appropriate real-valued dissimilarity matrix. Raw complex sample
matrices are rejected rather than silently discarding their imaginary parts.

#### `zid_test_from_dissimilarity` inputs

```python
zid_test_from_dissimilarity(
    D,
    n_x,
    alpha=0.05,
    n_perm=499,
    seed=None,
    **kwargs,
)
```

| input | meaning | default |
|-------|---------|---------|
| `D` | Real-valued pooled pairwise dissimilarity matrix with shape `(N, N)`. It must be finite, symmetric, nonnegative, have a zero diagonal, contain a positive off-diagonal value, and not have a constant off-diagonal. Rows and columns `[:n_x]` contain the reference sample; `[n_x:]` contain the comparison sample. A matrix satisfying these format conditions can still yield a singular permutation covariance, which is reported as an error. | required |
| `n_x` | Number of reference observations and split point in `D`; the split must leave at least four observations in each sample. | required |
| `alpha` | Nominal level for the equality decision; the same numerical threshold is used by the prespecified dispersion-diagnosis gate. | `0.05` |
| `n_perm` | Number of label permutations used to calibrate the complete six-arm ZID score. | `499` |
| `seed` | `None` or a nonnegative integer seed for reproducible label permutations; booleans are rejected. | `None` |
| `small_bandwidth_ratio` | GPK-small bandwidth as a multiple of the GPK-med bandwidth. | `0.175` |
| `sigma` | Optional positive absolute bandwidth for GPK-med; if omitted, ZID uses the median positive off-diagonal entry of `D`. | pooled median |
| `k` | Optional RISE neighborhood parameter. | `floor(N**0.65)` |
| `rank_type` | RISE graph construction: `"RgNN"` or `"RoNN"`. | `"RgNN"` |

The final four entries are optional expert settings passed through `**kwargs`.
Unlike `zid_test`, this function has no `metric` argument because the pooled
dissimilarities have already been supplied in `D`.

No multidimensional scaling or Euclidean embedding is performed.

All four public ZID test paths apply the same duplicate check exactly once.
`DuplicateSampleWarning` identifies near-zero cross-sample dissimilarities that
suggest near-matching observations; repeated positive distance values alone do
not trigger it. For an otherwise valid ZID input, a zero cross-sample
dissimilarity always triggers the warning. For nonzero near-matches, the
heuristic suppresses the warning when comparable within-sample near-repetition
is pervasive.

**What the ZID score/test does.** ZID takes the signed
`Z_W`/`Z_D` arms from **RISE**, **GPK at the median bandwidth**, and **GPK at a small
(0.175×median by default) bandwidth**, maps their magnitudes through the conventional two-sided
normal-reference tail `2 * Phi_bar(abs(Z))`,
and flat-Simes-combines all six into an unbounded ranking score that is coordinatewise
monotone in the observed arm magnitudes. Larger values indicate a more extreme standardized
departure from the reference when sample size, representation, reference, and preprocessing
are held fixed. The returned p-value is the plus-one
label-permutation tail of that same score, so it preserves dependence among the arms and is
finite-sample valid under exchangeability.
This validity requires the representation, pooled dissimilarity, and tuning choices to be
fixed independently of the two-sample labels. If any of them are selected using those labels,
the complete selection procedure must be repeated within every relabeling.

**Reference orientation matters only for the signed diagnostics.** Swapping `X` and `Y`
leaves the ZID score and its permutation reference distribution unchanged, while
reversing the member-level `Z_D` arms, `dispersion_net_score`, and any supported
`D_ZID` readout; it also interchanges supported `under-dispersion` and
`over-dispersion` diagnoses. The reported Monte Carlo p-value is subject to
ordinary Monte Carlo variation from the sampled relabelings. Put the reference
first for a directional read.

**How to read the diagnostics.** `W_component_magnitude`
and `D_component_magnitude` are the nonnegative aggregated `W`- and `D`-component
magnitudes, denoted `S_W` and `S_D`, not additive parts of the score or a causal
decomposition. `D_component_magnitude` is always nonnegative, whereas
`dispersion_net_score` applies the sign of the sum of the three member-level `Z_D` arms and is
available continuously for visualization. With `X` designated as the reference, the
coherence-qualified `D_ZID` equals `-D_component_magnitude` for `under-dispersion`
and `+D_component_magnitude` for `over-dispersion`; it is undefined for every other
diagnosis and is returned as Python `None`. These labels have a dispersion interpretation
only when that interpretation is supported by the chosen dissimilarity and application.
Let `p_W = res.extra["W_component_pvalue"]` and
`p_D = res.extra["D_component_pvalue"]` denote the permutation tails of the
aggregated three-arm `W`- and `D`-component magnitudes. These are component-level
diagnostic summaries; the equality decision uses the permutation p-value of the complete
six-arm ZID score. In particular, `p_D` is a prespecified diagnostic gate rather than an
additional, separately error-controlled test. Only `p_D` is used by the formal
`dispersion_diagnosis`, which requires
`p_D <= alpha`. A member `j` is reference-active when its normal-reference tail
`2 * Phi_bar(abs(Z_D^(j))) <= alpha`. Both this gate and the reference-active
member threshold use the `alpha` supplied to `zid_test`. If `p_D > alpha`, the diagnosis is `not assigned`;
if `p_D <= alpha` but no member is reference-active, it is `ambiguous`; and conflicting
reference-active member signs are reported as `member-sign conflict`. All member arms
remain available for inspection; `D_reference_active_members` records the members in
the descriptive reference-active set.

For a quick single-member check without permutation calibration, use
**`method="rise"`**: RISE uses the analytic asymptotic χ²₂ reference for
`Z_W²+Z_D²`; it is deterministic and requires no permutation.
RISE still constructs dense `N × N` dissimilarity and rank matrices, so its
memory use remains quadratic in the pooled sample size.

**RISE tied-distance extension.** The RISE paper and the original R
implementation in `GraphRankTest` do not define the label-invariant averaging rule
used here for equal-distance neighbors. `zidstats` adds the following convention.
For `RgNN`, tied neighbors receive the mean position weight over the rank positions
occupied by their tie block. For `RoNN`, if a tie block crosses the
`k`-nearest-neighbor cutoff, each tied observation receives its inclusion
probability under uniform tie-breaking. For each unordered pair, the two
directional neighbor-inclusion probabilities are combined into the probability
that at least one observation selects the other; this probability weights the
edge's global similarity midrank. These extensions change nothing when neighbor
distances are distinct and remove dependence on pooled-row order when ties occur.
When equal distances actually alter the RISE weights, analytic standalone RISE emits
`RiseTieWarning`: the published asymptotic reference has not been established
specifically for this project-authored extension. Use
`calibration="permutation"` for a label-permutation calibration of that same
tie-averaged statistic. ZID uses its outer score permutation with the same
label-invariant weights.

## Routing — which member when
Use ZID when broad coverage is desired and the departure type is not prespecified. If the
application instead provides a specific geometric, computational, or discrete-data reason to
use a standalone specialist, route by geometry rather than by the smallest p-value:

| your situation | route → | why |
|----------------|---------|-----|
| broad coverage / departure type not prespecified | **ZID** | one six-arm ranking score, its calibrated permutation tail, and conditional diagnostic readouts |
| **quick check / avoid permutation calibration** | **RISE** (`method="rise"`) | single-member test using an analytic asymptotic χ²₂ reference; deterministic after the pairwise dissimilarities are computed |
| non-Euclidean / **manifold** / graph features | **RISE** | local rank graph; a global kernel bandwidth can smooth away dispersion differences on a manifold |
| plain **Euclidean / Gaussian**, prioritizing location/dispersion sensitivity | **GPK** | median-bandwidth kernel test with strong sensitivity to high-dimensional location and dispersion alternatives |
| need **member-level inspection** | **ZID** (inspect `arms`) | retains all six signed arms and reports whether the reference-active dispersion signs are coherent |
| repeated observations or duplicated rows | **GET-discrete**, **GPK**, **RISE**, or **ZID** | GET-discrete implements the repeated-observation graph extension; GPK handles ties directly; standalone RISE and the RISE member inside ZID use label-invariant tie handling |
| other tied pairwise dissimilarities | **GPK**, **RISE**, or **ZID** | GET-discrete is designed for duplicated-row equivalence classes; GPK handles ties directly, and RISE uses label-invariant tie handling |
| need **component-level diagnosis** | **ZID components** | read `W_component_magnitude`, `D_component_magnitude`, and `dispersion_diagnosis`; inspect `extra["arms"]` for member-level attribution |

## Methods

The advanced `two_sample_test(..., method=...)` dispatcher exposes RISE, GPK,
fGPK, GET, and GET-discrete as standalone tests. ZID combines RISE with median- and
small-bandwidth GPK; neither GET method is a ZID arm. The
dispatcher is not a second ZID entry point in the public workflow.
Every package-level test entry point returns the same `TestResult` object.
Formula-level implementations used for verification remain in the method
submodules and are not part of the package-level workflow.
For a pooled dissimilarity matrix, the matching router is
`two_sample_test_from_dissimilarity`.

```python
from zidstats import two_sample_test

rise = two_sample_test(X, Y, method="rise")
gpk_med = two_sample_test(X, Y, method="gpk", n_perm=499, seed=0)
gpk_small = two_sample_test(
    X, Y, method="gpk", bandwidth_ratio=0.175, n_perm=499, seed=0
)
fgpk = two_sample_test(X, Y, method="fgpk")
get = two_sample_test(X, Y, method="get")
get_discrete = two_sample_test(
    X, Y, method="get-discrete", approach="averaging"
)

# RISE, GET, and GET-discrete can instead use label-permutation calibration.
rise_perm = two_sample_test(
    X,
    Y,
    method="rise",
    calibration="permutation",
    n_perm=499,
    seed=0,
)
```

RISE, GET, and GET-discrete use their published analytic asymptotic
chi-square references by default. For any of these three methods, set
`calibration="permutation"` to calibrate the same quadratic statistic by label
relabeling. The selected calibration determines `TestResult.pvalue`; under
permutation calibration, `extra` also retains `pvalue_asymptotic` and the
asymptotic component tails. Because `n_perm` is unused by analytic RISE, GET,
GET-discrete, and fGPK, those calls may set `n_perm=0`; ZID, GPK, and explicitly
permutation-calibrated graph tests require a positive value.

For standalone GPK, omitting both bandwidth arguments uses the pooled median
bandwidth. Set `bandwidth_ratio=0.175` for `0.175 * sigma_med`, or pass an
absolute `sigma`; the two arguments are mutually exclusive.
`small_bandwidth_ratio` configures the fine-scale member inside ZID and does not
alter standalone GPK.

`method="get-discrete"` implements the repeated-observation graph extension of
Zhang and Chen (2022).
It first forms a graph on the `K` distinct pooled values, using the union of all
optimal MST edges when the graph is not unique, and then removes observation-level
tie-breaking in one of two published ways. The default `approach="averaging"`
averages the within-sample edge counts over all induced optimal graphs;
`approach="union"` uses their observation-level union. Ordinary `method="get"`
continues to use edge-disjoint MSTs and raises an error only when that requested
MST sequence is actually non-unique. It never selects an arbitrary optimal tree.
Set `k` explicitly to use the denser graph formed from the first `k`
nearest-neighbor-link (NNL) graphs; the GET-discrete default is `k=1`.

fGPK is a separate fast analytic test, not an analytic calibration of the GPK
quadratic statistic. Its reported statistic is the minimum of the one-sided
normal-reference tails for `Z_W_r1` and `Z_W_r2` and the two-sided tail for
`Z_D`; its p-value applies the published three-way Bonferroni adjustment. The
published fGPK specification uses the ordered pair of weights `1.2` and `0.8`;
consequently, swapping the two samples can slightly change its statistic and
may change decisions near the significance boundary.

| method | type | p-value | notable strengths | tuning (default) |
|--------|------|---------|-------------------|------------------|
| `zid`  | six-arm score/test (default) | **Monte Carlo permutation tail of the ZID score** | broad coverage across controlled departures | `sigma`=median, small=0.175×; `n_perm`=499 |
| `gpk`  | kernel | **Monte Carlo permutation tail of the GPK statistic** | location + dispersion (Euclidean) | `bandwidth_ratio=1` (median) |
| `fgpk` | fast kernel test | **analytic asymptotic three-way Bonferroni test** | location + dispersion | `bandwidth_ratio=1` (median) |
| `rise` | graph-induced rank | **analytic asymptotic χ²₂ (default) or Monte Carlo permutation** | shape + multimodality + dependence | k = ⌊N^0.65⌋ |
| `get`  | unique-MST edge-count | **analytic asymptotic χ²₂ (default) or Monte Carlo permutation** | scale + dependence | k = ⌊N^0.5⌋ |
| `get-discrete` | repeated-observation edge-count | **analytic asymptotic χ²₂ (default) or Monte Carlo permutation** | discrete or repeated data | averaging over observation-level optimal graphs; one NNL graph on K distinct values |

The family tests are **nonparametric** implementations of the published
mathematical definitions. ZID and GPK use label permutations; RISE, GET, and
GET-discrete default to their published analytic asymptotic nulls and also
offer permutation calibration. See
[`PROVENANCE.md`](https://github.com/haochen6/zidstats/blob/main/PROVENANCE.md)
for the implementation and licensing record.

**Attribution arms.** For RISE, GET, and GET-discrete, `Z_W` and `Z_D` are asymptotically
`N(0,1)` and asymptotically independent under their published null conditions,
giving the χ²₂ reference for `Z_W²+Z_D²`. They provide **two-coordinate attribution**:
`Z_W` records aligned within-sample displacement and is often location-sensitive;
`Z_D` records opposing displacement and is dispersion-sensitive in validated scale regimes. Under the
default analytic calibration, each has an asymptotic p-value (`2·Φ(−|·|)`);
under permutation calibration, `Z_W_pvalue` and `Z_D_pvalue` are two-sided
Monte Carlo tails. For GPK, the equal-weight `Z_W`
coordinate exposed as `extra["Z_W"]` is the standardized MMD-equivalent
coordinate and need not be `N(0,1)`; its Monte Carlo permutation p-value is
returned as `extra["Z_W_pvalue"]`. The GPK `Z_D` coordinate is asymptotically
`N(0,1)`; its Monte Carlo component tail is returned as
`extra["Z_D_pvalue"]` for optional inspection. The default GPK verdict follows
only the permutation p-value of the GPK quadratic statistic. The separate fGPK
test instead exposes the weighted, asymptotically `N(0,1)` coordinates `Z_W_r1`
and `Z_W_r2` together with `Z_D`.

### Note on "kernel" (≠ MMD)
GPK uses a kernel representation but is not the ordinary scalar MMD test. For any
`m,n >= 2` and a fixed pooled kernel matrix, the unbiased empirical squared MMD is a positive
multiple of the centered `W` statistic, so its permutation-standardized statistic is the
`Z_W` coordinate. GPK also retains the complementary signed `Z_D` coordinate. Under
high-dimensional dispersion
alternatives, opposing within-sample shifts can cancel in `Z_W` while reinforcing in `Z_D`.

## Cite
ZID: Hao Chen (2026), [*What FID Hides: Detecting, Ranking, and Diagnosing
Deviations in Generative Evaluation*](https://arxiv.org/abs/2608.24881),
arXiv:2608.24881.

Constituent methods: GPK (Song & Chen 2024, *Biometrika* 111(3):755–770), RISE (Zhou & Chen 2023,
COLT/PMLR 195), GET (Chen & Friedman 2017, *JASA* 112(517):397–409), and the
repeated-observation graph extension (Zhang & Chen 2022, *Statistica Sinica* 32:391–415).
