Metadata-Version: 2.4
Name: iyipada
Version: 0.1.0
Summary: Drift detection with calibrated thresholds instead of folklore ones
Author: Kenny Obidele
License: MIT
Project-URL: Homepage, https://github.com/Kenny0bi/iyipada
Keywords: drift,monitoring,mlops,statistics,psi,data-drift
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Provides-Extra: benchmarks
Requires-Dist: pandas>=2.0; extra == "benchmarks"
Dynamic: license-file

# iyipada

Drift detection where the threshold is calculated instead of quoted.

Production drift monitoring runs on numbers that came from a blog post. The
most common rule in the industry is "PSI above 0.10 means watch it, above 0.25
means act". Those numbers are stated as though they were properties of the
statistic. They are not. On data where **nothing has changed at all**, that
first rule fires 85% of the time at 100 samples per batch, and it is
mathematically incapable of firing at 1,000.

This library computes the threshold from the null distribution, so a 1%
false-alarm rate is a 1% false-alarm rate.

![The threshold everyone uses is not a constant](assets/ruler.svg)

*iyipada* is Yoruba for change.

## The three ways a drift monitor lies to you

Each of these is measured in this repository, not asserted.

**One. A fixed threshold means different things at different sample sizes.**
PSI under the null is distributed as `(1/n + 1/m) * chi2_{B-1}`, so it depends
on both sample sizes and the bin count. The industry rule depends on neither.
False-alarm rate of the "watch" rule on identically distributed data:

| samples per batch | 5 bins | 10 bins | 20 bins |
|---|---|---|---|
| 50 | 64.3% | 98.2% | **100.0%** |
| 100 | 29.5% | 85.1% | 100.0% |
| 200 | 4.1% | 37.0% | 96.0% |
| 500 | 0.0% | 0.3% | 17.6% |
| 1,000 and up | 0.0% | 0.0% | 0.0% |

The calibrated 1% threshold across that same grid runs from 1.4476 down to
0.0004, a span of about 5,400. One number cannot serve all of it.

**Two. Twenty features is twenty chances to be wrong.** Twenty clean features
tested independently at alpha = 0.05 produce at least one alarm on 73% of clean
batches. With Benjamini-Hochberg, 5%. Monitoring is a multiple-testing problem
before it is a drift problem.

**Three. Fifty-two weekly checks is not one check.** A test calibrated to 5%,
run weekly, is wrong at least once during a year with probability 93%. Getting
the single check right and then scheduling it re-creates the problem it solved.

![A monitor that runs every week is not one test](assets/year.svg)

## Using it

```python
from iyipada import Detector, Monitor

d = Detector(reference_values, statistic="psi", alpha=0.01)
r = d.check(this_weeks_values)

r.drifted     # bool, decided against a calibrated threshold
r.pvalue      # what you actually want to look at
r.threshold   # computed from your sample sizes, not quoted from anywhere
r.warning     # set when the batch is too small for the test to mean much
```

Nothing in the API accepts a raw divergence threshold, because a raw divergence
threshold cannot be interpreted without knowing the sample sizes and the bin
count.

For a table of features, with the multiple testing handled:

```python
m = Monitor(reference_dataframe_as_dict, alpha=0.05, correction="fdr")
result = m.check(live_dataframe_as_dict)
result.drifted        # the features that moved, after FDR control
```

For a monitor that runs on a schedule, stop re-testing and accumulate:

```python
from iyipada import sequential as sq

h, arl = sq.calibrate_run_length("cusum", target_arl=260)   # weekly checks
# h = 3.75, one false alarm every 4.8 years of weekly monitoring
c = sq.CUSUM(h=h)
c.update(this_weeks_statistic)   # True when the evidence has accumulated
```

## Which detector should you actually use

Wasserstein, in most cases. Every statistic below was first calibrated to the
same 1% false-alarm rate, so this compares power and nothing else. Detection
rate at 1,000 samples a side:

| | mean shift | variance | skew | discretisation | mixture | tail only |
|---|---|---|---|---|---|---|
| **Wasserstein** | 100% | 100% | 100% | 100% | **99.5%** | **15.7%** |
| PSI | 99.5% | 100% | 100% | 100% | 86.2% | 3.5% |
| Jensen-Shannon | 99.5% | 100% | 100% | 100% | 85.8% | 3.5% |
| chi-square | 99.5% | 100% | 100% | 100% | 85.7% | 3.5% |
| Kolmogorov-Smirnov | 100% | 99.8% | 88.7% | 100% | 83.0% | 2.3% |
| total variation | 98.3% | 100% | 99.8% | 100% | 59.7% | 2.8% |

![Which detector actually catches which change](assets/power.svg)

Wasserstein wins or ties everywhere, and it reports in the units of the
variable, so a result reads as "the distribution moved by 0.3 mg/dL" rather
than as an abstract divergence. PSI, the industry default, is beaten on the two
hardest cases.

### The blind spot

![The change only one detector sees](assets/blindspot.svg)

A small fraction of values jumping to the far tail is close to invisible. At 5%
contamination Wasserstein catches it every time, PSI 29% of the time and
Kolmogorov-Smirnov 18%. At 1.5% nothing is reliable.

The binned statistics fail because quantile binning puts the reference's entire
top decile into one bin, so moving mass from +2 to +6 does not change the
histogram at all. KS fails because a few per cent in the far tail cannot move
the largest vertical gap between two cumulative curves. Wasserstein survives
because it measures horizontal distance.

If rare extreme values are the failure you fear, monitor the tail directly with
a quantile or an exceedance count. A distributional distance is the wrong
instrument, however well calibrated.

## What is verified, and how

![The theory, checked against simulation](assets/nullcheck.svg)

The asymptotic null is not taken on faith. Simulated 95th percentile divided by
theory, across ten sample sizes and three bin counts, sits within 2% of 1.0 from
200 samples upward. It is 6 to 9% optimistic around n = 100 and it breaks down
at n = 50 with 20 bins, where each bin holds two or three points and the
chi-square approximation has nothing to stand on.

So the library uses the closed form above 300 samples and simulates the null
below it, rather than trusting one answer everywhere. With that switch, the
calibrated threshold holds between 0.0% and 2.2% actual false-alarm rate across
the entire grid, against 0% to 100% for the fixed rule.

![What those rules deliver](assets/alarms.svg)

The ringed cell is 112 samples at 10 bins. That is a real case from
[loom](https://github.com/Kenny0bi/loom), an earlier project of mine, where a
fixed 0.10 rule fired a spurious retrain on identically distributed data. It
fires there 78.6% of the time. That bug is why this library exists.

## The score, animated

![Why a fixed threshold cannot work](assets/null.gif)

The null distribution of PSI collapsing as the sample grows, while the fixed
rule stays exactly where it is. Every number in the animation comes from
`benchmarks/false_alarms.py`. (Source:
[assets/manim_null.py](assets/manim_null.py), video in
[assets/null.mp4](assets/null.mp4).)

## What is in here

- [iyipada/divergence.py](iyipada/divergence.py) the statistics: PSI,
  Jensen-Shannon, total variation, Kolmogorov-Smirnov, Wasserstein,
  chi-square. None of them decides anything.
- [iyipada/null.py](iyipada/null.py) the null distributions: analytic for PSI,
  chi-square and KS, permutation for everything else, plus
  `false_alarm_rate()` for measuring what a threshold you already use is
  actually doing.
- [iyipada/detect.py](iyipada/detect.py) `Detector` and `Monitor`, the latter
  with FDR or Bonferroni control.
- [iyipada/sequential.py](iyipada/sequential.py) CUSUM and Page-Hinkley
  calibrated to an average run length, alpha spending across a known horizon,
  and `detection_delay()` so a monitor that never false-alarms can be checked
  for being merely deaf.
- [benchmarks/](benchmarks/) the three studies behind every number above.

## Honest limits

- **Univariate only.** Correlation between features can shift while every
  marginal stays put, and nothing here would see it. Multivariate drift needs a
  different tool.
- **The permutation path is slow.** Wasserstein and total variation have no
  closed-form null here, so their thresholds are simulated. That is a few
  seconds per feature, cached per sample size.
- **Independence is assumed across checks.** The Sidak correction and the run
  length calibration both assume successive batches are independent. Weekly
  batches of the same seasonal process are not, and the true false-alarm rate
  will be somewhat higher than advertised.
- **Detecting drift is not detecting harm.** A feature can move a long way
  without the model caring, and can stay still while the relationship between
  it and the target rots. This library measures the first thing. Watching the
  second needs labels.
- **Below about 100 samples, nothing works.** The thresholds are honest there,
  which mostly means they are honestly enormous. A detector that cannot fire is
  correctly calibrated and useless. `Detector` says so in `r.warning`.

## Running it

```bash
pip install numpy scipy pandas

python tests/test_null.py         # 9 contracts on the null distributions
python tests/test_detect.py       # 10 on the detector and monitor APIs
python tests/test_sequential.py   # 8 on run lengths and detection delay

python benchmarks/false_alarms.py # the headline table, about 7 minutes
python benchmarks/power.py        # detector against drift type, about 4 minutes
python assets/make_visuals.py     # the six figures
```

27 tests. The interesting ones assert the findings themselves, so if the
industry thresholds ever stop being miscalibrated, the suite will say so.

## Papers

- B. Yurdakul (2018), *Statistical properties of population stability index*,
  Western Michigan University dissertation. Derives the asymptotic
  distribution, and states plainly that the 0.10 and 0.25 traffic-light values
  have no support in the literature.
- Bracher et al. and the wider proper-scoring literature for why a decision
  rule needs a stated error rate rather than a threshold.
- Page (1954), *Continuous inspection schemes*, Biometrika. The original CUSUM,
  and the reason average run length is the right quantity for a monitor that
  runs forever.
- Benjamini and Hochberg (1995), *Controlling the false discovery rate*, JRSS-B.
  What `Monitor` uses when you hand it fifty features.
