Metadata-Version: 2.4
Name: driftprobe
Version: 1.0.0
Summary: ML dataset drift detection with severity scoring and HTML reports
Author-email: RamMohan Reddy K  <ramku3639@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/Ramku3639/driftprobe
Project-URL: Documentation, https://github.com/Ramku3639/driftprobe/blob/main/README.md
Project-URL: Repository, https://github.com/Ramku3639/driftprobe
Project-URL: Issues, https://github.com/Ramku3639/driftprobe/issues
Project-URL: Changelog, https://github.com/Ramku3639/driftprobe/releases
Keywords: machine-learning,drift-detection,data-quality,mlops,monitoring
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: scipy>=1.10.0
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Dynamic: license-file

# driftprobe

**ML Dataset Drift Detection Library**

Monitors when incoming data starts behaving differently from training data — catching model degradation before it happens, with severity scoring and shareable HTML reports.

---

## What It Does

| Module | What it catches |
|---|---|
| PSI (numerical) | Distribution shift in continuous features |
| KS Test (numerical) | Statistical significance of distributional change |
| Chi-Square (categorical) | Shift in category frequencies; new/missing categories |
| Schema Drift | Columns added, removed, or modified (dtype, nullability, cardinality) |
| Severity Scorer | Feature-importance-weighted overall drift score |
| Concept Drift | Prediction distribution shift over time |
| HTML Report | Visual summary shareable with stakeholders |

---

## Installation

```bash
pip install driftprobe
```

From a clone, including the test extras:

```bash
pip install -e ".[dev]"
```

### Compatibility

| | Supported |
|---|---|
| Python | 3.8+ |
| pandas | 2.x and 3.x |
| numpy | 1.24+ |
| scipy | 1.10+ |

Column type detection uses the `pd.api.types` predicates rather than dtype-string
comparison, so text columns are classified correctly under both pandas 2 (`object`)
and pandas 3 (`str`). See Design Decisions for why this matters.

## Quick Start

```python
import pandas as pd
import numpy as np
from driftprobe import DriftWatcher

# 1. Load reference data (e.g., your training set)
reference_df = pd.DataFrame({
    "age": np.random.normal(35, 10, 1000),
    "income": np.random.normal(60000, 15000, 1000),
})

# 2. Initialize watcher with feature importance (optional but recommended)
watcher = DriftWatcher(
    reference_data=reference_df,
    feature_importance={"income": 0.7, "age": 0.3}
)

# 3. Check incoming production data
current_df = pd.DataFrame({
    "age": np.random.normal(45, 12, 500), # drifted!
    "income": np.random.normal(55000, 20000, 500), # drifted!
})

results = watcher.detect(current_df)

print(results["overall_score"]["severity_label"]) 
# e.g., "CRITICAL"
```

---

## Feature Importance Weighting

Pass your model's feature importances so drift on critical features is weighted higher:

```python
watcher = DriftWatcher(
    reference_data=train_df,
    feature_importance={
        "income": 0.40,       # most important
        "credit_score": 0.30,
        "age": 0.20,
        "gender": 0.10,
    }
)
```

Without this, all features are weighted equally — which will understate risk on high-impact features.

---

## Schema Drift

Distribution tests only compare columns that exist in both datasets. A column that
disappears — or arrives with a different dtype — breaks a model just as surely as a
shifted distribution, but no statistical test will ever see it.

`detect()` now returns a `schema` block:

```python
results = watcher.detect(current_df)

results["schema"]["added_columns"]      # ["signup_channel"]
results["schema"]["removed_columns"]    # ["credit_score"]
results["schema"]["modified_columns"]   # {"age": {"changes": [...], "tier": "breaking"}}
results["schema"]["schema_score"]       # 0.0 – 1.0, importance-weighted
results["schema"]["severity_label"]     # "low" | "moderate" | "severe"
```

A column counts as **modified** when any of these change:

| Change | Detected when | Tier |
|---|---|---|
| `dtype_change` | pandas dtype differs | breaking |
| `type_class_change` | numerical ↔ categorical flip | breaking |
| `nullability_change` | was never null, now is | breaking |
| `nullability_change` | null rate moved more than `null_rate_tolerance` | warning |
| `cardinality_shift` | categorical unique count moved by `cardinality_shift_factor`× | warning |

Schema drift keeps its **own** score. `overall_score` stays a pure distribution-drift
statistic, so existing thresholds keep their meaning.

Columns that flip type class are reported and **skipped** rather than crashing the run:

```python
results["features"]["age"]["skipped"]   # True
results["features"]["age"]["reason"]    # "type_class_change"
```

Added columns are recorded but not scored — there is no reference distribution to
compare them against.

### Failing the run on schema breakage

```python
watcher = DriftWatcher(reference_data=train_df, strict_schema=True)
watcher.detect(current_df)   # raises SchemaDriftError on removed/breaking columns
```

```python
from driftprobe import SchemaDriftError

try:
    results = watcher.detect(current_df)
except SchemaDriftError as e:
    print(e.schema_result["removed_columns"])
```

Schema comparison alone, without running any distribution tests:

```python
watcher.detect_schema_drift(current_df)
```

---

## Data Quality

Null values are dropped before testing — the statistical tests cannot consume NaN — but
driftprobe records every drop so you can see how much data a result rests on:

```python
results["data_quality"]["dropped_nulls"]["age"]
# {"reference_dropped": 10, "current_dropped": 4,
#  "reference_null_rate": 0.1, "current_null_rate": 0.04}

results["features"]["age"]["reference_rows_used"]   # 90
```

Drops above 5% are logged as warnings, and the HTML report gets a Data Quality section.
A column left with no non-null values is skipped with `reason="no_non_null_values"`.

---

## Logging

driftprobe logs through the standard `logging` module under the `driftprobe` namespace and
never calls `basicConfig`. To see warnings about dropped columns, null-heavy features, and
missing importance weights:

```python
import logging
logging.basicConfig(level=logging.INFO)
```

---

## Configuration

All arguments to `DriftWatcher`:

| Argument | Default | Purpose |
|---|---|---|
| `reference_data` | required | Training/baseline DataFrame |
| `feature_importance` | `None` | Column → weight. Equal weighting if omitted |
| `categorical_columns` | auto-detected | Override type routing |
| `psi_buckets` | `10` | Target quantile buckets for PSI |
| `ks_threshold` | `0.05` | KS test p-value threshold |
| `chi2_threshold` | `0.05` | Chi-square p-value threshold |
| `strict_schema` | `False` | Raise `SchemaDriftError` on removed/breaking columns |
| `null_rate_tolerance` | `0.1` | Null-rate change flagged as a schema warning |
| `cardinality_shift_factor` | `10.0` | Categorical cardinality fold-change flagged as a warning |
| `default_weight` | mean of given weights | Weight for features absent from `feature_importance` |
| `psi_smoothing` | `1e-6` | Floor on PSI bucket proportions |
| `kl_threshold` | `0.1` | Concept drift KL threshold (classification) |
| `max_classes` | `20` | Distinct predictions at or below which concept drift is classification |

`default_weight` deserves a note. When you supply importances for only *some* columns,
the unlisted ones default to the **mean of the weights you gave**, not `1.0`. With
weights summing to 1 across five features, a `1.0` default would make an unlisted
column outweigh every listed one combined. Pass `default_weight` explicitly to override.

### Methods

| Method | Returns |
|---|---|
| `detect(current_data)` | Full results dict |
| `detect_schema_drift(current_data)` | Schema block only, no statistical tests |
| `detect_concept_drift(reference_predictions, current_predictions)` | Concept drift dict |
| `generate_report(drift_results, ...)` | Path to the written HTML file |

The detector classes take further arguments not surfaced on `DriftWatcher` — notably
`NumericalDriftDetector(psi_score_cap=...)` and `CategoricalDriftDetector(...)`. Use them
directly on a single pair of Series when you need that level of control:

```python
from driftprobe.detectors import NumericalDriftDetector

NumericalDriftDetector(psi_buckets=20, psi_score_cap=0.3).detect(ref_series, cur_series)
```

---

## What `detect()` Returns

```python
{
    "features":         {...},   # per-column drift results
    "overall_score":    {...},   # importance-weighted summary
    "drifted_features": [...],   # names of columns flagged as drifted
    "schema":           {...},   # added / removed / modified columns
    "data_quality":     {...},   # what was dropped before testing
}
```

`overall_score` is always a dict, even when no column could be compared:

| Key | Meaning |
|---|---|
| `overall_score` | Weighted mean drift score, 0.0 – 1.0 |
| `severity_label` | `"low"` \| `"moderate"` \| `"severe"` |
| `contributions` | Per-feature `{"drift_score": s, "weight": w, "weighted_contribution": s * w}` |
| `skipped_features` | Columns excluded from the score |

### Numerical feature results

| Key | Meaning |
|---|---|
| `type` | `"numerical"` |
| `is_drifted` | True if PSI or KS flags drift |
| `drift_score` | 0.0 – 1.0. PSI rescaled so PSI ≥ 0.5 reads as maximal drift |
| `method` | `"PSI + KS"` |
| `psi` | Population Stability Index — never negative |
| `psi_severity` | `"stable"` \| `"moderate"` \| `"severe"` |
| `psi_buckets_used` | Actual buckets after collapsing duplicate quantile edges |
| `out_of_range_rate` | Fraction of current values outside the reference range |
| `ks_statistic`, `ks_p_value` | Two-sample Kolmogorov–Smirnov test |
| `reference_mean`, `current_mean`, `reference_std`, `current_std` | Descriptives |

`psi_buckets_used` below `psi_buckets` means the reference had too few distinct
values to fill the requested quantiles — worth knowing before trusting the PSI.

### Categorical feature results

| Key | Meaning |
|---|---|
| `type` | `"categorical"` |
| `is_drifted` | True if p < `chi2_threshold`, or categories appeared/vanished |
| `drift_score` | Cramér's V — an effect size, not `1 - p` |
| `method` | `"Chi-Square (contingency)"` |
| `chi2_statistic`, `chi2_p_value`, `chi2_dof` | Test of independence |
| `cramers_v` | Same value as `drift_score`, named explicitly |
| `low_expected_frequency` | True if any expected cell count < 5 — chi-square is unreliable here |
| `new_categories`, `missing_categories` | Set differences, nulls excluded |
| `reference_distribution`, `current_distribution` | Raw value counts |

### Skipped features

A column that cannot be compared is recorded rather than dropped:

```python
results["features"]["age"]
# {"type": "numerical", "skipped": True, "reason": "type_class_change", ...}
```

| `reason` | Cause |
|---|---|
| `type_class_change` | Column flipped numerical ↔ categorical |
| `no_non_null_values` | Nothing left after dropping nulls |
| `detector_error` | The detector rejected the values. `detail` carries the message |

Skipped columns are excluded from `overall_score` rather than counted as zero drift —
a column you could not measure is not a column that did not drift.

---

## Error Handling

```python
from driftprobe import DriftProbeError, InvalidInputError, SchemaDriftError
```

| Exception | Raised when |
|---|---|
| `DriftProbeError` | Base class — catch this to catch everything |
| `InvalidInputError` | Unusable data or configuration. Also subclasses `ValueError` |
| `SchemaDriftError` | `strict_schema=True` and the schema broke. Carries `.schema_result` |

`InvalidInputError` subclasses `ValueError`, so existing `except ValueError` handlers
around driftprobe keep working unchanged.

Validation is eager — bad thresholds, negative importance weights, empty frames, and
duplicate column names are rejected at construction rather than surfacing mid-run.
A single unusable column raises `InvalidInputError` internally but is caught per
column, so one bad column costs you that column, not the whole report.

---

## Concept Drift

Detect when your model's output distribution shifts, even when inputs look fine:

```python
concept = watcher.detect_concept_drift(
    reference_predictions=train_predictions,
    current_predictions=live_predictions,
)

print(concept["is_drifted"])       # True/False
print(concept["kl_divergence"])    # for classification
print(concept["mean_shift"])       # for regression
```

Predictions are routed automatically. Non-numeric predictions, or numeric ones with at
most `max_classes` distinct values, are treated as **class labels**; everything else is
treated as **regression output**. Check `prediction_type` to see which branch ran:

| Branch | Test | Keys |
|---|---|---|
| `"classification"` | KL divergence vs `kl_threshold` | `kl_divergence`, `kl_threshold`, `reference_distribution`, `current_distribution` |
| `"regression"` | Two-sample KS vs `ks_threshold` | `ks_statistic`, `ks_p_value`, `mean_shift`, `std_shift`, `reference_mean`, `current_mean` |

Both branches also return `is_drifted`, `drift_score`, `method`, and `prediction_type`.

An integer-coded target is the case to watch: 30 distinct integers routes as regression
under the default `max_classes=20`. Raise it if those are really class labels.

```python
watcher = DriftWatcher(reference_data=train_df, max_classes=50)
```

Null predictions are dropped before comparison, same as feature columns.

---

## HTML Report

```python
watcher.generate_report(
    drift_results=results,
    concept_drift_results=concept,  # optional
    output_path="drift_report.html",
    model_name="Credit Risk Model",
)
```

Opens a standalone HTML file — no server needed. Share directly with stakeholders.

---

## Severity Score Thresholds

| Score | Label | Action |
|---|---|---|
| 0.0 – 0.2 | Low | Monitor |
| 0.2 – 0.5 | Moderate | Investigate |
| 0.5 – 1.0 | Severe | Retrain |

---

## PSI Thresholds (Numerical Features)

| PSI | Meaning |
|---|---|
| < 0.1 | No significant drift |
| 0.1 – 0.2 | Moderate shift, monitor closely |
| > 0.2 | Significant shift, action required |

---

## Run Tests

```bash
pip install -e ".[dev]"
```

```bash
pytest -v
```

With coverage:

```bash
pytest --cov=driftprobe --cov-report=html
```

Test paths and discovery patterns are configured in `pyproject.toml`, so bare `pytest`
picks up the suite from the project root.

---

## Project Structure

```
driftprobe/
├── driftprobe/                  # Main package
│   ├── core.py                  # DriftWatcher — the entry point
│   ├── exceptions.py            # DriftProbeError, InvalidInputError, SchemaDriftError
│   ├── validation.py            # Input validation helpers
│   ├── logging_utils.py         # Package logger setup (NullHandler)
│   ├── detectors/
│   │   ├── base.py              # BaseDriftDetector ABC (Series vs Series)
│   │   ├── numerical.py         # PSI + KS test
│   │   ├── categorical.py       # Chi-square contingency + Cramér's V
│   │   ├── concept.py           # Prediction drift (KL / KS)
│   │   └── schema.py            # Added / removed / modified columns; infer_type_class
│   ├── scoring/
│   │   └── severity.py          # Importance-weighted scoring
│   └── report/
│       └── html_report.py       # Standalone HTML reports
├── tests/
│   ├── test_detectors.py        # Numerical + categorical detectors
│   ├── test_concept.py          # Concept drift
│   ├── test_schema.py           # Schema drift
│   ├── test_scoring.py          # Severity scoring
│   ├── test_report.py           # HTML rendering
│   └── test_edge_cases.py       # Degenerate data, validation, integration
├── examples/
│   ├── basic_usage.py           # Feature + schema + concept drift, printed
│   └── sample_report.py         # Generates a report and opens it
└── README.md
```

`SchemaDriftDetector` deliberately does not subclass `BaseDriftDetector`: that ABC's
contract compares two Series, while schema comparison operates on whole DataFrames.

---

## Design Decisions

- **Auto-detects column type** from pandas dtype — no manual tagging needed unless you want to override with `categorical_columns`. Booleans, categoricals, and text are categorical; numerics, datetimes, and timedeltas are numerical.
- **PSI + KS together** for numerical: PSI catches magnitude of shift, KS catches statistical significance. One test alone is insufficient.
- **Chi-square for categorical**: surfaces not just frequency drift but new and missing categories — both are operationally dangerous.
- **Cramér's V, not `1 - p_value`, is the categorical drift score.** A p-value measures confidence that *some* difference exists, not how large it is. With large samples a trivial shift gives p ≈ 0 and would report as near-maximal drift. Cramér's V is a sample-size-independent effect size in [0, 1].
- **Chi-square as a contingency test, not goodness-of-fit.** The reference set is itself a finite sample, so treating its proportions as known population parameters overstates significance. A 2 × k contingency test accounts for sampling error on both sides.
- **PSI keeps signed terms.** PSI sums `(q - p) · ln(q/p)`; both factors always share a sign, so every term is non-negative and the total cannot go negative. Wrapping the difference in `abs()` would *break* this — it would make each term `|q - p| · ln(q/p)`, which is negative wherever `q < p`, and would understate real drift. The formula is left as-is deliberately.
- **PSI bucket edges are open-ended.** The outermost quantile edges extend to ±inf so current values outside the reference range land in the end buckets instead of being silently dropped — they matter most exactly when the distribution has moved.
- **Type routing has one source of truth.** `detectors.schema.infer_type_class` decides categorical vs numerical, and `DriftWatcher` derives its column split from it. Anything else risks the schema detector reporting a `type_class_change` for a column the drift tests routed the other way.
- **Type checks use the pandas dtype API, not dtype strings.** Under pandas 3 a text column's dtype is `str`, not `object`, so `dtype == "object"` classifies every string column as numerical — which silently disables type-class and cardinality detection. `pd.api.types.is_*` covers both pandas 2 and 3.
- **Schema drift scores separately** from distribution drift, so `overall_score` thresholds keep their meaning.
- **Concept drift is separate** from feature drift — you can have stable inputs but a shifted prediction distribution (e.g. label shift).
- **Nulls are dropped but never silently** — every drop is recorded in `data_quality` and surfaced in the report.
- **Unmeasurable columns are skipped, not zeroed.** A column that could not be compared is excluded from `overall_score` rather than contributing 0.0 drift, which would dilute the score and read as reassurance.
- **One bad column does not lose the report.** Per-column failures are caught and recorded as skipped, so a single unparseable column costs you that column rather than the whole run.
- **HTML report is self-contained** — no CDN, no external dependencies. Works offline. All user-supplied column names are HTML-escaped.
