Metadata-Version: 2.5
Name: zekan
Version: 0.1.3
Summary: A local-first ML trust-audit tool for finding and measuring data leakage
Project-URL: Homepage, https://github.com/91Sakthivel/Zekan-Audit
Project-URL: Repository, https://github.com/91Sakthivel/Zekan-Audit
Author: 91Sakthivel
License: MIT
License-File: LICENSE
Keywords: data-leakage,data-science,machine-learning,ml-testing,model-validation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: <3.15,>=3.10
Requires-Dist: jinja2>=3.1
Requires-Dist: joblib>=1.3
Requires-Dist: numpy<3.0,>=1.24
Requires-Dist: pandas<4.0,>=2.0
Requires-Dist: psutil>=5.9
Requires-Dist: pyarrow>=14.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: scikit-learn<1.10,>=1.9
Requires-Dist: threadpoolctl>=3.5.0
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Description-Content-Type: text/markdown

# Zekan

**Find out if your model is secretly cheating — and how much it's cheating.**

When you build a machine-learning model, information can accidentally leak into it that it wouldn't have in the real world. The model looks great in testing, then falls apart in production. This is called **data leakage**, and it's one of the most common and expensive mistakes in machine learning.

Most tools just tell you *whether* there's a leak. Zekan tells you what actually matters:

- **How much** the leak is costing you (measured as a drop in your model's real score)
- **Which feature** is causing it
- **Whether it's real** or just statistical noise
- **What Zekan did *not* check** — so a clean result never gives you false confidence

Zekan runs entirely on your own machine. Your data never leaves your computer.

Validated on two independent real-world datasets from different domains — a
hospital-readmission benchmark and a 91,323-row mortgage-servicing panel — with
every prediction stated before the second run, not fitted after the fact. See the
[validation study](https://github.com/91Sakthivel/Zekan-Audit/blob/main/zekan/benchmark/results/DATASET2_FREDDIEMAC_VALIDATION_STUDY.md).

---

## Does Zekan fit my project?

Zekan works when **all** of these are true:

- You're predicting a **yes/no outcome** (e.g. will this customer churn? will this patient be readmitted?).
- Your data is a **table** (rows and columns — CSV or Parquet).
- The same thing appears **more than once over time** — e.g. one customer with several monthly snapshots. Zekan uses this time structure to detect leaks.

If your data has one row per item with no repeats over time, Zekan's core check won't apply yet.

---

## The idea in one line

Zekan trains your model the normal way, then trains an **honest** version with the leak removed, and reports the difference. That difference is the performance that was fake — the part that would vanish in production.

---

## Install

You'll need Python 3.10 or newer.

```bash
pip install zekan
```

Once installed, you can type `zekan` as a command.

To work on Zekan itself (running tests, editing the source):

```bash
git clone https://github.com/91Sakthivel/Zekan-Audit.git
cd Zekan-Audit

# Set up an isolated environment (recommended)
python -m venv .venv

# Turn it on:
#   Windows (PowerShell):
.venv\Scripts\Activate.ps1
#   macOS / Linux:
source .venv/bin/activate

# Install Zekan in editable mode
pip install -e ".[dev]"
```

---

## Try it in 60 seconds

No data of your own required. `zekan demo` runs a real `--tier audit` against a small built-in example with a genuine, measured leak — the exact same code path as `zekan audit`, just pointed at a fixture that ships inside the package:

```bash
zekan demo
```

Takes about 15 seconds and reports `UNCONFIRMED HIGH DAMAGE` on the fixture's `retention_outreach_flag` column — a real leak Zekan measured, not a canned result. For the full statistical confirmation (the permutation null), run `--tier certify` on your own data (see below).

---

## A small worked example

Let's walk through that example so the pieces make sense. The data (`fixture.csv`) is a tiny customer-churn table — a few columns, one row per customer per month:

| customer_id | snapshot_date | tenure_months | spend_last_30d | leaky_col | churned |
|---|---|---|---|---|---|
| 1001 | 2023-01-31 | 5 | 42.0 | ... | 0 |
| 1001 | 2023-02-28 | 6 | 38.5 | ... | 1 |
| 1002 | 2023-01-31 | 22 | 110.0 | ... | 0 |

Here's how each column maps to what Zekan asks you:

- **`customer_id`** → the **entity_id**. It's the same customer appearing across several months — that's the "recurring over time" structure Zekan needs.
- **`snapshot_date`** → the **prediction_time**. Each row is a monthly snapshot.
- **`churned`** → the **target**. The yes/no thing we're predicting (1 = churned, 0 = stayed).
- **`spend_last_30d`, `tenure_months`** → normal features. Fair game — they'd be known at prediction time.
- **`leaky_col`** → declared **forbidden**. In this example it's a stand-in for a column that gives away the answer, so we tell Zekan it must not be used.

The matching config (`zekan.yml`) simply writes those choices down:

```yaml
contract:
  entity_id: customer_id
  prediction_time: snapshot_date
  target: churned
  available_features_until: snapshot_date
  forbidden_after_prediction:
    - leaky_col
```

When you run the audit, Zekan checks whether that forbidden column (or any other) is secretly leaking, measures how much, and prints a verdict — for this clean example, **TRUSTED**. Swap in your own data and your own column names, and it's the same three steps below.

---

## Use it on your own data (3 steps)

### Step 1 — Let Zekan set up a config for you

```bash
zekan init --data your_data.csv
```

Zekan reads your file, lists your columns, and asks you a few questions. For each one you can type **the column name or its number** — whichever is easier. It asks for:

- **entity_id** — the column that identifies *one thing* tracked over time (e.g. `customer_id`).
- **prediction_time** — the column with the date/time of each row (e.g. `snapshot_date`).
- **target** — the yes/no thing you're predicting (e.g. `churned`).
- **available_features_until** — usually the same as your time column. It means "only information known up to this point is fair to use."
- **forbidden_after_prediction** — any columns that would *give away the answer* if the model used them. Leave empty if you're not sure; you can add them later. (You can list several, by name or number.)

This writes a file called `zekan.yml`.

> **Not sure what counts as "forbidden"?** A forbidden feature is anything that couldn't actually be known at the moment you'd make the prediction — for example, a field that's only filled in *after* the outcome happens. If in doubt, leave it empty and run the audit; Zekan also screens for leaks you didn't declare.

### Step 2 — Run the audit

```bash
zekan audit --data your_data.csv --config zekan.yml
```

No `--tier` needed — this runs `--tier audit` by default. It computes the real `fl` leakage decomposition, but it does **not** run the permutation null, so it can't yet statistically confirm a measured leak is real rather than a one-off fold split. Add `--tier certify` for that confirmation.

### Step 3 — Read the verdict

Zekan gives you one of four results:

| Verdict | What it means |
|---|---|
| **TRUSTED** | No leakage found *in what you declared and what Zekan checks*. (Not a guarantee the whole pipeline is perfect — Zekan tells you its scope.) |
| **RISKY** / **FAILED** | A real, confirmed leak — with how much it's costing you and which feature is responsible. |
| **INCONCLUSIVE** | The result wasn't stable enough to trust. |

---

## What the verdict is measuring

- **`fixable_leakage` (`fl`) = AUC(B) − AUC(C)** — model B keeps every
  column you declared forbidden; model C has them removed. Both are
  scored on the identical, honest, forward-in-time fold split, so the
  only thing that differs is whether the model got to see those columns.
  The gap is the accuracy that would evaporate in production, and Zekan
  attributes it back to whichever feature(s) caused it.
- **Two different questions, not one number.** `p` / `NSL` (from the
  permutation null) answer *"is this leak distinguishable from noise?"* —
  that's statistical power, not damage. `fl` answers *"how much is it
  costing you?"* — that's severity. A confidently-real leak can still be
  small; a large leak can still be statistically unconfirmed at small
  sample sizes. Don't read NSL as a severity score — it isn't one.
- **`evidence_level`: `SCREENED` / `AUDITED` / `CERTIFIED`** — how much of
  the check actually ran, separately from the verdict itself. `--tier
  scan` → `SCREENED` (structural checks only, no `fl`). `--tier audit` →
  `AUDITED` (`fl` is computed, but no permutation null ran, so nothing
  was confirmed real). `--tier certify` → `CERTIFIED` (the full check,
  including the null). Because `AUDITED` never ran the null, it can never
  resolve to a bare `TRUSTED` — a low `fl` at that tier reports
  `UNCONFIRMED_LOW_DAMAGE` instead, so a clean-looking audit-tier result
  can't be mistaken for a confirmed-clean one.
- **`--with-precision`** (audit tier only) turns on the bootstrap
  confidence interval around `fl` at `--tier audit`; it's off there by
  default for speed. `--tier certify` always computes it.

See [`METHODOLOGY.md`](https://github.com/91Sakthivel/Zekan-Audit/blob/main/METHODOLOGY.md) §1–3 for the full A/B/C
decomposition and the exact formulas.

---

## Getting a machine-readable result (for automation)

To use Zekan in a CI pipeline or script, add `--json`:

```bash
zekan audit --data your_data.csv --config zekan.yml --json
```

This prints structured JSON (the human-readable text goes to the error stream instead), so another tool can read the verdict automatically.

---

## Commands at a glance

| Command | What it does |
|---|---|
| `zekan init` | Ask a few questions and write a config for you. |
| `zekan audit` | Run the leakage audit and print a verdict. |
| `zekan diff` | Compare two audits to see if leakage got better or worse. |
| `zekan report` | Produce a report from an audit. |
| `zekan explain <feature>` | Show everything Zekan already knows about one column, from a saved audit JSON (`--json`) or a fresh run (`--config`). No new measurement. |

Useful `zekan audit` options (run `zekan audit --help` for all of them):

| Option | What it does |
|---|---|
| `--json` | Machine-readable output for automation. |
| `--stability` | Rerun the permutation null across N seeds; downgrade to INCONCLUSIVE if the verdict depends on which null draws were sampled. Does not reseed model fitting -- see "Honest limitations" below. |
| `--dry-run` | Check your config is valid without running the full audit. |
| `--estimator NAME` | Choose the model type (`histgb` by default). |
| `--sep CHAR` | CSV delimiter, e.g. `--sep ';'`. Zekan also auto-detects a non-comma delimiter and tells you what it found, but `--sep` is the reliable override. |
| `--from-period VALUE` | Restrict the audit to rows at or after this value in your `prediction_time` column — useful for re-running on recent data without rebuilding a whole frame. |

---

## Performance: what each tier costs, and why

`zekan audit` — the default, `--tier audit` — fits two models per run: a
real A/B/C leakage decomposition on the honest, forward-in-time fold
split. It does **not** run the permutation null, so it never
statistically confirms a leak is real rather than a one-off fold split.

`--tier certify` adds that confirmation on top of the same decomposition:
a permutation null (2 channels × 100 draws), fitting roughly **200
models** per run in total. That cost is intrinsic to what certify
measures, not an oversight — eight separate attempts to cut it were tried
and refuted against real data (a cheaper model changed a verdict;
subsampling swung the detection margin; several others failed their own
correctness checks before ever reaching a speed measurement). See
[`METHODOLOGY.md`](https://github.com/91Sakthivel/Zekan-Audit/blob/main/METHODOLOGY.md) §6 for the full cost model, every
figure's actual measurement scale (91,323 and 944,355 rows are real
measurements; anything larger is a labeled projection, not a
measurement), and why `--jobs` (and therefore wall-clock) is
hardware-dependent by design.

`--tier scan` is lighter still: structural checks only, no model fits at all.

**Results, not just cost, were checked across hardware.** The same
944,355-row run produced byte-identical `fl` and bootstrap CI values on a
laptop and on a GCP Sapphire Rapids instance, despite NumPy/SciPy
dispatching a different OpenBLAS kernel on each. One CPU pair, not a
universal guarantee — see
[`zekan/benchmark/results/CLOUD_CROSS_HARDWARE_REPRODUCIBILITY_FINDINGS.md`](https://github.com/91Sakthivel/Zekan-Audit/blob/main/zekan/benchmark/results/CLOUD_CROSS_HARDWARE_REPRODUCIBILITY_FINDINGS.md)
for the full record.

**The tested stack**, exactly, across both 944,355-row runs above:
Python 3.12 (3.12.3–3.12.4), NumPy 2.4.6–2.5.2, SciPy 1.18.0–1.18.1, and
scikit-learn **1.9.0 exact** on both machines — scikit-learn is the
version-sensitive one (see `pyproject.toml`'s own `<1.10` upper bound),
so this is the range actually exercised, not merely permitted.

**Python 3.14 runs Zekan correctly.** 0.1.0 was confirmed there with a
full fixture audit (`TRUSTED` verdict, every preflight check passing).
0.1.2 was independently re-confirmed installing and running correctly
there too (preflight checks and `--tier scan`, verified for this release
— a full `--tier certify` run was not repeated here, since certify is
never run outside an explicit release gate). Permitted via
`requires-python < 3.15`, but not part of the tested stack above — none
of the byte-identity measurements this project gates on have been run on
Python 3.14.

---

## Honest limitations

Zekan is under active development, and we'd rather tell you its limits than oversell it:

- It's been validated on real data (a large public hospital-readmission dataset, plus a second, independent mortgage-servicing dataset) through experiments designed in advance — but broader testing across more datasets is still in progress.
- **The severity thresholds (`warn_floor`/`fail_floor`) were calibrated on one dataset, and a second-dataset test found they do NOT transfer as absolute numbers.** The same injected leak that Zekan correctly detected and correctly ranked as more severe than a clean control on both datasets only crossed `warn_floor` on the dataset the thresholds were calibrated against — on the second dataset, a real, confirmed leak topped out at roughly a sixth of `warn_floor`, because that dataset's honest model ceiling left far less headroom for any leak to register in. Detection and relative ordering (clean vs. leaky) held on both; the absolute floor did not. See the [validation study](https://github.com/91Sakthivel/Zekan-Audit/blob/main/zekan/benchmark/results/DATASET2_FREDDIEMAC_VALIDATION_STUDY.md) §2 and [`METHODOLOGY.md`](https://github.com/91Sakthivel/Zekan-Audit/blob/main/METHODOLOGY.md) §4 for the full account.
- It needs data where the same entity recurs over time; it doesn't fit ordinary one-row-per-item datasets yet.
- Version 1 focuses on yes/no predictions on tabular data.
- **`--stability` only checks whether the verdict depends on which permutation-null draws were sampled.** It does not measure model-fitting variance, n-sensitivity, or cross-dataset drift. A dedicated model-refit-variance check was designed and measured, then deliberately not shipped: for the default estimator at the scale of every real dataset this project has tested, refitting under a different random seed changed nothing at all — not "stable," but not a check that has anything to detect at that scale either. See [`STABILITY_CI_PREREGISTRATION.md`](https://github.com/91Sakthivel/Zekan-Audit/blob/main/zekan/benchmark/results/STABILITY_CI_PREREGISTRATION.md).

Every verdict Zekan gives includes a note about what it did and did not check. That honesty is the point — see [`COVERAGE_MAP.md`](https://github.com/91Sakthivel/Zekan-Audit/blob/main/COVERAGE_MAP.md) for the full, itemized account of what Zekan checks, what it doesn't, and the known boundaries of each.

---

## License

MIT — see [LICENSE](https://github.com/91Sakthivel/Zekan-Audit/blob/main/LICENSE). Free to use, modify, and share.
