Metadata-Version: 2.4
Name: prepro-auto
Version: 2.0.0
Summary: AI-assisted, human-in-the-loop preprocessing for tabular and time-series data — profile, clean, transform, forecast and export, with a reproducible pipeline, from a notebook or the web.
Author: Shivanshu Pandey
License-Expression: MIT
Project-URL: Homepage, https://github.com/Chilliflex/prepro_auto
Project-URL: Repository, https://github.com/Chilliflex/prepro_auto
Project-URL: Documentation, https://github.com/Chilliflex/prepro_auto#readme
Project-URL: Issues, https://github.com/Chilliflex/prepro_auto/issues
Project-URL: Releases, https://github.com/Chilliflex/prepro_auto/releases
Keywords: data-preprocessing,data-cleaning,machine-learning,pandas,data-quality,feature-engineering,etl,data-science,time-series,forecasting,renewable-energy
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.110
Requires-Dist: uvicorn[standard]>=0.29
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: pydantic>=2.6
Requires-Dist: pydantic-settings>=2.2
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: pandas>=2.0
Requires-Dist: numpy>=1.24
Requires-Dist: scikit-learn>=1.3
Requires-Dist: scipy>=1.10
Requires-Dist: reportlab>=4.0
Requires-Dist: pyarrow>=14.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: python-dotenv>=1.0
Requires-Dist: psutil>=5.9
Requires-Dist: nest-asyncio>=1.5
Provides-Extra: groq
Requires-Dist: groq>=0.11; extra == "groq"
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == "anthropic"
Provides-Extra: gemini
Requires-Dist: google-generativeai>=0.7; extra == "gemini"
Provides-Extra: mistral
Requires-Dist: mistralai>=1.0; extra == "mistral"
Provides-Extra: ai
Requires-Dist: groq>=0.11; extra == "ai"
Requires-Dist: openai>=1.40; extra == "ai"
Requires-Dist: anthropic>=0.40; extra == "ai"
Requires-Dist: google-generativeai>=0.7; extra == "ai"
Requires-Dist: mistralai>=1.0; extra == "ai"
Provides-Extra: timeseries
Requires-Dist: statsmodels>=0.14; extra == "timeseries"
Provides-Extra: timesfm
Requires-Dist: timesfm[torch]<4,>=2.0.2; extra == "timesfm"
Provides-Extra: cfsv2
Requires-Dist: eccodes>=1.6; extra == "cfsv2"
Provides-Extra: hosting
Requires-Dist: psycopg2-binary>=2.9; extra == "hosting"
Requires-Dist: boto3>=1.34; extra == "hosting"
Requires-Dist: alembic>=1.13; extra == "hosting"
Requires-Dist: celery>=5.3; extra == "hosting"
Requires-Dist: redis>=5.0; extra == "hosting"
Provides-Extra: ml
Requires-Dist: lightgbm>=4.0; extra == "ml"
Requires-Dist: xgboost>=2.0; extra == "ml"
Requires-Dist: catboost>=1.2; extra == "ml"
Requires-Dist: skl2onnx>=1.16; extra == "ml"
Requires-Dist: onnx>=1.15; extra == "ml"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Dynamic: license-file

# PrePro Auto

[![PyPI version](https://img.shields.io/pypi/v/prepro-auto.svg)](https://pypi.org/project/prepro-auto/)
[![Python](https://img.shields.io/pypi/pyversions/prepro-auto.svg)](https://pypi.org/project/prepro-auto/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Tests](https://img.shields.io/badge/tests-1013%20passing-brightgreen.svg)](https://pypi.org/project/prepro-auto/)
[![Downloads](https://static.pepy.tech/badge/prepro-auto)](https://pepy.tech/project/prepro-auto)

**AI-assisted tabular data preprocessing with human-in-the-loop control.**

Profile, clean, transform, and export any tabular dataset — from a Jupyter notebook or a local web UI — with every step undoable, auditable, and reproducible. The same engine drives both interfaces, so results are identical wherever you call it from.

```bash
pip install prepro-auto
```

> Author: [Shivanshu Pandey](https://github.com/Chilliflex) · Package: [pypi.org/project/prepro-auto](https://pypi.org/project/prepro-auto/)

> **About this repo.** `pip install prepro-auto` is the supported way to get PrePro Auto — that's where the working code ships from and stays up to date. This repository hosts the documentation, the full user guide, screenshots, and a runnable example notebook.

---

## See it in action

Watch a dataset go from messy to model-ready — review each fix, approve it, and watch the issue count drop to **0**.

[![PrePro Auto Demo](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/complete_workflow.gif)](https://github.com/Chilliflex/prepro_auto/blob/main/examples/Bengaluru_House_Prices_Workflow.mp4)

**▶ Click the GIF to watch the full walkthrough in HD on GitHub.**

📦 **GitHub Repository:** https://github.com/Chilliflex/prepro_auto

🎥 **Demo Video:** https://github.com/Chilliflex/prepro_auto/blob/main/examples/Bengaluru_House_Prices_Workflow.mp4

> **Performance shown in the GIF:** When every AI recommendation is applied automatically (**AI-only mode**), the demonstrated workflow achieves **97.2% accuracy**. Enabling **Human-in-the-Loop (HITL)** review—where users approve or override recommendations before execution—raises the final accuracy to **99.9%** on our benchmark evaluations.

---

## Contents

- [Quickstart](#quickstart) — get going in 30 seconds
- [Ways to give PrePro Auto your data](#ways-to-give-prepro-auto-your-data) — 5 from notebook, 3 from web UI
- [1. Input functions (notebook)](#1-input-functions-notebook) — how to load data into a session
- [2. Preprocessing functions](#2-preprocessing-functions) — clean, transform, visualize
- [3. Output functions](#3-output-functions) — DataFrames, files, audit PDFs, pipelines
- [4. Train functions (optional ML layer)](#4-train-functions-optional-ml-layer) — fit, compare, predict, export models
- [AI providers](#ai-providers-optional) — optional, 5 providers supported
- [REST API reference](#rest-api-reference)
- [Documentation](#documentation)

---

## Quickstart

**One call, no browser (fastest — for the impatient):**

```python
import prepro_auto

result = prepro_auto.quickclean("your_data.csv", target="label")
df = result.df          # cleaned, model-ready DataFrame
print(result)           # what was applied vs. left for review
```

`quickclean` runs the whole engine headlessly and applies every **confident** decision, leaving the uncertain ones for you (raise the bar or go full auto-pilot with `apply_all=True`). Then feed `result.df` straight into `model.fit(X, y)`.

**Notebook + visual workbench (recommended for careful cleaning):**

```python
import prepro_auto

# Point at a file, auto-detects encoding (handles Latin-1, cp1252, BOM)
session = prepro_auto.launch_file(r"C:\path\to\your_data.csv")
# Click the printed http://127.0.0.1:8721/workbench?job=... link
df = session.current()        # pull the UI-edited DataFrame back into the notebook

df["amount_per_item"] = df["amount"] / df["quantity"]
session.update(df)            # push it back — the open tab refreshes itself
```

The notebook and the browser tab share **one live dataset**, in both directions:
`current()` returns whatever you last executed in the UI, and `update()` appears
in the open tab within a couple of seconds — no reload, no re-launch. Call
`session.status()` any time to see which version is live and whether the UI is
holding decisions you approved but haven't executed yet.

> **Snapshot vs. live.** `df = session.current()` hands you an ordinary
> DataFrame — a snapshot taken at that moment. Nothing that happens in the
> browser afterwards can reach into that local variable, so `df.head()` keeps
> showing what it showed when you assigned it. Call `session.current()` again
> for fresh data, or use **`session.df`** (the session's own live property —
> not your local `df`), which re-reads on every access:
>
> ```python
> df = session.current()        # snapshot — stays as-is
> session.df.head()             # live — always the current version
> session.df.shape
> ```

**Web UI (recommended for analysts):** open Command Prompt (not Jupyter) and run:

```bash
prepro_auto
```

Then open `http://127.0.0.1:8000/workbench` and drag-drop a file.

> **Note:** `prepro_auto` typed inside a Jupyter cell just prints the module object — it doesn't start a server. The CLI command runs from a terminal only. Inside a notebook, use `prepro_auto.launch_file(path)` or `prepro_auto.launch(df)` instead.

---

## Screenshots

### Step 1 — Upload
Drop any CSV, Parquet, Excel, or JSON file. Encoding and delimiter are auto-detected. The sidebar shows live RAM-aware upload limits for your machine.

![Upload](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Upload.png)

### Step 2 — Profile
Per-column semantic type inference, missing rates, cardinality, and a 0–100 dataset quality score — all in one pass, before any data is changed. **Profiling runs automatically** when you open this step (no button to click).

**What the quality score is (and isn't).** It starts at 100 and subtracts weighted penalties across six dimensions — completeness (missing cells, empty and mostly-empty columns), validity (columns mixing numbers and text, untypeable columns), consistency (categories that differ only by case or whitespace), uniqueness (duplicate rows, constant columns), distribution (IQR-fence outlier prevalence), and sufficiency (very small samples). The Profile step shows the full breakdown: every factor, what it measured, and what it cost.

Treat it as a **progress indicator for one dataset's cleaning session** — "did what I just did help?" — which is exactly how the before/after delta uses it. It is **not** a validated metric: the weights are considered judgements, not constants calibrated against downstream model performance, and each factor is capped, so past a point more damage stops lowering the score. Two different datasets both scoring 85 are not therefore equally good, and the number should not be reported as a measurement in a paper without independent validation. Use the per-dimension breakdown, which says what is actually wrong, rather than the scalar alone.

![Profile](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Profile.png)

### Step 3 — View data
Live table toggling between Original (raw) and Current (cleaned). Version label, row/column count, and quality score update after every operation.

![View data](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/View%20Dataset.png)

### Step 4 — Clean (human-in-the-loop)
**Selecting a stage tab (Missing Values, Outliers, Scaling, Correlation, Encoding) runs it automatically — no "Run stage" button**; opening this step auto-runs Missing Values first. Each stage generates per-column decision cards. Every card shows **contextual alternatives with their own confidence score and a one-line reason**, and the recommended option always leads. Approve, Override (your choice sticks), Skip, or Drop. A **live issue-count badge** on each stage tab updates after you execute — watch it drop to `0 ✓` — and executing a stage **auto-advances to and runs the next one** so its cards are ready immediately. Prefer one click? **⚡ Quick clean** runs all five stages and applies every recommendation at once (fully undoable) — the visual equivalent of `prepro_auto.quickclean(..., apply_all=True)`.

![Clean](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Clean.png)

### Step 5a — Preset Operations
18 built-in transforms (rename, cast, filter, merge, math, string ops, regex, group-aggregate, sort, dedup, and more). Every operation is a single undoable version.

![Preset Operations](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Transform_Preset_Operation.png)

### Step 5b — Expression Editor
Write the right-hand side of any pandas expression and name the column it lands in — e.g. expression `df["revenue"] - df["cost"]`, target `profit`.

**Preview (dry run) first.** It evaluates against your current data and shows the shape delta, added/removed columns, dtype changes, and a before/after sample — committing nothing. Then apply, as one undoable version.

What works: arithmetic and comparisons, `.str` methods, `.astype`, `.fillna`, `.clip(lower=0)`, `.round(2)`, `.where`, `.map({...})`, `pd.to_numeric(df["x"], errors="coerce")`, `pd.to_datetime`, `pd.cut`, `np.log1p`/`np.where`, and `df.groupby("g")["v"].transform("mean")`. Keyword arguments are fine, and so is any column name — validation reads the **parsed syntax tree**, not the raw text, so a column called `important_flag` or `opening_price` is not mistaken for `import`/`open`.

Blocked: imports, dunder access, lambdas and comprehensions, `.apply`/`.applymap`/`.pipe` (they execute arbitrary callables), `.eval`/`.query`, and every `read_*`/`to_*` file operation. See [SECURITY.md](SECURITY.md) for the mechanism and threat model — this is a **single-user, local-machine** trust boundary, not a hostile-multi-tenant sandbox.

![Expression Editor](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Transform_Experssion(Advance).png)

### Step 5c — AI Assistant
Describe a transform in plain English. The AI proposes the pandas code, shows a preview, and waits for your confirmation before touching the data.

![AI Assistant](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Transform_AI_Assistant.png)

### Step 5d — Visualization
Histograms, bar charts, scatter plots, and condition-based metrics — all rendered live against the current dataset version.

![Visualization](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Transform_Visualization.png)

### Step 5e — Before & After Dashboard
KPI tiles comparing raw upload to current cleaned version: quality score delta, per-column type changes, and data samples side-by-side.

![Before and After Dashboard](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Transform_Dashboard(Before%20and%20After).png)

### Step 5f — Data Drift Detection
Upload a second dataset (e.g. last month's production data) and compare distributions. PSI + KS test per column with stable / moderate / significant severity bands.

![Data Drift](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Transform_Data_Drift.png)

### Step 7b — Zero-shot forecast with TimesFM (time-series jobs, a card of its own in Step 7)
Zero-shot forecasting with TimesFM 2.5 on the cleaned series, in its own card before the custom-model card: **nothing is fit, no training split, nothing to tune**. The most recent windows are held back and scored against persistence and seasonal-naive first; the q10–q90 band's coverage is measured, not assumed. The forecast never enters the dataset — it leaves as a PREDICTED-marked CSV, a self-contained HTML chart, or the forecast store, where it sits beside trained runs in one **actual vs predicted** chart. `pip install "prepro-auto[timesfm]"` (TimesFM 2.5 is Apache-2.0; 3.0's weights are non-commercial and are not used).

### Step 6 — Results & Export
Quality score before vs after, column-type changes, and three downloads: cleaned data (CSV or Parquet), audit PDF, and a standalone pipeline script.

> **Reproducibility guarantee (no train/serve skew, no leakage).** The exported `pipeline.py` is **transform-only**. Every parameter a step needs — means, medians, scaler statistics, category maps, target means — is **fitted once inside PrePro Auto and baked into the script as a literal value**. It never re-fits on the data it is later given. So running it on a held-out/test set applies the *training* statistics (no test-set leakage), one-hot encoding always emits the *same* columns in the *same* order regardless of which categories appear in new data, and target encoding reuses the stored per-category means so the target can never leak back into the features. This is verified by `tests/test_reproducibility.py`.

![Results and Export](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Result.png)

### Step 7 — Train a model (optional)

*New in b5 — the `app/ml` training layer.* Reached from the Results & Export step, or directly at `/train?job=<job_id>`.

**Data source & upload.** Choose **From this cleaning session** — trains on your **current cleaned dataset** (the active version, reflecting every stage, transform, and undo/redo), so the model's features and the Test window match exactly what you prepared — or **Upload a cleaned dataset** (`.csv`/`.parquet`/`.xlsx`/`.json`) for a file already cleaned elsewhere. Both are trusted-input: the finishing steps re-fit per CV fold, preprocessing you already applied is baked in, and the data is scanned for obvious leaks (target leakage warnings surface on the leaderboard). The banner reflects the source.

![Train — Data Source & Upload](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Train_Data_Source.png)

**Setup & suggest.** Pick the **target** from a dropdown of your current/updated columns (type-ahead search), add a plain-English business goal (PrePro Auto maps it to the right ranking metric), and click **Suggest setup** — it detects the task, recommends a metric *with the reason*, and pre-checks the recommended models. Then **choose any models** from a **searchable, categorized picker**: Linear, Tree & ensemble, Gradient boosting (XGBoost/LightGBM/CatBoost), Neighbors, SVM, Naive Bayes, Neural network — every model carries a one-line reason, a *recommended* badge on the defaults, and a *needs `[ml]`* / *not installed* badge where an extra is required. Leave everything unchecked to train the recommended defaults. The **hyperparameter-tuning** dropdown (None / Randomized / Grid / Halving / Optuna) sits alongside an explicit install note — search strategies are scikit-learn (already installed); only Optuna and the boosters need extra installs.

![Train — Setup & Suggest](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Train_Setup.png)

**Leaderboard.** Every candidate is cross-validated on the training split, then scored once on held-out test data. Each row shows the **CV metric**, **accuracy** (classification) or **R²** (regression), the holdout score, fit time, and *why the model was a candidate*. The winner is tagged **BEST** and gets a plain-English callout explaining why it was chosen (ranking metric + held-out accuracy + reason) — the same "decision card" style as the Clean step. A narrative and any leakage warnings sit below.

![Train — Leaderboard](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Train_Leaderboard.png)

**Test it.** A form generated from the model's feature schema — type raw values and predict through the *full* pipeline (preprocessing + model), exactly as production would.

![Train — Test It](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Train_Test_Playground.png)

**Export.** Download a `.joblib` bundle, a standalone scorer script, or ONNX — all three ship preprocessing *and* the model together. The run itself is **persisted** (storage + a database row keyed on `run_id`), so it survives a server restart.

![Train — Export](https://raw.githubusercontent.com/Chilliflex/prepro_auto/main/docs/screenshots/Train_Export.png)

---

## Ways to give PrePro Auto your data

There are **4 input methods in the notebook** and **3 in the web UI**. Pick whichever fits your workflow.

### From a Jupyter notebook (5 ways)

| # | Method | When to use it |
|---|---|---|
| 1 | `prepro_auto.quickclean(data)` | **No browser at all.** One call cleans the whole dataset headlessly and returns `result.df`. Fastest path to a model-ready frame. |
| 2 | `prepro_auto.launch_file(path)` | You have a file on disk — CSV, Excel, JSON, Parquet, etc. Auto-detects encoding and delimiter. **No `pd.read_csv()` needed.** |
| 3 | `prepro_auto.launch(df)` | You already have a pandas DataFrame in memory (from a database query, API response, generated data, or a tricky read you handled yourself). |
| 4 | `session.update(df)` | You already have a session and want to push a new DataFrame to it (e.g. after notebook-side edits). Commits a new undoable version. |
| 5 | Web upload, then notebook reads | Start the server with the CLI, upload via browser, then in the notebook do `prepro_auto.Session(job_id, port).current()` to pull the data back into Python. Rare but valid. |

### From the web UI (3 ways)

| # | Method | When to use it |
|---|---|---|
| 1 | **Drag-and-drop upload** on Step 1 of the workbench | Standard. Drop a CSV/Parquet/Excel/JSON file into the upload box. The engine auto-detects encoding and delimiter. |
| 2 | **File picker** on Step 1 | Same as drag-and-drop, just clicked. Useful when dragging is awkward (split screens, touchpads). |
| 3 | **URL parameter** `?job=<id>` | When the notebook launched the session, the printed URL already includes `?job=...` — no upload needed, the workbench adopts the existing job. |

### Supported file formats

| Format | Extensions | Notes |
|---|---|---|
| CSV | `.csv`, `.tsv`, `.txt` | Auto-detects encoding (utf-8 / utf-8-sig / latin-1 / cp1252) and delimiter (comma, tab, semicolon, pipe) |
| Excel | `.xlsx`, `.xls`, `.xlsm` | First sheet by default; multi-sheet handling via the upload form |
| Parquet | `.parquet`, `.pq` | Fastest format for large datasets, preserves dtypes |
| JSON | `.json`, `.jsonl`, `.ndjson` | JSON-records and JSON-lines both supported |
| Feather | `.feather` | Apache Arrow's native columnar format |

**Not supported:** PDF, DOCX, HTML, images. PrePro Auto is a tabular-data tool — these formats need a dedicated extraction step first (Camelot or pdfplumber for PDFs, BeautifulSoup for HTML).

#### Have a PDF with a table?

Extract it to a DataFrame first, then hand it to PrePro Auto:

```python
import pdfplumber, pandas as pd, prepro_auto

with pdfplumber.open("report.pdf") as pdf:
    rows = pdf.pages[0].extract_table()        # pick the right page
df = pd.DataFrame(rows[1:], columns=rows[0])    # first row is the header
session = prepro_auto.launch(df)                # now clean it like any DataFrame
```

For PDFs with merged cells or complex layouts, try `camelot-py` (better for bordered tables) or `tabula-py` (requires Java). PrePro Auto deliberately leaves PDF extraction to specialised tools because generic PDF-to-table conversion succeeds only ~30–70% of the time depending on the document — bundling it would mean silent extraction errors hidden under PrePro Auto's name.

---

## 1. Input functions (notebook)

Everything you call **before** preprocessing starts. The functions that get data into a session.

| Function | Parameters | Returns | What it does |
|---|---|---|---|
| `prepro_auto.quickclean(data, target=None, threshold=0.90, apply_all=False, stages=None, dataset_type="tabular")` | `data`: DataFrame or file path | `CleanResult` (`.df`, `.report`, `.job_id`) | **Headless, no browser.** Runs the whole engine and applies every decision at/above `threshold`, skipping the uncertain ones (or `apply_all=True` for full auto-pilot). Returns the cleaned DataFrame plus a report of what was applied vs. left for review. `dataset_type="time_series"` runs the time-index stage first. |
| `prepro_auto.launch_file(file_path, domain="general", dataset_type="tabular", port=None, open_browser=False)` | `file_path`: str or Path | `Session` | Reads a file from disk with auto-encoding-detection, starts the local workbench, returns a session. Handles all supported formats. Prints the workbench URL. |
| `prepro_auto.launch(df, domain="general", dataset_type="tabular", port=None, open_browser=False)` | `df`: pandas DataFrame | `Session` | Registers an in-memory DataFrame as a job (no upload, no file I/O), starts the workbench, returns a session. Use when you already have a DataFrame. Pass `dataset_type="time_series"` for time-ordered data — adds the time-index stage. |
| `prepro_auto.Session(job_id, port)` | `job_id`: str, `port`: int | `Session` | Reconnect to an existing session by ID. Use when the notebook restarted but the server is still running, or to attach to a job created from the web UI. |
| `prepro_auto.set_api_key(provider, api_key, model=None)` | `provider`: one of `"groq" / "openai" / "anthropic" / "gemini" / "mistral"` | `dict` with `ok`, `verified`, `provider`, `model`, `reason` | Configures the AI provider at runtime (in-memory only — not written to disk). Makes a tiny test call to verify the key works. Call before `launch()` if you want AI features active for the session. |

**Example — most common pattern:**

```python
import prepro_auto

# Optional: enable AI features for this session
prepro_auto.set_api_key("openai", "sk-...")

# Load a file (auto-encoding-detection)
session = prepro_auto.launch_file(r"C:\Users\me\data\sales.csv")
```

---

## 2. Preprocessing functions

The work itself — clean, transform, version. These are called **on the session object** that input functions returned, or via REST endpoints under `/api/v1/`.

### Profile and clean

| Function / Endpoint | What it does |
|---|---|
| `POST /datasets/{job_id}/profile` | Per-column type inference, missing rates, 0–100 quality score. Run once after upload. **Time-series jobs** are scored with 7 time-aware dimensions (temporal completeness 25 · timestamp integrity 20 · cell completeness 15 · outliers+stuck 15 · frequency regularity 10 · type mismatch 10 · span sufficiency 5) — every deduction is weight × severity with the rule printed beside the measured numbers, outage rows count as missing time steps rather than missing cells, and engineered feature columns are excluded from the arithmetic with the exclusion disclosed. |
| `POST /datasets/{job_id}/stages/time_index` | **Time-series jobs only** (upload with `dataset_type=time_series`; 422 otherwise). Detects the time column (parse-rate / order / uniqueness evidence), then cards for: parse (UTC-normalised when zoned, epoch units auto-detected), chronological sort, duplicate timestamps (keep last/first/average — held for review when values conflict), and frequency + gap inventory with optional reindex to the detected grid (gaps become countable NaN rows). Optional `?time_column=` overrides the detected choice. |
| `POST /datasets/{job_id}/stages/missing_values` | Detect missingness mechanism (MCAR / MAR / MNAR), recommend fill strategy per column. Creates decision cards. **Time-series jobs** get the gap-structured path instead: run lengths decide the method (time-weighted interpolation / forward-fill for runs ≤ 4 steps, long outages left NaN), every filled cell is flagged in a `__was_imputed` shadow column, and KNN/MICE appear greyed-out with the reason they'd leak across time. |
| `POST /datasets/{job_id}/stages/outliers` | IQR + modified Z-score + Isolation Forest. Classifies findings as data errors vs rare events. **Time-series jobs** instead use a Hampel filter (rolling median ± 3×windowed MAD — values judged against their own neighbourhood, so daily peaks aren't flagged) plus stuck-sensor run detection; values are replaced or masked with shadow-column flags, never dropped (dropping a row deletes a timestamp), and long runs of the dominant value (night zeros) are review-only. |
| `POST /datasets/{job_id}/stages/scaling` | Normality-driven scaler choice: Standard / Robust / Box-Cox / Yeo-Johnson / MinMax / log1p. |
| `POST /datasets/{job_id}/stages/correlation` | Find correlated pairs, detect constant / ID-like / target-leaking columns. For time-series jobs each pair card also shows the DIFFERENCED correlation (step-to-step changes) — two trending series correlate spuriously, and the drop-confidence follows the differenced number. |
| `POST /datasets/{job_id}/stages/encoding` | Categorical encoding routed by cardinality: label / ordinal / one-hot / frequency / target. |
| `POST /datasets/{job_id}/stages/ts_physics` | **Time-series jobs only.** Renewables physics features from site facts sent in the body once and stored for reuse (`{"domain": "wind"/"solar", latitude, longitude, hub_height, rated_kw, utc_offset_hours, wind_speeds: [{column, height_m}], columns: {...overrides}}`). Wind: hub-height speed via the per-timestep two-height shear fit (α clamped [0, 0.5]) with an explicit fallback hierarchy and a per-row `method` column, air density p/(R·T) with unit auto-detection disclosed, v³, specific power ½ρv³, power-curve region flags. Solar: NOAA sun position (naive timestamps require `utc_offset_hours`), Haurwitz clear-sky GHI + clear-sky index kt (only with the sun ≥ 5° up), NOCT cell temperature. The QC card flags physically implausible power (night production, above rated, production at calm) in a `__was_implausible` shadow column — values never altered — which the TS quality score counts. Everything is closed-form numpy, so the exported pipeline.py reproduces it standalone. |
| `GET /ml/runs/{run_id}/contract` · `POST /ml/runs/{run_id}/prune` | **Model card + evidence-gated pruning.** The contract is introspected from the shipped artifact: every required raw column with dtype, the preprocessing branch it actually flows through inside the pipeline, permutation importance normalised 0–1, real predict behaviour (missing column → NaN for the imputer; extras ignored), and forecast semantics + clip bounds. Prune drops features below a normalised-importance threshold and re-runs the champion's CV on the kept set — a CV-only before/after comparison (no test read); `apply: true` trains the pruned configuration as a new sealed run. Forecast runs additionally store an actual-vs-predicted-over-time series (with the persistence baseline) captured during the single test read, drawn in the workbench's Phase-5 panel. |
| Forecast training (time-series) | Any training endpoint/spec accepts `forecast_horizon` (predict the target N steps ahead — label shifted, unlabelable tail dropped, H-row purge between train/test and a `gap=H` in every walk-forward fold, all disclosed in the Phase-1 preview), `forecast_baseline_col` (a physics estimate scored beside the always-on **persistence baseline**, same folds + same single test read, champion skill = 1 − error_model/error_baseline), and `clip_min`/`clip_max` (bounds applied INSIDE the shipped pipeline). `prepro_auto.train_horizons(df, target, horizons=[...])` returns the skill-vs-lead-time curve, one sealed run per horizon. Time-series jobs auto-fill the time column from the Time Index stage. |
| `POST/GET /datasets/{job_id}/templates` · `.../templates/presets` · `POST .../templates/{name}/render` | **Deliverable templates (time-series).** Explicit per-column recipes — source, rename, aggregation method, scale factor with your stated reason ("kW over 15 min → MWh") — rendered raw/15-min/hourly/daily with an optional India 96-block column (only when the output step is truly 15 min). The manifest travels in the CSV as `#` header lines (pandas: `comment='#'`); forecast-sourced columns (`"forecast:latest"`) are **permanently marked PREDICTED** with run/model/horizon and the latest-issue join rule, and future target times are appended as rows with the count disclosed. Presets are generated from your job's own columns. Notebook: `session.save_template / .templates / .template_presets / .render_template`. |
| `POST /datasets/{job_id}/forecasts/issue` · `.../reconcile` · `GET .../accuracy` · `POST/GET/DELETE .../schedule` | **Forecast store + scheduler (time-series).** Issue predicts from the latest data row for each trained forecast run — issue_time is the data's own timestamp (deterministic; re-issuing on unchanged data is an idempotent skip), target_time = issue + horizon × the detected step, and the **persistence prediction is captured from the same row at the same instant**. Reconcile fills actuals only from realized rows (nothing interpolated); accuracy reports live MAE/RMSE and skill = 1 − rmse_model/rmse_persistence per (run, horizon) — overlapping scheduled runs give the skill-vs-lead-time curve for free. The interval scheduler (default 2 h) lives with the server process and says so in its status; for production point OS cron at the issue endpoint. Notebook: `session.issue_forecasts / .reconcile_forecasts / .forecast_accuracy / .schedule_forecasts`. |
| `POST /datasets/{job_id}/farm` · `GET .../farm` | **Time-series jobs only.** Store/read the per-asset farm layout (`{assets: [{id, latitude, longitude, rated_kw, hub_height_m/tilt_deg/azimuth_deg}]}`), validated (unique ids, plausible coordinates, ≤ 500 km span) with a computed summary — span km and how many weather-model grid cells the assets cover at a stated spacing (default 9 km): more than one cell is the measured argument for per-asset weather. |
| `POST /datasets/{job_id}/weather/fetch` | **Time-series jobs only.** Fetch Open-Meteo weather for every farm asset and merge it into the dataset as a versioned, audited transformation (undo works). `source: "archive"` (default) uses the **historical-forecast archive** — past forecasts, never reanalysis, so training inputs match what the live system receives; `"live"` fetches the current forecast. Per-asset columns (≤ 8 assets) plus a capacity-weighted farm aggregate; requested-vs-returned coordinates, the 15-min-interpolation caveat of hourly-native models, and the merge match rate are all recorded; naive timestamps require `utc_offset_hours` for the join. The pipeline.py export marks the step as a loud network-dependent gap instead of silently dropping the columns. CFSv2 (16–30 day, coarse, `pip install prepro-auto[cfsv2]`) ships as an operational helper with bilinear multi-asset grid extraction. |
| `POST /datasets/{job_id}/foundation/timesfm/check` · `POST …/runs` · `GET …/runs/{run_id}` · `GET …/runs/{run_id}/export?fmt=csv\|html` · `POST …/runs/{run_id}/issue` · `DELETE …/runs/{run_id}` | **Time-series jobs only.** Zero-shot forecasting with TimesFM 2.5 (`pip install "prepro-auto[timesfm]"`). `check` runs every readiness test with its number and never loads the model; `runs` starts one batched forecast (8 held-back windows + the final horizon) on a worker thread and returns 202; the result carries MAE/RMSE by lead, persistence and seasonal-naive on the same windows, skill (`1 − error_model/error_baseline`), measured q10–q90 coverage, the checkpoint revision and the fixed config. Exports are marked PREDICTED; `issue` writes one forecast-store record per lead. `GET /foundation/timesfm/status` says whether the model is usable here. |
| `GET /datasets/{job_id}/forecasts/series` · `GET …/series/export.html` | Actual vs every issued forecast in the store (trained runs and TimesFM) on one axis, persistence underneath, with each run's MAE on the **same rows**; `join=latest` or `join=lead:<h>`. The `.html` variant is one self-contained file (Plotly and data embedded) that opens offline. |
| `POST /datasets/{job_id}/stages/ts_features` | **Time-series jobs only** (422 otherwise; runs last). Proposes lag features guided by measured lag coefficients against a ±z/√n significance band with z raised for the number of candidates tested (Bonferroni; PACF for short lags with `prepro-auto[timeseries]`, plain autocorrelation otherwise and always for the daily/weekly lags — the card names the method and the band), rolling mean/std over ~1 h and ~1 day windows **shifted by one step** (the window at time t ends at t−1 — leakage-safe by construction), and calendar features the span can actually support (cyclical sin/cos for time-of-day and day-of-year). Additive only: rows untouched, existing columns never overwritten; a pure-noise column earns no lag card. |
| `POST /datasets/{job_id}/stages/{stage_name}/execute` | Apply your approved decisions, commit a new version. `stage_name` is one of the stages above. |

### Decision cards (the human-in-the-loop)

| Endpoint | What it does |
|---|---|
| `GET /datasets/{job_id}/decisions?stage=<stage>` | List decision cards for a stage |
| `POST /decisions/{decision_id}/approve` | Use the recommended action |
| `POST /decisions/{decision_id}/override` | Use an alternative action (body: `{"action": "...", "reason": "..."}`) |
| `POST /decisions/{decision_id}/skip` | Don't change this column |
| `POST /decisions/{decision_id}/drop-column` | Drop the column entirely |

### Manual transforms (when you need more control)

| Endpoint | What it does |
|---|---|
| `GET /datasets/{job_id}/transform/operations` | List all 18 preset operations and their parameters |
| `POST /datasets/{job_id}/transform/preset` | Apply one preset op (rename, drop, cast, fillna, filter, merge, math, map, string ops, regex, **group-aggregate**, sort, dedup, extract-number) |
| `POST /datasets/{job_id}/transform/expression` | Apply a sandboxed pandas expression (e.g. `df["revenue"] - df["cost"]` into target `profit`). In the UI this runs only from a preview's "Looks right — apply it" — never directly. |
| `POST /datasets/{job_id}/transform/expression/preview` | Dry-run the same expression: shape delta, added/removed columns, dtype changes and a before/after sample — commits nothing |
| `POST /datasets/{job_id}/transform/expression/evaluate` | **Read-only answer mode** (same sandbox): a scalar comes back as its value (`((df["a"]==0) & (df["b"]==0)).sum()` → the count), a boolean mask as its true-count + share (with one-click drop/keep follow-up filters in the UI), a Series as stats + first values, a frame as shape + sample. Touches nothing, logs nothing — the inspect step before preview → approve. |
| `GET /datasets/{job_id}/transformations` | The audit trail: every applied step with the snapshot version it produced |
| `POST /ml/split/{job_id}` | Phase 1 of training: the exact split the run will use — row counts, target balance per side, CV fold plan, leakage scan — computed without fitting anything |
| `POST /ml/candidates/{job_id}` | Phase 2: every catalog model scored against the training partition — base + named factor adjustments, each citing the measured numbers — ranked, with a family-diverse preselection |
| `POST /ml/tuning/{job_id}` | Phase 3: each selected model's search space (standard ranges reshaped by the measured training partition, every reshape explained), the resolved method (`auto` = grid iff it fits the budget), and the cost in fits — computed by the same functions the trainer runs |
| `POST /ml/runs/{run_id}/select` | Phase 4: evaluate the chosen champion — the single read of the run's test partition, in the configuration that ships. Selection happens on the CV-only board (`selection: "manual"` on train); a second read is refused. The response carries the Phase-5 evaluation detail (predicted-vs-actual, residuals, confusion matrix, ROC/PR), captured during that read and rendered offline with a vendored Plotly |
| `POST /ml/ai/plan/{job_id}` | C1, the AI trainer: an LLM reads computed facts about the current dataset and proposes the plan a person would fill in by hand, with reasons; the server validates every field against the real data (corrections disclosed), the engine derives the metric, and approval runs the ordinary phased flow — the AI computes no number the seal protects |
| `GET /ml/runs/{run_id}/export?fmt=train_script` | C2: a Python script that re-runs this training outside the app — same pinned columns, split, seed, preprocessing and champion — and checks every reproduced metric against the recorded ones (exit 0 = the report reproduces). Trusted runs are standalone pandas+sklearn; strict runs embed the exact recipe and rebuild it via `prepro_auto` |
| `GET /datasets/{job_id}/state` | Cheap polling endpoint: active version + a revision token that changes when the data changes (drives notebook↔browser sync) |
| `POST /datasets/{job_id}/transform/batch` | Apply one operation across many columns as a single undoable step |

### AI-assisted transforms (optional, needs an API key)

| Endpoint | What it does |
|---|---|
| `POST /datasets/{job_id}/transform/ai-propose` | Describe a change in plain English; AI proposes a concrete transform with preview |
| `POST /datasets/{job_id}/transform/ai-advise` | Ask the AI for advice on a column without changing anything |
| `POST /datasets/{job_id}/transform/assistant` | One-shot assistant call (full message) |
| `POST /datasets/{job_id}/transform/chat` | Multi-turn conversation preserving history |

### Versioning and history

| Endpoint | What it does |
|---|---|
| `GET /datasets/{job_id}/view` | Current (active-version) data with shape, dtypes, sample rows |
| `GET /datasets/{job_id}/history` | Full version history with labels |
| `POST /datasets/{job_id}/undo` | Move active pointer back one version |
| `POST /datasets/{job_id}/redo` | Move active pointer forward one version |
| `GET /datasets/{job_id}/snapshots` | List all committed snapshots |

### Visualization and monitoring

| Endpoint | What it does |
|---|---|
| `POST /datasets/{job_id}/viz/chart` | Build a histogram, bar, or scatter chart |
| `POST /datasets/{job_id}/viz/metric` | Compute a condition-based metric (e.g. "rows where price > 1000") |
| `POST /datasets/{job_id}/viz/compare` | Compare one column's distribution raw vs current |
| `GET /datasets/{job_id}/viz/dashboard` | Power-BI-style before/after dashboard (KPI tiles + per-column comparison) |
| `POST /drift/compare` | Compare two uploaded datasets for distribution drift (PSI + KS) |

---

## 3. Output functions

The artifacts you take away from a session. Notebook methods return Python objects; REST endpoints return downloadable files.

### From the notebook (Python objects)

| Method | Returns | Where to use it |
|---|---|---|
| `prepro_auto.quickclean(data)` | `CleanResult` — `.df` (cleaned DataFrame), `.report` (per-stage summary), `.job_id` | Headless one-call clean. `result.df` drops straight into `model.fit(X, y)`; `print(result)` shows what was applied vs. left for review. |
| `session.current()` | pandas DataFrame | The current (active-version) DataFrame as it stands in the UI right now. Drop straight into `model.fit(X, y)`. Warns if the UI holds approved decisions whose stage has not been executed — those are not in the data yet. |
| `session.update(df)` | the same `Session` | Pushes a notebook-edited DataFrame as a new undoable version; an open workbench tab refreshes itself within ~2s. Decision cards computed against the replaced data are cleared so the UI re-analyses (`reanalyze=False` keeps them). |
| `session.df` | pandas DataFrame | The **live** dataset, re-read on every access — `session.df.head()` is never stale. `session.current()` by contrast returns a snapshot you can hold onto. |
| `session.state()` / `session.status()` | dict / dict + printout | Live version, shape, undo/redo availability, decisions still open, and anything approved in the UI but not yet executed. |
| `session.version` | int | The active version number (0 = the raw data you launched with). |
| `session.url` | str | The workbench URL for this session — useful for re-opening after closing the tab. |
| `session.job_id` | str | The internal job ID — use it for raw REST API calls. |
| `session.port` | int | The local port the server is running on. |

### From the REST API or web UI (downloadable files)

| Endpoint | File | Where to use it |
|---|---|---|
| `GET /datasets/{job_id}/export/data?format=csv` | Cleaned CSV | Share with teammates, load into BI tools (Tableau, Power BI, Looker), commit to a versioned data repo. |
| `GET /datasets/{job_id}/export/data?format=parquet` | Cleaned Parquet | Faster and smaller than CSV for large datasets; preserves dtypes exactly. |
| `GET /datasets/{job_id}/export/audit` | Audit PDF | Compliance trail listing every transformation with parameters, before/after stats, who approved. Attach to a model-card or hand to a data-governance reviewer. |
| `GET /datasets/{job_id}/export/pipeline` | Runnable `.py` script | **Transform-only** reproduction of the exact cleaning (no re-fitting; uses the statistics fitted at clean time), with no PrePro Auto dependency. Auto-detects CSV/Parquet/Excel/JSON input. Drop into Airflow / Prefect / GitHub Actions. Run with `python pipeline.py raw.csv ready.csv`. |
| `POST /drift/compare` (returns JSON) | Drift report | Per-column PSI / KS verdicts with severity bands. Plug into a monitoring dashboard, alert on `overall_verdict == "significant_drift"`. |

### Two typical workflows end-to-end

```python
# Workflow 1 — notebook to model, no file I/O:
session = prepro_auto.launch_file(r"C:\data\sales.csv")
# ...clean visually in the browser, then:
X = session.current().drop(columns=["target"])
y = session.current()["target"]
model.fit(X, y)

# Workflow 2 — clean once, productionize the pipeline:
# 1) Download pipeline.py from the workbench's Export step
# 2) Commit it to your model repo
# 3) In production:
#    subprocess.run(["python", "pipeline.py", "incoming.csv", "ready.csv"])
```

---

## 4. Train functions (optional ML layer)

*New in b5* — `app/ml`. Once a dataset is clean, fit and compare candidate models on your **current cleaned data**, predict, and export. The finishing steps re-fit per CV fold and the data is scanned for obvious leaks (trusted-input).

| Function | Parameters | Returns | What it does |
|---|---|---|---|
| `prepro_auto.list_models(task="classification")` | `task`: `"classification"` or `"regression"` | pandas DataFrame | Lists every model **id** you can pass to `models=[...]`, with its category, whether it's a recommended default, and whether any extra install is needed. The notebook equivalent of the workbench's model picker. |
| `session.train(target, business_goal=None, models=None, test_size=0.2, cv=5, tuning="none", tuning_iter=25, time_column=None, group_column=None, calibrate=False, importance=False, tune_threshold=False)` | Available on `Session` and `CleanResult` (chained automatically) | `TrainResult` | Trains on your **current cleaned dataset** (the active version — every stage, transform, and undo/redo), so the features match what you prepared. Trusted-input: finishing steps re-fit per fold, scanned for leaks. |
| `prepro_auto.train(data, target, **kwargs)` | `data`: DataFrame or file path that's *already clean*; same keyword args as above | `TrainResult` | **Trusted-input mode.** Adds no target encoding of its own and scans for obvious leaks, surfacing warnings. |

**`TrainResult`:** `.report` (problem type, primary metric, leaderboard, leakage guarantee), `.best` (winning model's metrics), `.leaderboard` (pandas DataFrame, one row per model — includes accuracy/R², CV and holdout scores), `.leakage` (guarantee level + warnings), `.predict(records)` (runs the *full* fitted pipeline), `.export(path, fmt="joblib"|"pipeline"|"onnx")`.

**Choosing models** — `models=` takes a list of ids resolved against the full catalog (run `prepro_auto.list_models()` to see them all). Leave it `None` to train the recommended defaults.

| Category | Classification ids | Regression ids |
|---|---|---|
| Recommended defaults | `logistic`, `random_forest`, `gradient_boosting`, `neural_net` | `linear`, `random_forest`, `gradient_boosting`, `neural_net` |
| Linear | `ridge_classifier`, `sgd_classifier` | `linear_regression`, `lasso`, `elasticnet` |
| Tree & ensemble | `decision_tree`, `extra_trees`, `hist_gradient_boosting`, `adaboost`, `bagging` | same |
| Neighbors / SVM | `knn`, `svc`, `linear_svc` | `knn`, `svr`, `linear_svr` |
| Naive Bayes | `gaussian_nb`, `bernoulli_nb` | — |
| Gradient boosting (needs `[ml]`) | `xgboost`, `lightgbm`, `catboost` | same |

**Tuning** — `tuning=` is one of `"none"` (defaults), `"random"`, `"grid"`, `"halving"` (all scikit-learn, already installed) or `"optuna"` (needs `pip install optuna`); `tuning_iter` is the search budget.

**Real-world rigor options** (same engine the UI's Step 7 uses): `time_column` (chronological holdout, no future-row leakage), `group_column` (grouped holdout, no entity straddles train/test), `calibrate` (wrap the winner in `CalibratedClassifierCV`), `tune_threshold` (F1-optimal binary decision threshold), `importance` (permutation feature importance on the holdout split).

```python
import prepro_auto

prepro_auto.list_models()                   # discover model ids by category

session = prepro_auto.launch_file("sales.csv")
# ...clean visually in the workbench...
res = session.train(
    target="churn",
    business_goal="catch likely churn",
    models=["logistic", "random_forest", "xgboost"],   # or omit for the defaults
    tuning="random", tuning_iter=20,
    calibrate=True, tune_threshold=True,               # trustworthy probabilities + F1-optimal cutoff
)

print(res)                                  # leaderboard + metric + leakage guarantee
res.leaderboard                             # pandas DataFrame (incl. accuracy / R²)
res.predict({"tenure_months": 14, "plan": "pro"})
res.export("model.joblib")                  # bundled preprocessing + model
```

**The phased workflow, from the notebook** — the same five sealed phases the workbench walks, as session methods (each calls the same service layer, so the seals hold: the comparison is cross-validation only, and the test partition is read exactly once, by the champion you pick):

```python
session.plan_split("churn")                       # Phase 1: the EXACT split, nothing fit
session.candidates("churn")                       # Phase 2: catalog scored for THIS data (.df)
session.plan_tuning("churn", models=["gradient_boosting", "logistic"])   # Phase 3: fits priced

cmp = session.compare("churn", models=["gradient_boosting", "logistic"],
                      tuning="auto")              # Phase 4: CV-only leaderboard
cmp.leaderboard                                   # no test numbers exist yet
res = cmp.select()                                # Phase 5: the single test read
# cmp.select("logistic")                          # ...or override the CV winner (recorded)
# cmp.select() a second time raises: the test set has been spent

plan = session.ai_plan("catch likely churn — rare churners matter")   # AI plans,
print(plan)                                       # reasons + validator corrections
res = plan.compare().select()                     # ...the engine runs

res = session.train(target="churn", mode="strict")   # one-shot, recipe replayed from raw
```

**Run persistence:** every training run is saved to durable storage (a joblib-serialised bundle) and indexed by an `MLRun` database row keyed on `run_id`. Predict, export, and drift all reload a run by ID on demand — a run survives a server restart or redeploy, not just an in-memory session.

| Endpoint | What it does |
|---|---|
| `GET /api/v1/ml/catalog` | The full model catalog grouped by category (classification + regression), tuning strategies, and install hints — powers the searchable picker |
| `GET /api/v1/ml/columns/{job_id}` | Current columns (name + dtype) of a cleaning job — populates the target dropdown |
| `POST /api/v1/ml/columns/upload` | Columns of an uploaded cleaned file — target dropdown on the upload path |
| `POST /api/v1/ml/suggest/{job_id}` | DEPRECATED — pre-phase suggestion; use `POST /ml/candidates/{job_id}`, which scores against the Phase-1 training partition |
| `POST /api/v1/ml/train/session/{job_id}` | Train in the chosen mode (strict replays the recipe from raw per fold); `selection: "manual"` returns the Phase-4 CV-only board for `/runs/{run_id}/select` |
| `POST /api/v1/ml/train/upload` | Training on an uploaded already-clean dataset (trusted-input mode) |
| `GET /api/v1/ml/runs/{run_id}` | Full report: leaderboard, metrics, leakage guarantee, selection + evaluation detail |
| `POST /api/v1/ml/runs/{run_id}/predict` | Test playground — runs the full pipeline on raw records |
| `GET /api/v1/ml/runs/{run_id}/export?fmt=joblib\|pipeline\|onnx\|train_script` | Download the trained bundle, a standalone scorer script, an ONNX export, or the self-verifying training script (C2) |
| `POST /api/v1/ml/runs/{run_id}/drift` | Drift on the model's input features (reuses the product's PSI/KS drift engine) |

Optional boosters for stronger tabular models (the layer falls back to scikit-learn's `HistGradientBoosting` if none are installed):

```bash
pip install prepro-auto[ml]   # LightGBM, XGBoost, CatBoost, ONNX export
```

---

## What it does

- **Profile** — per-column type inference, missing rates, 0–100 quality score
- **Clean (guided)** — five HITL stages: missing values, outliers, scaling, correlation/leakage, encoding
- **Transform (manual)** — 18 preset ops (incl. group-aggregate features), sandboxed expressions with dry-run preview, multi-column batches
- **Visualize** — histogram, bar, scatter, box plot (grouped, with outlier counts), correlation heatmap, missing-value overview, grouped aggregates, time series, and a shared-axis raw-vs-current overlay
- **AI assistant** — optional; describe a change in plain English; preview before applying
- **Dashboard & drift** — before/after KPI tiles with per-column change status; PSI/KS drift between any two datasets
- **Data drift** — PSI + KS test between two datasets
- **Undo/redo** — every change is a version
- **Export** — cleaned data (CSV/Parquet), audit PDF, runnable Python pipeline
- **Train (optional, Step 7)** — fit and compare candidate models on your current cleaned dataset, calibrate probabilities, tune the decision threshold, get permutation importance, predict, and export a bundled joblib/ONNX/scorer-script — runs persist across restarts

## Methods

Field-standard methods throughout: MICE / KNN / median imputation, IQR + MAD + Isolation Forest for outliers, normality-driven scaling (Standard / Robust / Box-Cox / Yeo-Johnson), label / ordinal / one-hot / frequency / target encoding. No accuracy compromises — the same algorithms a data scientist would write by hand.

---

## AI providers (optional)

AI features are **optional**. Everything works offline without a key. PrePro Auto supports five providers:

| Provider | ID | Install | Get a key |
|---|---|---|---|
| Groq (free tier, fast) | `groq` | `pip install prepro-auto[groq]` | https://console.groq.com |
| OpenAI / GPT | `openai` | `pip install prepro-auto[openai]` | https://platform.openai.com |
| Anthropic Claude | `anthropic` | `pip install prepro-auto[anthropic]` | https://console.anthropic.com |
| Google Gemini | `gemini` | `pip install prepro-auto[gemini]` | https://aistudio.google.com/app/apikey |
| Mistral | `mistral` | `pip install prepro-auto[mistral]` | https://console.mistral.ai |

Or install all five at once: `pip install prepro-auto[ai]`.

### Data privacy — what leaves your machine

The **preprocessing engine is 100% local**: profiling, cleaning, scaling, encoding, drift, export, versioning, and storage all run in-process, with no network calls and no telemetry.

The **optional AI features are the only part that uses the network.** When you enable a provider, PrePro Auto sends **only**:

- column **names** and **dtypes**, and
- a **few sample values per column** (default 3–4, never the full dataset), plus
- a **2-row before/after preview** when you use the AI assistant to apply a transform.

It never uploads your dataset, and AI is off until you add a key. Two controls reduce egress further (both on by default where noted):

| Setting | Default | Effect |
|---|---|---|
| `LLM_MASK_PII` | `True` | Redacts values that look like PII (emails, phones, long IDs, card-like numbers) and fully masks samples from PII-named columns (`email`, `ssn`, `name`, `address`, …) **before** anything is sent. |
| `LLM_SEND_VALUES` | `True` | Set `False` to send **no raw values at all** — only column names, dtypes, and aggregate descriptors. Maximum privacy; slightly lower suggestion quality. |

> **Will the provider "learn" my data?** Your sample values are sent to the provider you choose, governed by that provider's data-processing terms (the major APIs generally do **not** train on API traffic, and several offer zero-retention modes — check your provider). PrePro Auto adds the masking above so obvious PII never leaves the machine in the first place. **For zero network egress, leave AI disabled, or point it at a local model** (e.g. an OpenAI-compatible endpoint such as Ollama).


### Three ways to give PrePro Auto your API key

**1. Notebook (in-memory, session-only — safest):**
```python
prepro_auto.set_api_key("openai", "sk-...")
```

**2. Web UI:** click **AI Provider → Configure API key** in the side rail, paste key, click Test & apply.

**3. `.env` file (survives restarts):**
```bash
LLM_PROVIDER=openai
OPENAI_API_KEY=sk-...
```

**Security note:** the `.env` file is plain text. Fine for a personal machine; never enable disk-persistence on a shared or hosted deployment.

---

## REST API reference

The web app and SDK both call the same endpoints under `/api/v1`. Once the server is running, the interactive Swagger UI is at `http://localhost:8000/docs`.

For the full table organized by category, see [Section 2 — Preprocessing functions](#2-preprocessing-functions) and [Section 3 — Output functions](#3-output-functions) above. System endpoints:

| Endpoint | Purpose |
|---|---|
| `GET /api/v1/health` | Liveness check |
| `GET /api/v1/system/limits` | Live RAM-aware upload limits |
| `GET /api/v1/system/llm` | List providers + active one |
| `POST /api/v1/system/llm/configure` | Set provider + key at runtime |

---

## Documentation

- [Complete Guide (PDF)](https://github.com/Chilliflex/prepro_auto/blob/main/docs/PrePro_Auto_Complete_Guide.pdf) — project overview, architecture, all ML/stats models used, accuracy benchmarks, full user guide for notebook and web UI
- Interactive Swagger at `http://localhost:8000/docs` (once running)


## License

MIT
