Metadata-Version: 2.5
Name: xplainable-preprocessing
Version: 0.4.1
Summary: Shared preprocessing pipeline package for xplainable
Requires-Python: >=3.9
Requires-Dist: cloudpickle>=3.0
Requires-Dist: numpy>=1.24
Requires-Dist: pandas<3,>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: scikit-learn>=1.3
Requires-Dist: scipy>=1.8
Provides-Extra: dev
Requires-Dist: openpyxl>=3.1; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: xlrd>=2.0.1; extra == 'dev'
Provides-Extra: excel
Requires-Dist: openpyxl>=3.1; extra == 'excel'
Requires-Dist: xlrd>=2.0.1; extra == 'excel'
Description-Content-Type: text/markdown

# xplainable-preprocessing

Shared preprocessing pipeline library for [xplainable](https://xplainable.io). Define, compile, fit, serialise, and apply data transformation pipelines using a declarative JSON spec.

## Install

```bash
pip install xplainable-preprocessing
pip install "xplainable-preprocessing[excel]"   # io.read_table for .xlsx/.xls
```

Upgrading from 0.3.x? 0.4.0 has breaking changes; see [CHANGELOG.md](CHANGELOG.md).

## Quick Start

### Define a pipeline as JSON

```python
from xplainable_preprocessing.schema import PipelineSpec
from xplainable_preprocessing.compiler import compile_spec

spec = PipelineSpec(
    version="2.0",
    steps=[
        {"id": "drop_ids", "type": "DropColumnsTransformer", "params": {"columns": ["customer_id", "row_id"]}},
        {"id": "fill_numeric", "type": "FillMissingTransformer", "columns": ["tenure", "charges"], "params": {"strategies": {"tenure": "median", "charges": "median"}}},
        {"id": "clip_outliers", "type": "ClipTransformer", "columns": ["charges"], "params": {"min_val": 0, "max_val": 500}},
        {"id": "missing_flags", "type": "MissingFlagTransformer", "columns": ["tenure", "charges"]},
        {"id": "condense_plan", "type": "CategoryCondenseTransformer", "columns": ["plan_type"], "params": {"max_categories": 10}},
        {"id": "extract_dates", "type": "DateTimeExtractTransformer", "columns": ["signup_date"], "params": {"components": ["month", "dayofweek"], "drop_original": true}},
    ]
)

pipeline = compile_spec(spec)
```

### Fit and transform

```python
import pandas as pd

df = pd.read_csv("data.csv")
pipeline.fit(df)
transformed = pipeline.transform(df)
```

### Serialise and load

```python
from xplainable_preprocessing.serialization import save_pipeline, load_pipeline

# Save
binary = save_pipeline(pipeline)

# Load
loaded = load_pipeline(binary)
result = loaded.transform(new_data)
```

## Available Transformers

### Data Cleaning
| Transformer | Description |
|-------------|-------------|
| `DropColumnsTransformer` | Drop specified columns |
| `RenameColumnsTransformer` | Rename columns |
| `TypeCastTransformer` | Cast column dtypes (e.g. str to float64, with error coercion) |
| `ClipTransformer` | Set numeric values outside (min, max), or per-column `bounds`, to NaN |
| `ParseNumericTransformer` | Parse numbers stored as text (%, currency, `1,234`, `36 months`, ranges, `10+`) |
| `NullTokensTransformer` | Turn placeholder text (`?`, `N/A`) and sentinel numbers (`999`) into NaN |

### Missing Values
| Transformer | Description |
|-------------|-------------|
| `FillMissingTransformer` | Fill nulls with median, mode, constant, or other strategies |
| `MissingFlagTransformer` | Create binary 0/1 indicator columns for nulls |

### Feature Engineering
| Transformer | Description |
|-------------|-------------|
| `ExpressionTransformer` | Create derived columns from expressions (allow-listed evaluator) |
| `DateTimeExtractTransformer` | Extract year, month, dayofweek, elapsed time, etc. from datetime columns |
| `GroupByAggTransformer` | Aggregate features by group (count, sum, mean) |
| `GroupedLagTransformer` | Create row-shift lags within groups |
| `RollingAggTransformer` | Rolling window aggregations |
| `TimeLagTransformer` | Lags and trailing windows at exact time offsets over strictly earlier rows |
| `GroupBroadcastTransformer` | Write a group aggregate (e.g. a holiday flag per date) to every row of the group |
| `TextFeaturesTransformer` | Numeric features from free text (length, digits, URLs, keyword flags) |

### Categorical
| Transformer | Description |
|-------------|-------------|
| `CategoryCondenseTransformer` | Condense high-cardinality categoricals to top N + "Other" |
| `TextCleanTransformer` | Clean text cells: lowercase, strip, remove HTML/extra whitespace, fix encoding, mask digits/URLs/emails |
| `OrdinalMapTransformer` | Map an ordered scale (grades, ratings, bands) to numbers |
| `CrossCategoriesTransformer` | Cross 2-3 low-cardinality columns into one category |

### Scaling (use with caution -- breaks xplainable model explainability)
| Transformer | Description |
|-------------|-------------|
| `StandardScaler` | Standardise to zero mean, unit variance |
| `MinMaxScaler` | Scale to [0, 1] range |
| `RobustScaler` | Scale using median and IQR |
| `PowerTransformer` | Apply power transform for normality |
| `QuantileTransformer` | Transform to uniform or normal distribution |

> **Note:** Scaling transformers are available but should NOT be used with xplainable models. xplainable models are inherently explainable and require raw feature values. Scaling destroys interpretability.

### Other sklearn
| Transformer | Description |
|-------------|-------------|
| `SimpleImputer` | sklearn's imputer |
| `OneHotEncoder` | One-hot encode categoricals |
| `OrdinalEncoder` | Ordinal encode categoricals |
| `KBinsDiscretizer` | Discretise continuous features into bins |
| `Binarizer` | Threshold features to binary |

## Pipeline Spec Format

Pipelines are defined as JSON for portability across the xplainable platform, MCP tools, and API:

```json
{
    "version": "2.0",
    "steps": [
        {
            "id": "step_name",
            "type": "TransformerName",
            "columns": ["col1", "col2"],
            "params": {"param1": "value1"},
            "description": "Optional description"
        }
    ]
}
```

- `id`: unique step identifier
- `type`: transformer class name (from the registry)
- `columns`: optional list of columns to apply to (wraps in `DataFrameColumnTransformer`)
- `params`: constructor arguments for the transformer
- `description`: optional human-readable description

## Architecture

```
PipelineSpec (JSON)
    ↓ compile_spec()
DataFramePipeline (sklearn Pipeline of transformers)
    ↓ fit() / transform()
Transformed DataFrame
    ↓ save_pipeline()
Binary (cloudpickle)
    ↓ load_pipeline()
DataFramePipeline (restored)
```

Key modules:
- `schema.py` -- Pydantic models for `PipelineSpec` and `StepSpec`
- `compiler.py` -- compiles JSON spec into sklearn pipeline
- `pipeline.py` -- `DataFramePipeline` wrapper around sklearn `Pipeline`
- `registry.py` -- maps transformer names to classes, generates LLM-ready catalog
- `serialization.py` -- cloudpickle save/load; portable across Python versions
- `guard.py` -- step safety checks and step/spec previews
- `preview.py` -- before/after preview with schema deltas
- `contracts.py` -- which engineered columns each step computes (`dependency_map`, `partial_transform`)
- `detect/` -- detectors that describe a raw frame (`scan_all`, `Issue`)
- `io.py` -- `read_table`, the shared CSV/Excel reader
- `numeric_text.py` -- parse numbers stored as text

## Development

```bash
pip install -e ".[dev]"
pytest
```

## License

MIT
