Metadata-Version: 2.4
Name: cleanflow-kit
Version: 1.2.3
Summary: A comprehensive, modular Python tool for automated data cleaning, EDA, and feature engineering.
Author: CleanFlow Contributors
License: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.0.0
Requires-Dist: numpy>=1.20.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: matplotlib>=3.5.0
Requires-Dist: joblib>=1.1.0
Provides-Extra: smote
Requires-Dist: imbalanced-learn>=0.9.0; extra == "smote"
Provides-Extra: drift
Requires-Dist: scipy>=1.7.0; extra == "drift"
Provides-Extra: all
Requires-Dist: imbalanced-learn>=0.9.0; extra == "all"
Requires-Dist: scipy>=1.7.0; extra == "all"
Provides-Extra: web
Requires-Dist: Flask; extra == "web"
Requires-Dist: Flask-SQLAlchemy; extra == "web"
Requires-Dist: PyJWT; extra == "web"
Requires-Dist: gunicorn; extra == "web"
Requires-Dist: openpyxl; extra == "web"
Provides-Extra: api
Requires-Dist: FastAPI; extra == "api"
Requires-Dist: uvicorn; extra == "api"
Requires-Dist: pydantic; extra == "api"
Dynamic: license-file

# CleanFlow (`cleanflow-kit`)

A comprehensive, intelligent Python package for automated **data cleaning**, **exploratory data analysis (EDA)**, **feature engineering**, **AutoML model training**, **data drift detection**, **synthetic data generation**, and **HTML/Markdown report generation**.

CleanFlow eliminates the repetitive 80% of data science work — replacing hundreds of lines of boilerplate Pandas/Scikit-learn code with a clean, chainable API.

---

## 📦 Installation

```bash
pip install cleanflow-kit
```

Install with optional extras:

```bash
# SMOTE for class imbalance handling
pip install "cleanflow-kit[smote]"

# Drift detection (requires scipy)
pip install "cleanflow-kit[drift]"

# Everything
pip install "cleanflow-kit[all]"

# Web application (Flask UI)
pip install "cleanflow-kit[web,api]"
```

**Requirements:** Python 3.8+, pandas ≥ 2.0, numpy ≥ 1.20, scikit-learn ≥ 1.0, matplotlib ≥ 3.5

---

## 🚀 Quick Start (3 Lines)

```python
from cleanflow import DataPipeline

pipeline = DataPipeline()
pipeline.load("your_data.csv")

cleaned_df, final_df = pipeline.run_full_pipeline(
    target_col="price",
    problem_type="regression"   # or "classification" — auto-detected if omitted
)

pipeline.save_data("./output")
```

That's it. Your data is cleaned, analyzed, feature-engineered, and saved.

---

## 📖 Complete Usage Guide

### 1. One-Click Full Pipeline

The simplest way to use CleanFlow. Runs validation → cleaning → EDA → feature engineering automatically.

```python
from cleanflow import DataPipeline

pipeline = DataPipeline()
pipeline.load("housing_prices.csv")

# Run everything
cleaned_df, final_df = pipeline.run_full_pipeline(
    target_col="price",
    problem_type="regression",
    show_eda_plots=True,
    cleaning_config={                        # Optional overrides
        "missing_numeric_strategy": "mean",
        "outlier_method": "zscore"
    },
    feature_config={                         # Optional overrides
        "scale_method": "minmax",
        "create_polynomial_features": True,
        "auto_evolve_features": True
    }
)

# Access reports
print(pipeline.pipeline_report['validation'])
print(pipeline.pipeline_report['cleaning'])
print(pipeline.pipeline_report['eda'])
print(pipeline.pipeline_report['feature_engineering'])

# Save outputs
pipeline.save_data("./output", format="csv")       # or "parquet"
```

---

### 2. Step-by-Step Pipeline (Full Control)

Call each stage individually for granular control over every parameter.

```python
from cleanflow import DataPipeline

pipeline = DataPipeline()
pipeline.load("customer_churn.csv")

# ── Step 1: Validate ──
report = pipeline.validate()
# Returns: shape, dtypes, missing values, duplicates, constant columns, potential ID columns

# ── Step 2: Clean ──
pipeline.clean(
    remove_duplicates=True,
    handle_missing=True,
    missing_numeric_strategy='median',       # 'mean', 'median', 'zero'
    missing_categorical_strategy='mode',     # 'mode', 'unknown'
    missing_drop_threshold=0.4,              # Drop columns with >40% missing
    fix_types=True,                          # Auto-infer numeric, datetime, category types
    handle_outliers=True,
    outlier_method='iqr',                    # 'iqr' or 'zscore'
    outlier_action='clip',                   # 'clip', 'remove', or 'nan'
    clean_categorical=True,                  # Strip whitespace, lowercase
    drop_constant=True                       # Remove constant/single-value columns
)

# ── Step 3: Exploratory Data Analysis ──
eda_results = pipeline.analyze(
    target_col="churn",
    show_plots=True,
    save_plots=True,
    output_dir="./eda_plots"
)
# Generates: distributions.png, boxplots.png, correlation.png, categorical.png, target_relationships.png

# ── Step 4: Feature Engineering ──
pipeline.engineer_features(
    target_col="churn",
    problem_type="classification",
    encode_categorical=True,                 # Auto: OneHot for low-cardinality, Label for high
    scale_features=True,
    scale_method='standard',                 # 'standard' or 'minmax'
    create_datetime_features=True,           # Extracts year, month, day, dayofweek, is_weekend
    create_polynomial_features=False,        # Generates degree-2 polynomial terms
    polynomial_degree=2,
    drop_low_variance=True,                  # Removes near-zero variance features
    drop_high_correlation=True,              # Removes one of each highly correlated pair
    correlation_threshold=0.95,
    handle_imbalance=True,                   # Applies SMOTE for imbalanced classes
    imbalance_method='smote',                # 'smote', 'oversample', 'undersample'
    auto_evolve_features=False               # Auto-generates & selects best interaction features
)

# ── Step 5: Train Models (AutoML) ──
model_results = pipeline.train_model(
    target_col="churn",
    problem_type="classification",
    export_path="./best_model.pkl"           # Optional: export trained model
)

print("Best Model:", model_results['best_model']['name'])
print("Metrics:", model_results['best_model']['metrics'])
print("Reliability:", model_results['reliability'])
print("Feature Importance:", model_results['feature_importance'])

# ── Step 6: Generate Reports ──
pipeline.generate_html_report("./report.html")
pipeline.generate_markdown_report("./report.md")

# ── Access Data at Any Stage ──
raw_df     = pipeline.get_raw_data()
cleaned_df = pipeline.get_cleaned_data()
final_df   = pipeline.get_final_data()
full_report = pipeline.get_report()
```

---

### 3. Using Individual Modules Directly

For maximum flexibility, use the standalone classes without the pipeline orchestrator.

```python
from cleanflow import DataLoader, CleanFlower, EDAAnalyzer, FeatureEngineer, ModelTrainer
```

#### DataLoader

```python
loader = DataLoader()

# Load from file (CSV, Excel, JSON, Parquet)
df = loader.load("data.csv")
df = loader.load("data.xlsx")
df = loader.load("data.json")
df = loader.load("data.parquet")

# Load from existing DataFrame
df = loader.load(my_dataframe)

# Validate and print profile
report = loader.validate()
loader.print_summary()
# Reports: shape, memory usage, column types, missing values, duplicates,
#          constant columns, near-constant columns, potential ID columns
```

#### CleanFlower (Data Cleaner)

```python
cleaner = CleanFlower(df)

# Chain operations fluently
cleaner.remove_duplicates()

cleaner.handle_missing_values(
    numeric_strategy='median',               # 'mean', 'median', 'zero'
    categorical_strategy='mode',             # 'mode', 'unknown'
    drop_threshold=0.4                       # Drop cols with >40% missing
)

cleaner.fix_data_types()                     # Auto-infer numeric, datetime, category

cleaner.handle_outliers(
    method='iqr',                            # 'iqr' or 'zscore'
    threshold=1.5,                           # IQR multiplier or Z-score threshold
    action='clip'                            # 'clip', 'remove', 'nan'
)

cleaner.clean_categorical_values(
    lowercase=True,
    strip_whitespace=True,
    replace_mapping={"city": {"nyc": "new york", "sf": "san francisco"}}
)

cleaner.clean_numeric_text()                 # Converts "$1,200" → 1200, "1.5k" → 1500

cleaner.normalize_features(method='standard') # 'standard', 'minmax', 'robust'

cleaner.encode_categorical(method='onehot')  # 'onehot' or 'label'

cleaner.rename_columns({"old_name": "new_name"})

cleaner.extract_regex_feature(
    source_col="description",
    pattern=r'ID: (\d+)',
    new_col_name="extracted_id"
)

cleaner.drop_columns(
    columns=["unwanted_col"],
    drop_constant=True,                      # Remove single-value columns
    drop_id_like=True                        # Remove columns where every value is unique
)

# Data quality scoring (0–100)
quality = cleaner.validate_quality()
# Returns: score, grade (A/B/C/D/F), missing_values, duplicate_rows, constant_columns, warnings

# AI-like cleaning suggestions
suggestions = cleaner.generate_suggestions()
# Returns list of: {column, issue, action, reason} for every detected problem

# Get results
cleaned_df = cleaner.get_cleaned_data()
summary = cleaner.get_cleaning_summary()
# Summary includes: original_shape, final_shape, operations log, row-level change audit trail
cleaner.print_summary()
```

#### EDAAnalyzer

```python
eda = EDAAnalyzer(df, target_col="price")

# Individual analyses
stats = eda.summary_statistics()             # Descriptive stats + skewness + kurtosis
cat_summary = eda.categorical_summary()      # Value counts & percentages per categorical col
corr_matrix = eda.correlation_analysis(
    method='pearson',                        # 'pearson', 'kendall', 'spearman'
    threshold=0.7                            # Flag pairs above this correlation
)
distributions = eda.distribution_analysis()  # Mean, median, std, skewness, kurtosis, IQR, range
target_info = eda.target_analysis()          # Stats for continuous, class distribution for categorical

# Generate visualizations
eda.plot_distributions(save_path="dist.png")
eda.plot_boxplots(save_path="box.png")
eda.plot_correlation_heatmap(save_path="corr.png")
eda.plot_categorical(save_path="cat.png")
eda.plot_target_relationships(save_path="target.png")

# Or run everything at once
results = eda.run_full_analysis(show_plots=True, save_plots=True, output_dir="./plots")

# Get auto-generated insights
insights = eda.get_insights()
# Examples: "Highly skewed features detected: [col1, col2]"
#           "High cardinality in 'city': 342 unique values"
#           "Class imbalance detected in target: ratio = 0.12"
```

#### FeatureEngineer

```python
engineer = FeatureEngineer(df, target_col="churn", problem_type="classification")

# Encoding
engineer.encode_categorical(
    method='auto',                           # 'auto', 'onehot', 'label'
    max_categories=10,                       # OneHot threshold; Label above this
    drop_first=True
)

# Scaling
engineer.scale_features(method='standard')   # 'standard' or 'minmax'

# Datetime extraction
engineer.create_datetime_features(
    features=['year', 'month', 'day', 'dayofweek', 'quarter', 'hour', 'is_weekend']
)

# Polynomial & interaction features
engineer.create_polynomial_features(degree=2, interaction_only=False)

# Binned features
engineer.create_binned_features(n_bins=5, strategy='quantile')  # 'quantile' or 'uniform'

# Feature selection
engineer.compute_feature_importance(n_features=20)  # Uses mutual information
engineer.drop_low_importance_features(threshold=0.01)
engineer.drop_low_variance_features(threshold=0.01)
engineer.drop_highly_correlated(threshold=0.95)

# Class imbalance handling
engineer.handle_class_imbalance(
    method='smote',                          # 'smote', 'oversample', 'undersample'
    sampling_strategy='auto'
)

# Auto-evolve: automatically generates interaction features, evaluates them
# with mutual information, and keeps only the best ones
engineer.auto_evolve(max_new_features=10)

# Get results
final_df = engineer.get_transformed_data()
summary = engineer.get_summary()
engineer.print_summary()
```

#### ModelTrainer

```python
trainer = ModelTrainer(df, target_col="price", problem_type="regression")

# Full pipeline
results = trainer.run()
# Internally runs: detect_target → detect_problem_type → prepare_features →
#                  train_all_models → get_best_model → get_metrics_dashboard

# Or step by step:
trainer.detect_target()                      # Auto-detects target using heuristics
trainer.detect_problem_type()                # Classification vs regression by cardinality
trainer.prepare_features()                   # Encode, scale, impute, drop non-numeric
trainer.train_all_models()                   # CV + RandomizedSearchCV for all models
best = trainer.get_best_model()              # Select winner by F1 (clf) or R² (reg)
importance = trainer.get_feature_importance(top_n=10)
reliability = trainer.get_reliability_score() # 0–100 score + grade (A/B/C/D)
dashboard = trainer.get_metrics_dashboard()  # Complete results dict

# Export model as .pkl
trainer.export_model("./model.pkl")
# Saved: model, scaler, label_encoder, feature_names, metrics, reliability

# Models trained:
# Classification: Logistic Regression, Random Forest (with class_weight='balanced' if imbalanced)
# Regression:     Ridge Regression, Random Forest
# All with hyperparameter tuning via RandomizedSearchCV and 3-fold Stratified/KFold CV
```

---

### 4. Data Drift Detection

Compare a baseline dataset against new production data to detect schema and statistical drift.

```python
from cleanflow.drift_detector import DriftDetector

detector = DriftDetector(baseline_df=train_df, current_df=production_df)
report = detector.run()

print(report['schema_drift'])         # Missing or new columns
print(report['statistical_drift'])    # Per-column KS-test p-values
print(report['drifted_features'])     # List of columns with drift (p < 0.05)
print(report['drift_detected'])       # True/False
print(report['score'])                # Health score 0–100
```

Uses the **Kolmogorov-Smirnov test** on shared numeric columns. Penalizes schema changes and statistical deviations to produce a single health score.

---

### 5. Synthetic Data Generation

Generate privacy-safe statistical clones of your dataset using Kernel Density Estimation (KDE).

```python
from cleanflow.synthetic_generator import SyntheticGenerator

gen = SyntheticGenerator(df)
gen.fit()                                    # Learn distributions
synthetic_df = gen.generate(n_rows=1000)     # Generate 1000 synthetic rows

# Numeric columns: sampled from KDE (Gaussian kernel)
# Categorical columns: sampled from observed probability distribution
# Constant columns: replicated as-is
```

---

### 6. Working with DataFrames Directly

No file needed — pass a DataFrame directly.

```python
import pandas as pd
from cleanflow import DataPipeline

df = pd.DataFrame({
    'feature1': [1.0, 2.5, None, 4.0, 100.0],
    'category': ['A', 'B', 'A', None, 'B'],
    'target':   [0, 1, 0, 1, 0]
})

pipeline = DataPipeline()
pipeline.load(df)
cleaned_df, final_df = pipeline.run_full_pipeline(
    target_col='target',
    problem_type='classification'
)
```

---

### 7. Flask Web Application

An optional browser-based UI with drag-and-drop uploads, user authentication, and an iOS 26-inspired glassmorphism design.

```bash
pip install "cleanflow-kit[web,api]"
python app.py
```

---

## 📋 Complete Feature Reference

### Data Loading & Validation
| Feature | Details |
|---------|---------|
| Multi-format support | CSV, Excel (.xlsx/.xls), JSON, Parquet, DataFrame |
| Smart profiling | Auto-detects dtypes, missing values, duplicates |
| Noise detection | Flags constant columns, near-constant columns, potential ID columns |
| Memory reporting | Reports dataset memory usage in MB |

### Data Cleaning (CleanFlower)
| Feature | Details |
|---------|---------|
| Duplicate removal | Exact row deduplication |
| Missing value imputation | mean / median / zero (numeric), mode / "Unknown" (categorical) |
| Threshold-based column dropping | Drops columns exceeding a configurable missing % (default 40%) |
| Auto type inference | Converts object → numeric, object → datetime, low-cardinality → category |
| Outlier handling | IQR or Z-score detection with clip / remove / NaN actions |
| Categorical normalization | Strip whitespace, lowercase, custom value mapping |
| Numeric text parsing | Converts "$1,200" → 1200, "1.5k" → 1500, "2M" → 2000000 |
| Regex feature extraction | Extract structured data from text using capture groups |
| Column renaming | Batch rename via dictionary mapping |
| Column dropping | Drop by name, drop constants, drop ID-like columns |
| Feature scaling | Standard (Z-score), MinMax, Robust scalers |
| Categorical encoding | One-Hot or Label encoding |
| Quality scoring | 0–100 data quality score (A/B/C/D/F grade) based on completeness, uniqueness, consistency |
| Cleaning suggestions | Auto-generated per-column recommendations with issue, action, and reason |
| Row-level audit trail | Logs every individual cell change (old value → new value, operation, reason) |

### Exploratory Data Analysis (EDA)
| Feature | Details |
|---------|---------|
| Summary statistics | Count, mean, std, min/max, quartiles, skewness, kurtosis |
| Categorical profiling | Value counts and percentages per category |
| Correlation analysis | Pearson/Kendall/Spearman with high-correlation pair detection |
| Distribution analysis | Mean, median, std, skewness, kurtosis, IQR, range per feature |
| Target analysis | Continuous stats or class distribution + imbalance detection |
| Auto-generated insights | Flags skewed features, high-cardinality columns, class imbalance |
| Visualizations | Histograms, Boxplots, Correlation Heatmaps, Category Bar Charts, Feature-Target Scatters |

### Feature Engineering
| Feature | Details |
|---------|---------|
| Smart encoding | Auto-selects OneHot (≤ threshold) or Label (above threshold) per column |
| Feature scaling | StandardScaler or MinMaxScaler on numeric features |
| Datetime extraction | year, month, day, dayofweek, quarter, hour, minute, is_weekend |
| Polynomial features | Degree-N polynomial and/or interaction-only terms |
| Binned features | Quantile or uniform binning of numeric columns |
| Mutual information importance | Ranks features by mutual information with the target |
| Low-importance dropping | Removes features below an importance threshold |
| Low-variance dropping | Removes near-zero variance features via VarianceThreshold |
| High-correlation dropping | Removes one of each pair exceeding a correlation threshold |
| Class imbalance handling | SMOTE, random oversampling, or random undersampling |
| Auto-evolve | Automatically generates interaction features, evaluates with MI, keeps the best |

### AutoML Model Training
| Feature | Details |
|---------|---------|
| Auto target detection | Heuristic scan for columns named "target", "label", "price", "churn", etc. |
| Auto problem type detection | Classification (categorical/few unique) vs Regression (continuous) |
| Classification models | Logistic Regression, Random Forest (with balanced class weights if needed) |
| Regression models | Ridge Regression, Random Forest |
| Hyperparameter tuning | RandomizedSearchCV with configurable iteration budget |
| Cross-validation | 3-fold StratifiedKFold (classification) or KFold (regression) |
| Multi-metric evaluation | Accuracy, Precision, Recall, F1, ROC-AUC (clf) / R², MAE, RMSE (reg) |
| Imbalance detection | Flags majority class > 70% and auto-applies SMOTE if available |
| Reliability score | Custom 0–100 score based on dataset size, CV variance, metric quality (grade A/B/C/D) |
| Feature importance | Extracts from tree `feature_importances_` or linear `coef_` |
| Baseline comparison | Trains a naive Decision Tree on raw data to quantify cleaning improvement |
| Model export | Saves model + scaler + encoder + feature names + metrics as `.pkl` |

### Drift Detection
| Feature | Details |
|---------|---------|
| Schema drift | Detects missing or new columns between baseline and current data |
| Statistical drift | Kolmogorov-Smirnov test on shared numeric columns (p < 0.05 = drift) |
| Health score | 0–100 composite score penalizing schema and statistical drift |

### Synthetic Data Generation
| Feature | Details |
|---------|---------|
| Numeric columns | Kernel Density Estimation (Gaussian) with automatic bandwidth selection |
| Categorical columns | Probabilistic sampling from observed frequency distributions |
| Integer preservation | Rounds generated values for originally integer-typed columns |

### Report Generation
| Feature | Details |
|---------|---------|
| HTML report | Publication-quality report with executive summary, methodology, data profiling, model architecture, performance evaluation, and future recommendations |
| Markdown report | Lightweight report with data transformation summary, model performance table, and key observations |

---

## 🗂️ Package Structure

```
cleanflow/
├── __init__.py              # Package exports (DataPipeline, DataLoader, CleanFlower, etc.)
├── pipeline.py              # DataPipeline orchestrator
├── data_loader.py           # Multi-format data loading & validation
├── data_cleaner.py          # CleanFlower — 30+ cleaning operations
├── eda.py                   # EDAAnalyzer — statistics & visualizations
├── feature_engineer.py      # FeatureEngineer — encoding, scaling, selection
├── model_trainer.py         # ModelTrainer — AutoML with CV, tuning, reliability
├── drift_detector.py        # DriftDetector — schema & statistical drift
├── synthetic_generator.py   # SyntheticGenerator — KDE-based data generation
├── report_generator.py      # ReportGenerator — HTML/Markdown reports
└── _compat.py               # String normalization compatibility layer
```

---

## 🔌 API Quick Reference

```python
from cleanflow import DataPipeline

pipeline = DataPipeline()
pipeline.load(source)                        # str path or pd.DataFrame
pipeline.validate()                          # → dict
pipeline.clean(...)                          # → DataPipeline (chainable)
pipeline.analyze(...)                        # → dict
pipeline.engineer_features(...)              # → DataPipeline (chainable)
pipeline.run_full_pipeline(...)              # → (cleaned_df, final_df)
pipeline.train_model(...)                    # → dict (metrics dashboard)
pipeline.generate_html_report(path)          # → str (file path)
pipeline.generate_markdown_report(path)      # → str (file path)
pipeline.save_data(dir, format='csv')        # saves cleaned_data + final_data
pipeline.get_raw_data()                      # → pd.DataFrame
pipeline.get_cleaned_data()                  # → pd.DataFrame
pipeline.get_final_data()                    # → pd.DataFrame
pipeline.get_report()                        # → dict (full pipeline report)
```
