Metadata-Version: 2.4
Name: Datascrubber
Version: 0.3.0
Summary: A data cleaning and visualisation toolkit for data science projects
Project-URL: Homepage, https://github.com/muganga-charles
Author-email: Charles Muganga <mugangacharles5@gmail.com>
License: MIT
License-File: LICENSE
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Requires-Dist: matplotlib
Requires-Dist: missingno
Requires-Dist: numpy
Requires-Dist: openpyxl
Requires-Dist: pandas>=2.0
Requires-Dist: scikit-learn
Requires-Dist: scipy
Requires-Dist: seaborn
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: fastio
Requires-Dist: pyarrow; extra == 'fastio'
Provides-Extra: imbalance
Requires-Dist: imbalanced-learn; extra == 'imbalance'
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'
Description-Content-Type: text/markdown

# Datascrubber

**A data cleaning, preprocessing and visualisation toolkit for data science projects.**

Datascrubber wraps the tedious parts of a data-science workflow — reading messy
files, fixing types, handling missing values and outliers, encoding and scaling
for modelling, checking data quality, and visualising relationships — behind one
class with a consistent, chainable API. Every cleaning action is logged, and
fit-based transforms can be saved as a **recipe** and replayed on new data
without leakage.

- **Author:** Charles Muganga
- **Licence:** MIT
- **Python:** 3.10+

---

## Table of contents

- [Installation](#installation)
- [Quickstart](#quickstart)
- [Core concepts](#core-concepts)
- [Reading and writing data](#reading-and-writing-data)
- [Profiling and quality](#profiling-and-quality)
- [Cleaning: missing values, duplicates, outliers](#cleaning)
- [Preprocessing for modelling](#preprocessing-for-modelling)
- [Reproducible recipes and scikit-learn](#reproducible-recipes-and-scikit-learn)
- [Class imbalance](#class-imbalance)
- [Visualisation](#visualisation)
- [The one-call pipeline](#the-one-call-pipeline)
- [Project structure](#project-structure)
- [Method reference](#method-reference)
- [Testing](#testing)

---

## Installation

From the project root (the folder containing `pyproject.toml`):

```bash
pip install Datascrubber
```
---

## Quickstart

```python
from Datascrubber import Datacleaning

dc = Datacleaning()
dc.read_data("Datasets/DiamondPricesData.xlsx")   # auto-describes the file

dc.auto_clean()          # standardise names, dedupe, impute, clip outliers
dc.savedata()            # -> cleaned_data.xlsx
```

`auto_clean()` prints a short summary:

```
Cleaning complete:
- Missing values have been handled (20 filled).
- Outliers have been handled.
- 146 duplicate row(s) removed.
- Column names standardised.

Call report() for the detailed, per-column log.
```

Everything is **chainable** (methods return `self`) and **logged**
(`dc.report()` gives the full per-column detail).

---

## Core concepts

**One object, shared state.** A `Datacleaning` instance holds your dataframe and
every operation acts on it in place. Internally the class is composed of small
focused mixins, but the public
surface is a single flat API.

**Fluent chaining.** Mutating methods return `self`:

```python
dc.standardize_columns().drop_duplicates().impute().remove_outliers()
```

**Action log.** Every change is recorded. `dc.report()` returns the detailed log;
`dc.diff(before)` summarises what changed between two `snapshot()`s.

**Notebook-friendly returns.** Plots and pipelines return `self` (not a raw
figure or string) so notebooks don't double-render images or echo escaped text.
Pass `show=False` to a plot to get the figure back for saving/embedding instead.

---

## Reading and writing data

```python
dc.read_data("data.csv")                       # csv, xlsx, xls, json, txt, parquet, feather
dc.read_data("book.xlsx", sheet_name="Sheet2") # pick a sheet (name or index)
dc.read_url("https://example.com/data.csv")    # straight from a URL
dc.read_sql("SELECT * FROM sales", engine)      # any pandas-compatible connection

dc.savedata()                 # mirrors the input format -> cleaned_data.<ext>
dc.savedata("out.parquet")    # or choose a path/format explicitly
```

Multi-sheet Excel files no longer block on a prompt: with no `sheet_name` given,
the first sheet is read and a note tells you how to pick another.

---

## Profiling and quality

```python
dc.summary()          # describe(include="all")
dc.columns()          # column index
dc.missing_values()   # per-column missing counts (+ a missingno matrix)
dc.data_types()       # split into categorical vs continuous

dc.smart_types()      # coerce "1,234" -> number, "yes"/"no" -> bool, dates -> datetime

dc.health_check()     # 0-100 quality score + flagged issues
```

`health_check()` flags missing data, duplicates, constant / quasi-constant
columns, high-cardinality columns, and mixed-type columns, and returns a dict:

```python
{"score": 99.7, "rows": 53940, "cols": 10, "issues": ["146 duplicate row(s)."]}
```

**Schema validation** checks your data against expectations:

```python
dc.validate_schema({
    "price": {"dtype": "int", "min": 0},
    "cut":   {"allowed": ["Ideal", "Premium", "Very Good", "Good", "Fair"]},
})
```

---

## Cleaning

**Missing values** — `impute()` supports several strategies:

```python
dc.impute()                              # 'auto': mode for categoricals; for
                                         # numerics, median if skewed else mean
dc.impute(strategy="median")
dc.impute(strategy="knn")                # KNN imputation (numeric columns)
dc.impute(strategy="ffill")              # forward fill
dc.impute(strategy="median", group_by="region")   # group-wise
dc.drop_missing_values()                 # or just drop rows with NaNs
```

**Duplicates and structure:**

```python
dc.drop_duplicates()
dc.standardize_columns()                 # snake_case, lowercase, de-punctuated
dc.drop(["id", "notes"])
dc.rename("old", "new")
dc.remove_empty_columns()
```

**Outliers** — clip with the IQR method, or use the flexible detector:

```python
dc.remove_outliers()                     # clip all numeric columns to IQR fences

dc.detect_outliers("price", method="zscore", action="flag")    # add 'is_outlier'
dc.detect_outliers(method="iqr", action="remove")              # drop outlier rows
dc.detect_outliers(method="isolation_forest", action="flag")   # multivariate
```

`method` ∈ `iqr`, `zscore`, `modified_zscore`, `isolation_forest`;
`action` ∈ `flag`, `clip`, `remove`.

**Datetime features:**

```python
dc.parse_dates("order_date")
dc.extract_date_features("order_date")   # _year, _month, _day, _dayofweek,
                                         # _quarter, _is_weekend
```

---

## Preprocessing for modelling

**Encoding** categorical columns:

```python
dc.encode("cut", method="onehot")                    # dummy columns
dc.encode("grade", method="ordinal",
          ordering={"grade": ["low", "mid", "high"]})
dc.encode("city", method="frequency")                # replace with frequency
dc.encode("city", method="target", target="price")   # smoothed target mean
dc.encode(["a", "b"], method="label")
```

**Scaling** and **binning**:

```python
dc.scale(["price", "carat"], method="standard")      # standard / minmax / robust
dc.scale(method="minmax")                             # all numeric columns
dc.bin_column("age", bins=4, strategy="quantile")    # -> age_binned
```

---

## Reproducible recipes and scikit-learn

Fit-based transforms (`impute`, `encode`, `scale`) record the parameters they
learn. Save that **recipe** and replay it on a test set to get identical
transformations **without leakage** (no re-fitting on test data):

```python
# --- fit on training data ---
train = Datacleaning(); train.dataframe = X_train
train.impute(strategy="median") \
     .encode("cut", method="onehot") \
     .scale(["price", "carat"], method="standard")
train.save_recipe("recipe.json")

# --- apply the SAME learned params to test data ---
test = Datacleaning()
test.load_recipe("recipe.json").apply_recipe(X_test)
X_test_clean = test.getdata()
```

Or use the **scikit-learn transformer** in a normal `Pipeline`:

```python
from Datascrubber import DatascrubberTransformer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

pre = DatascrubberTransformer(
    impute=True,
    scale="standard",
    encode={"columns": ["cut"], "method": "onehot"},
)

model = Pipeline([("prep", pre), ("clf", LogisticRegression())])
model.fit(X_train, y_train)      # prep fits on train only
model.predict(X_test)            # prep transforms test with train-learned params
```

---

## Class imbalance

```python
dc.class_balance("target")                  # distribution + imbalance ratio
dc.balance("target", method="oversample")   # duplicate minority up to majority
dc.balance("target", method="undersample")  # sample majority down to minority
dc.balance("target", method="smote")        # synthesise minority (needs
                                            # imbalanced-learn; falls back to
                                            # oversampling if unavailable)
```

SMOTE requires all features to be numeric — encode categoricals first.

---

## Visualisation

All plots render once inline and return `self`; pass `show=False` to get the
matplotlib figure back for saving or embedding.

```python
dc.distributions()          # histogram grid of continuous columns
dc.cat_dist()               # countplot grid of categorical columns
dc.corr_matrix()            # correlation heatmap
dc.outliers()               # boxplot grid

dc.cont_to_cont("price", "carat")     # scatter + correlation
dc.cat_to_cat("cut", "color")         # countplot + contingency + chi-square
dc.cont_to_cat("price", "cut")        # boxplot + one-way ANOVA
dc.lineplot("cut", "price")

fig = dc.corr_matrix(show=False)      # get the figure instead of showing it
fig.savefig("corr.png")
```

An HTML profile report bundles the dataset overview and cleaning log:

```python
dc.profile_report("report.html")
```

---

## The one-call pipeline

```python
dc.auto_clean(
    impute_strategy="auto",   # any impute() strategy
    standardize=True,         # snake_case column names
    dedupe=True,              # drop duplicate rows
    handle_outliers=True,     # clip to IQR fences
    verbose=True,             # print the short summary
)
```

`auto_clean()` resets the log, runs the steps, prints a concise summary and
returns `self`. The detailed per-column log is always available via `report()`.
`data_cleaning()` is kept as a backward-compatible alias (impute + clip).

---

## Method reference

| Category | Methods |
|----------|---------|
| **Read / write** | `read_data`, `read_url`, `read_sql`, `savedata` |
| **Profile** | `summary`, `columns`, `head`, `data_types`, `cat_cols`, `cont_cols`, `missing_values`, `col_missing_value`, `data_explanation`, `smart_types`, `snapshot`, `diff` |
| **Quality** | `health_check`, `validate_schema` |
| **Clean** | `impute`, `remove_missingvalues`, `drop_missing_values`, `drop_duplicates`, `standardize_columns`, `drop`, `rename`, `remove_empty_columns`, `remove_outliers`, `remove_outliers_single`, `detect_outliers` |
| **Dates** | `parse_dates`, `extract_date_features` |
| **Encode / scale** | `encode`, `scale`, `bin_column` |
| **Imbalance** | `class_balance`, `balance` |
| **Recipe** | `get_recipe`, `save_recipe`, `load_recipe`, `apply_recipe`, `DatascrubberTransformer` |
| **Visualise** | `distributions`, `col_dist`, `cat_dist`, `col_cat_dist`, `corr_matrix`, `cont_corr`, `outliers`, `outliers_single`, `cont_to_cont`, `cat_to_cat`, `cont_to_cat`, `lineplot`, `countplot`, `contingency_table`, `Chi_square`, `singleAnova`, `binary_anova` |
| **Pipelines / report** | `auto_clean`, `data_cleaning`, `report`, `profile_report` |

---