Metadata-Version: 2.4
Name: datasetsanity
Version: 0.0.1
Summary: Sanity checks for ML datasets: missing values, class imbalance, and data leakage.
Home-page: https://github.com/RiduanAziz/DataSetSanity-PyPi-Package
Author: RiduanAziz
Author-email: riduan.aziz46@gmail.com
License: MIT
Project-URL: Bug Tracker, https://github.com/RiduanAziz/DataSetSanity-PyPi-Package/issues
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8,<3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas<2.0.0,>=1.3.0
Requires-Dist: numpy<2.0.0,>=1.21.0
Requires-Dist: scikit-learn<1.2.0,>=1.0.0
Provides-Extra: dev
Requires-Dist: pytest==6.2.5; extra == "dev"
Requires-Dist: pytest-cov==3.0.0; extra == "dev"
Requires-Dist: tox==3.25.1; extra == "dev"
Requires-Dist: flake8==5.0.4; extra == "dev"
Requires-Dist: black==22.12.0; extra == "dev"
Requires-Dist: isort<6.0.0,>=5.10.1; extra == "dev"
Requires-Dist: mypy==0.991; extra == "dev"
Requires-Dist: types-requests==2.31.0; extra == "dev"
Requires-Dist: build==0.10.0; extra == "dev"
Requires-Dist: twine==4.0.2; extra == "dev"
Requires-Dist: setuptools==65.5.0; extra == "dev"
Requires-Dist: wheel==0.37.1; extra == "dev"
Requires-Dist: pre-commit==2.20.0; extra == "dev"
Requires-Dist: jupyterlab==3.6.3; extra == "dev"
Requires-Dist: coverage==6.5.0; extra == "dev"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# DatasetSanity 🧪

**DatasetSanity** is a lightweight Python package that helps data scientists and machine learning engineers identify common dataset issues before model training. It performs automated sanity checks on tabular datasets to detect missing values, severe class imbalance, and potential target leakage—issues that often lead to misleading model performance. DatasetSanity provides clear, actionable reports and integrates easily into existing ML workflows.

It helps detect common but critical data issues—such as missing values, class imbalance, and potential data leakage—*before* model training.

> Fix data problems before the model hides them.

![PyPI](https://img.shields.io/pypi/v/datasetsanity)
![Python](https://img.shields.io/pypi/pyversions/datasetsanity)
![License](https://img.shields.io/pypi/l/datasetsanity)
![CI](https://github.com/RiduanAziz/datasetsanity/actions/workflows/ci.yml/badge.svg)
---

## 🚀 Why DatasetSanity?

Machine learning models can silently fail or produce misleading results due to poor dataset quality.  
DatasetSanity is designed to **fail fast** by surfacing dataset issues early in the ML pipeline.

It is:
- ✅ Simple to use
- ⚡ Lightweight
- 📦 Easy to integrate
- 🎓 Student-friendly
- 🏗 Production-ready

---

## ✨ Features

- 🔍 **Missing Value Analysis**
  - Percentage of missing values per column
  - Detection of empty columns and sparse rows

- ⚠ **Class Imbalance Detection**
  - Binary and multi-class classification support
  - Configurable imbalance thresholds

- 🚨 **Data Leakage Detection**
  - Target-correlated features
  - Duplicate target columns
  - Train-test overlap checks (optional)

- 📊 **Clear Reports**
  - Human-readable console output
  - JSON output for pipelines (HTML planned)

---

## 📦 Installation
 **How to run?**
### Create a conda environment
```bash
conda create -n datasanity python=3.8 -y
```

```bash
conda activate datasanity
```

```bash
pip install -r requirements_dev.txt
```

```bash
pip install datasetsanity
```
---

## 🧠 Quick Start (Python API)

```python
from DatasetSanity import DatasetSanity
import pandas as pd

df = pd.read_csv("data.csv")

ds = DatasetSanity(
    df=df,
    target="label",
    task="classification"
)

report = ds.run()
report.summary()
```
---

## 🖥 Command Line Interface (CLI)

```bash
datasetsanity check data.csv --target label
```

### Example output:
```bash
✔ Missing values check passed
⚠ Class imbalance detected
❌ Potential data leakage found
```
---

## 📄 Report Export
```python
report.to_json("report.json")
```

**(HTML reports planned in future releases.)**
---

## 🎯 Use Cases
- ML students validating datasets
- Data scientists performing pre-model checks
- ML engineers integrating dataset validation into pipelines
- Educators teaching data quality concepts

---

## 📁 Project Structure

```bash
DataSetSanity-PyPi-Package/
├── .github/
│   └── workflows/
│       ├── .gitkeep
│       └── ci.yml
├── docs/                # optional documentation
├── src/
│   └── datasetsanity/
│       ├── __init__.py
│       ├── custom_exception.py
│       ├── logger.py
│       └── py.typed
├── tests/
│   ├── __init__.py
│   ├── unit/
│   │   ├── __init__.py
│   │   └── test_unit.py
│   └── integration/
│       ├── __init__.py
│       └── test_int.py
├── .gitignore
├── LICENSE
├── MANIFEST.in
├── README.md
├── CONTRIBUTING.md
├── requirements.txt
├── requirements_dev.txt
├── setup.py
├── setup.cfg
├── pyproject.toml
├── template.py           # cookiecutter template
├── pypi.ipynb            # PyPI release notebook
├── pypi.excalidraw       # PyPI release diagram
├── tox.ini
└── test.py              # sandbox testing
```
---

## 🛣 Roadmap
- Regression task support
- Feature drift detection
- HTML & visual reports
- sklearn pipeline integration
- CI/CD dataset checks
---

## 🤝 Contributing

Contributions are welcome!
Please read CONTRIBUTING.md
 before submitting issues or pull requests.
---

## 📄 License

This project is licensed under the MIT License.
See LICENSE for details.

---
