Metadata-Version: 2.4
Name: cleanengine
Version: 0.1.2
Summary: The Ultimate Data Cleaning & Analysis Toolkit
Home-page: https://github.com/I-invincib1e/CleanEngine
Download-URL: https://github.com/I-invincib1e/CleanEngine/archive/v0.1.1.tar.gz
Author: CleanEngine Community
Author-email: CleanEngine Community <contact@cleanengine.com>
Maintainer-email: CleanEngine Community <contact@cleanengine.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/I-invincib1e/CleanEngine
Project-URL: Documentation, https://github.com/I-invincib1e/CleanEngine#readme
Project-URL: Repository, https://github.com/I-invincib1e/CleanEngine
Project-URL: Bug Tracker, https://github.com/I-invincib1e/CleanEngine/issues
Project-URL: Changelog, https://github.com/I-invincib1e/CleanEngine/blob/main/CHANGELOG.md
Keywords: data-cleaning,data-analysis,data-profiling,machine-learning,data-science,pandas,scikit-learn,data-validation,rule-engine,cli-tool,data-quality,outlier-detection,clustering,anomaly-detection,correlation-analysis,feature-importance,data-visualization,streamlit,yaml-config,automation
Platform: any
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Filters
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Visualization
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=1.5.0
Requires-Dist: numpy>=1.21.0
Requires-Dist: scikit-learn>=1.1.0
Requires-Dist: scipy>=1.9.0
Requires-Dist: matplotlib>=3.5.0
Requires-Dist: seaborn>=0.11.0
Requires-Dist: plotly>=5.0.0
Requires-Dist: kaleido>=0.2.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: pyarrow>=8.0.0
Requires-Dist: fastparquet>=0.8.0
Requires-Dist: openpyxl>=3.0.0
Requires-Dist: xlrd>=2.0.0
Requires-Dist: psutil>=5.8.0
Requires-Dist: rich>=12.0.0
Requires-Dist: typer[all]>=0.7.0
Requires-Dist: questionary>=1.10.0
Requires-Dist: watchdog>=2.1.0
Requires-Dist: streamlit>=1.20.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: pytest-mock>=3.10.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: flake8>=6.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: pre-commit>=3.0.0; extra == "dev"
Requires-Dist: tox>=4.0.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=6.0.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.2.0; extra == "docs"
Requires-Dist: myst-parser>=1.0.0; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints>=1.23.0; extra == "docs"
Provides-Extra: full
Requires-Dist: jupyter>=1.0.0; extra == "full"
Requires-Dist: ipython>=8.0.0; extra == "full"
Requires-Dist: notebook>=6.5.0; extra == "full"
Dynamic: author
Dynamic: download-url
Dynamic: home-page
Dynamic: license-file
Dynamic: platform
Dynamic: requires-python

# 🧹 CleanEngine

[![GitHub stars](https://img.shields.io/github/stars/I-invincib1e/CleanEngine?style=social)](https://github.com/I-invincib1e/CleanEngine)
[![GitHub forks](https://img.shields.io/github/forks/I-invincib1e/CleanEngine?style=social)](https://github.com/I-invincib1e/CleanEngine)
[![GitHub issues](https://img.shields.io/github/issues/I-invincib1e/CleanEngine)](https://github.com/I-invincib1e/CleanEngine/issues)
[![PyPI version](https://badge.fury.io/py/cleanengine.svg)](https://badge.fury.io/py/cleanengine)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](#)
[![License: MIT](https://img.shields.io/badge/license-MIT-yellow)](./LICENSE)
[![Tests](https://img.shields.io/badge/tests-passing-brightgreen)](#)
[![Downloads](https://img.shields.io/pypi/dm/cleanengine)](https://pypi.org/project/cleanengine/)

> **🚀 The Ultimate Data Cleaning & Analysis CLI Tool**  
> Transform messy datasets into clean, insights-rich data with intelligent cleaning and advanced ML analysis.

CleanEngine is a powerful command-line toolkit that handles missing values, removes duplicates, detects outliers, and provides comprehensive statistical analysis using machine learning techniques.

![CleanEngine Demo](https://img.shields.io/badge/demo-available-blue)

### 📊 **Comparison with Other Tools**

| Feature | **CleanEngine** 🧹 | pandas-profiling | Sweetviz | Great Expectations |
|---------|-------------------|------------------|-----------|-------------------|
| **Data Cleaning** | ✅ **Complete Pipeline** | ❌ No | ❌ No | ⚠️ Limited |
| **Profiling & Stats** | ✅ **Advanced Analytics** | ✅ Yes | ✅ Yes | ⚠️ Minimal |
| **Correlation Analysis** | ✅ **Multi-Method** | ✅ Yes | ✅ Yes | ❌ No |
| **Feature Importance** | ✅ **ML-Powered** | ❌ No | ❌ No | ❌ No |
| **Clustering & Patterns** | ✅ **3 Algorithms** | ❌ No | ❌ No | ❌ No |
| **Anomaly Detection** | ✅ **2 Methods** | ❌ No | ❌ No | ❌ No |
| **Rule Engine** | ✅ **YAML-Driven** | ❌ No | ❌ No | ✅ Yes |
| **Interfaces** | ✅ **CLI + GUI + Watcher** | CLI/Notebook | Notebook | CLI/Notebook |
| **Automation** | ✅ **Folder Watcher** | ❌ No | ❌ No | ✅ Yes |

---

## 🚀 Installation

### Using pip (Recommended)
```bash
pip install cleanengine
```

### From source
```bash
git clone https://github.com/I-invincib1e/CleanEngine.git
cd CleanEngine
pip install -e .
```

### Verify Installation
```bash
cleanengine --help
```

---

## 🎯 Quick Start

### Clean a CSV file
```bash
cleanengine clean data.csv
```

### Analyze data without cleaning
```bash
cleanengine analyze data.xlsx
```

### Generate sample data to test
```bash
cleanengine samples
```

### Launch web interface
```bash
cleanengine gui
```

---

## 📋 CLI Commands

### Core Commands
| Command | Flags | Description | Example |
|---------|-------|-------------|---------|
| `clean` | `--output, -o`, `--verbose, -v`, `--force` | Clean a dataset with full pipeline | `cleanengine clean data.csv --output ./cleaned/ --verbose` |
| `analyze` | `--output, -o`, `--verbose, -v` | Analyze data without cleaning | `cleanengine analyze data.csv --output ./analysis/ --verbose` |
| `validate-data` | `--verbose, -v` | Validate data with rules | `cleanengine validate-data data.csv --verbose` |
| `profile` | `--output, -o`, `--verbose, -v` | Generate data profile report | `cleanengine profile data.csv --output ./profile/ --verbose` |
| `clean-only` | `--output, -o`, `--verbose, -v` | Clean without analysis | `cleanengine clean-only data.csv --output ./cleaned/ --verbose` |
| `samples` | `--output, -o`, `--count, -n`, `--verbose, -v` | Create sample datasets | `cleanengine samples --output ./samples/ --count 5 --verbose` |
| `test` | `--verbose, -v`, `--coverage` | Run test suite | `cleanengine test --verbose --coverage` |
| `gui` | `--port, -p`, `--host, -h` | Launch Streamlit web interface | `cleanengine gui --port 8501 --host localhost` |
| `info` | None | Show CleanEngine information | `cleanengine info` |

### Advanced Analysis Commands
| Command | Flags | Description | Example |
|---------|-------|-------------|---------|
| `correlations` | `--method, -m`, `--threshold, -t`, `--output, -o`, `--verbose, -v` | Analyze variable correlations | `cleanengine correlations data.csv --method pearson --threshold 0.7 --verbose` |
| `features` | `--output, -o`, `--verbose, -v` | Analyze feature importance | `cleanengine features data.csv --output ./features/ --verbose` |
| `clusters` | `--method, -m`, `--output, -o`, `--verbose, -v` | Discover data clusters | `cleanengine clusters data.csv --method kmeans --output ./clusters/ --verbose` |
| `anomalies` | `--method, -m`, `--contamination, -c`, `--output, -o`, `--verbose, -v` | Detect anomalies/outliers | `cleanengine anomalies data.csv --method isolation_forest --contamination 0.1 --verbose` |
| `quality` | `--output, -o`, `--verbose, -v` | Assess data quality | `cleanengine quality data.csv --output ./quality/ --verbose` |
| `statistics` | `--output, -o`, `--verbose, -v` | Perform statistical analysis | `cleanengine statistics data.csv --output ./stats/ --verbose` |

---

## 📁 Supported File Formats

- **CSV**: Comma-separated values
- **Excel**: .xlsx and .xls files
- **JSON**: JavaScript Object Notation
- **Parquet**: Columnar storage format

---

## 📊 Output Structure

After processing, CleanEngine creates a `Cleans-<dataset_name>/` folder with:

```
Cleans-data/
├── cleaned_data.csv          # Your cleaned dataset
├── cleaning_report.json      # Detailed cleaning summary
├── analysis_report.json      # Comprehensive analysis results
├── visualizations/           # Generated charts and plots
└── logs/                     # Processing logs
```

---

## ⚙️ Configuration

### Custom Configuration File
Create a `config.yaml` file in your working directory:

```yaml
cleaning:
  missing_values:
    strategy: "auto"  # auto, mean, median, mode, drop
  outliers:
    method: "iqr"     # iqr, zscore, custom
  encoding:
    categorical: true
    normalize: true

analysis:
  correlation:
    method: "pearson"  # pearson, spearman, kendall
  clustering:
    method: "kmeans"   # kmeans, dbscan, hierarchical
```

---

## 🎨 CLI Features

- **Rich Terminal Output**: Beautiful tables, progress bars, and colors
- **Interactive Help**: `cleanengine --help` and `cleanengine <command> --help`
- **Auto-completion**: Tab completion for commands and file paths
- **Progress Tracking**: Real-time progress bars for long operations
- **Error Handling**: Clear error messages with suggestions

---

## 📈 Performance

- **Small Datasets** (< 1MB): < 1 second
- **Medium Datasets** (1-100MB): 1-30 seconds
- **Large Datasets** (100MB-1GB): 30 seconds - 5 minutes
- **Very Large Datasets** (> 1GB): Configurable chunking

---

## 🔧 Advanced Usage

### Batch Processing Multiple Files
```bash
# Process all CSV files in current directory
for file in *.csv; do cleanengine clean "$file"; done
```

### Custom Output Directory
```bash
cleanengine clean data.csv --output-dir ./my-clean-data/
```

### Configuration File
```bash
cleanengine clean data.csv --config ./my-config.yaml
```

### Verbose Output
```bash
cleanengine clean data.csv --verbose
```

---

## 🐍 Python API

For programmatic use:

```python
from cleanengine import DatasetCleaner

# Initialize cleaner
cleaner = DatasetCleaner()

# Clean dataset
cleaned_df = cleaner.clean_dataset('data.csv')

# Get analysis results
analysis_results = cleaner.analyze_dataset('data.csv')
```

---

## 🤝 Contributing

We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details on:

- Setting up a development environment
- Code style and standards
- Testing and quality assurance
- Pull request process

---

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

---

## 🙏 Acknowledgments

- **pandas** for data manipulation
- **scikit-learn** for machine learning algorithms
- **Typer & Rich** for beautiful CLI interfaces
- **Streamlit** for web interface

---

<div align="center">

**Made with ❤️ for data scientists and analysts**

[GitHub](https://github.com/I-invincib1e/CleanEngine) •
[PyPI](https://pypi.org/project/cleanengine/) •
[Documentation](https://github.com/I-invincib1e/CleanEngine#readme)

</div>
