Metadata-Version: 2.4
Name: datapruning
Version: 2.1.6
Summary: Intelligent dataset optimization for cleaner, smaller training data
License: Proprietary
Project-URL: Homepage, https://www.datapruning.com
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: scikit-learn>=1.3.0
Provides-Extra: torch
Requires-Dist: torch>=2.0.0; extra == "torch"
Provides-Extra: clip
Requires-Dist: torch>=2.0.0; extra == "clip"
Requires-Dist: transformers>=4.30.0; extra == "clip"
Requires-Dist: pillow>=9.0.0; extra == "clip"
Dynamic: license-file

# DataPruning

Intelligent dataset optimization — reduces dataset size by selecting the most informative rows while preserving model quality.

## Installation

```bash
pip install datapruning
```

For image/text multimodal support (CLIP encoding):

```bash
pip install datapruning[clip]
```

## Quick Start

### CSV / Tabular Data

```python
import pandas as pd
from datapruning.sdk import DatasetEngine

df = pd.read_csv("dataset.csv")
engine = DatasetEngine(df, target_col="label")

# Analyze the dataset
analysis = engine.analyze()

# Get a recommendation
rec = engine.recommend()
print(f"Best method: {rec.display_name} ({rec.confidence}% confidence)")

# Optimize — keep 50% of most informative rows
result = engine.optimize(keep_ratio=0.5)
print(f"Kept {result['runtime_seconds']:.1f}s — {result['explainability']['rows_removed']} rows removed")

# Download the optimized dataset
result["optimized_df"].to_csv("optimized.csv", index=False)
```

### Image Datasets

```python
from datapruning.sdk import DatasetEngine

# Create engine from a folder of images
engine = DatasetEngine.from_images(
    folder_path="./photos",
    labels=[0, 1, 0, 1, ...],  # one label per image
    keep_ratio=0.5,
)

# Analyze and optimize
analysis = engine.analyze()
result = engine.optimize(keep_ratio=0.5)
result["optimized_df"].to_csv("optimized_images.csv", index=False)
```

### Text Datasets

```python
engine = DatasetEngine.from_texts(
    texts=["label A sample", "label B sample", ...],
    labels=[0, 1, 0, 1, ...],
)
result = engine.optimize(keep_ratio=0.5)
```

## Algorithms

DataPruning selects the best algorithm automatically based on dataset characteristics:

| Method | What it does | Best for |
|--------|-------------|----------|
| **Margin Optimization** | Analyzes classification confidence gap via cross-validation, keeps boundary samples | General tabular data |
| **Importance Scoring** | Measures sample difficulty via model prediction confidence across CV folds | Quick baseline, any dataset |
| **Diversity Selection** | Picks the most representative subset (K-Center) | High-dimensional data |
| **Noise Filtering** | Drops redundant and potentially mislabeled samples via CV-based difficulty | Noisy/messy data |
| **Full Pipeline** | Combines all signals for maximum quality | Complex datasets |

All scoring methods use **out-of-fold (OOF) model logits** — each sample is scored by a model that never saw it during training, producing unbiased difficulty estimates. The optimizer also includes a **"must beat random" gate** that validates the selected strategy actually improves over random subsampling on an internal holdout; if not, it safely falls back to random selection.

## Limits

- Minimum: 1,000 rows (tabular) or 10 images
- Maximum: 300,000 rows (tabular) or 1,000 images
- Supported image formats: JPG, PNG, BMP, GIF, TIFF, WebP

## Requirements

- Python >= 3.10
- Pandas 2.0+
- NumPy 1.24+
- scikit-learn 1.3+

Optional (for image/text support):
- PyTorch 2.0+
- transformers 4.30+
- Pillow 9.0+

## Features

- Runs locally — no data uploaded anywhere
- Auto-detects task type (classification/regression) and target column
- Auto-encodes categorical/string columns
- 5 optimization algorithms with automatic selection
- Preserves minority class balance
- CSV and image dataset support
- Pure Python — works on any platform

## Links

- Website: [datapruning.com](https://www.datapruning.com)
- PyPI: [pypi.org/project/datapruning](https://pypi.org/project/datapruning/)

## License

Proprietary. See LICENSE file.
