Metadata-Version: 2.4
Name: pycurate
Version: 0.4.0
Summary: Automatic data curation and curation auto-research: signals as zero-copy columns, recipes as views, agents that search recipes against a fixed eval.
License: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: pyarrow>=14
Requires-Dist: lancedb>=0.40
Requires-Dist: pylance>=0.30
Provides-Extra: torch
Requires-Dist: torch>=2.2; extra == "torch"
Provides-Extra: text
Requires-Dist: transformers>=4.45; extra == "text"
Requires-Dist: tiktoken; extra == "text"
Requires-Dist: fasttext-numpy2; extra == "text"
Provides-Extra: vision
Requires-Dist: transformers>=4.45; extra == "vision"
Requires-Dist: open_clip_torch; extra == "vision"
Requires-Dist: pillow; extra == "vision"
Requires-Dist: webdataset; extra == "vision"
Provides-Extra: dense
Requires-Dist: cleanlab>=2.6; extra == "dense"
Requires-Dist: pycocotools; extra == "dense"
Requires-Dist: scipy; extra == "dense"
Requires-Dist: transformers>=4.45; extra == "dense"
Requires-Dist: pillow; extra == "dense"
Provides-Extra: research
Requires-Dist: optuna; extra == "research"
Requires-Dist: anthropic; extra == "research"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: scikit-learn>=1.2; extra == "test"
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.25; extra == "docs"
Provides-Extra: all
Requires-Dist: pycurate[dense,research,text,torch,vision]; extra == "all"
Dynamic: license-file

# pycurate

Curation is a compute multiplier: train on the right subset of your data and you reach the same quality with less
compute. pycurate does it in one call. Point it at a table of training data. It detects what kind of data it is,
scores every row with the signals that matter for that kind, applies the strongest established public filter, and
hands you the subset to train on as a view: read straight from the table without copying it, in a globally shuffled
order. Seven kinds of data are routed from the schema: web text, noisy labels, image-text pairs, SFT chats,
preference pairs, RL prompts and robot episodes. Nothing in a default path calls an LLM. Give it your evaluation and
it tunes the recipe; add `agent=True` and an LLM agent surveys the literature and writes recipes. A tuned recipe is
applied only if it beats the default on your held-out metric. This is research code, and APIs may change.

## Install

```bash
pip install "pycurate[all]"   # torch, transformers, fastText (DCLM), open_clip, optuna (TPE search), anthropic (opt-in agent)
# from a checkout: pip install -e ".[all]"
```

## One verb, three levels

```python
import pycurate as pc

ds = pc.load("data/*.parquet", name="pool", uri="./db")      # parquet, pandas, pyarrow, hf://... into a Lance table

def train_eval(recipe, ids):                                 # your proxy training; higher is better
    model = train(ds.view(ids, columns=["text"]))
    return {"score": evaluate(model), "heldout": evaluate(model)}   # use a held-out set the search never sees
```

```python
cur = pc.curate(ds)                                          # 1. default: domain recipe, no LLM, no training
cur = pc.curate(ds, eval=train_eval, budget=8)               # 2. tuned: TPE on your eval, guarded by the default
cur = pc.curate(ds, eval=train_eval, budget=8, agent=True)   # 3. agent: literature signals + LLM recipes, guarded
print(cur, cur.report["level"], cur.report["guard"])         # what ran, what was picked and why
loader = cur.to_torch(columns=["text"], batch_size=256)      # zero-copy, globally shuffled batches
```

`pc.diagnose(ds)` runs pre-flight checks before you spend compute. Level 2 is TPE by default (no LLM); level 3 is
opt-in and the only path that calls an LLM (a logged-in `claude` CLI, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` or the
`codex` CLI). Without `eval=`, `agent=True` only runs signal discovery and applies the default. Every signal is a
column, cataloged with its citations and provenance. Tables live on local disk or an object store (S3).

## Defaults

| kind of data | detected from | `pc.curate(ds)` applies | vs no curation |
|---|---|---|---|
| web text | one long text column | DCLM fastText top 10% + SemDeDup | 1.8× (GPT-2-small, 4B tokens) |
| image-text pairs | image + caption | DFN top 30% + SemDeDup | 3.3× (DataComp-medium) |
| noisy labels | a class label column | lowest k-NN label disagreement per class | 1.0-6.0×, median 2.8× (13 tasks) |
| SFT chats | `messages` or prompt/response | reward-model top 50% + prompt SemDeDup | 2.2× (Tulu-3 → Qwen2.5-0.5B) |
| preference pairs | prompt/chosen/rejected | top 75% by reward-model margin | better held-out z; no curve |
| RL prompts | problem/answer | learnability: 0 < pass rate < 1, ranked by p(1-p) | ≈1× |
| robot episodes | episode/action columns | lowest k-NN task-label disagreement per task | n.s. on small sims |
| object detection / instance masks | box-list or COCO annotation column | ObjectLab: drop flagged images | not benchmarked |
| semantic segmentation | image + label map | cleanlab.segmentation score ≥ 0.1 | not benchmarked |

Each is a compute multiplier against training on the whole pool at matched quality, read off measured curves and
never extrapolated (gross; DataComp-medium is the 80.2M-pair pool). An agent-searched recipe scored above DFN top 30%
there on all 3 seeds (3.90×, p = 0.051). Net of curation compute, and against the strongest public filter, see
[results](docs/results.md#headline-results). Fallbacks when a model cannot run are logged in `cur.report`.

## Docs

`pip install -e ".[docs]" && mkdocs serve`, or read them in [docs/](docs/index.md):
[quickstart](docs/quickstart.md) · [which level do I need?](docs/which_level.md) ·
[the guard](docs/guard.md) · [guides](docs/guides/lm_pretraining.md) · [agent mode](docs/agent.md) ·
[API](docs/api/index.md) · [results](docs/results.md) · [benchmarks](docs/benchmarks.md) ·
[contributing](docs/contributing.md).
Runnable before/after examples on real noisy data (CIFAR-10N, FineWeb, COCO, VOC): [docs/examples.md](docs/examples.md#example-scripts).

**Continuing the project?** Start at [HANDOFF.md](HANDOFF.md) (state, results, open items); the full account of what was built and why is [docs/DEVELOPMENT_HISTORY.md](docs/DEVELOPMENT_HISTORY.md), and the day-by-day log is [bench/PLAN.md](bench/PLAN.md).

## License

Apache-2.0.
