Metadata-Version: 2.4
Name: corpus-quality
Version: 1.0.0
Summary: Document quality profiler for ML pipelines. Score, deduplicate, and validate your corpus before embedding. Zero mandatory dependencies.
Author: Linda Oraegbunam
License: MIT
Project-URL: Homepage, https://github.com/obielin/doc-quality
Keywords: rag,document-quality,data-preparation,nlp,llm,corpus-analysis,deduplication,data-profiling,machine-learning,text-quality
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0.0; extra == "dev"
Requires-Dist: black>=24.0.0; extra == "dev"
Requires-Dist: ruff>=0.4.0; extra == "dev"
Dynamic: license-file

# doc-quality

**Document quality profiler for ML pipelines. Score, deduplicate, and validate your corpus before embedding. Zero mandatory dependencies.**

[![Tests](https://img.shields.io/badge/Tests-74%20passing-brightgreen?style=flat-square)](tests/)
[![PyPI](https://img.shields.io/pypi/v/corpus-quality?style=flat-square)](https://pypi.org/project/corpus-quality/)
[![Dependencies](https://img.shields.io/badge/Dependencies-zero-brightgreen?style=flat-square)](pyproject.toml)
[![Python](https://img.shields.io/badge/Python-3.10%2B-blue?style=flat-square)](pyproject.toml)
[![License](https://img.shields.io/badge/License-MIT-green?style=flat-square)](LICENSE)
[![LinkedIn](https://img.shields.io/badge/-Linda_Oraegbunam-blue?logo=linkedin&style=flat-square)](https://www.linkedin.com/in/linda-oraegbunam/)

## Install

```bash
pip install corpus-quality
```

## The problem

Most RAG failures start at ingestion -- when nobody checked whether the documents were actually
clean before embedding them. Teams spend days tuning chunk sizes and prompt templates when the
real problem is that 20% of their corpus is failed PDF extractions, near-duplicates, or
wall-of-text that will split badly at any chunk boundary.

**doc-quality runs before any of that.**

## Quick start

```python
from doc_quality import CorpusProfiler

profiler = CorpusProfiler(corpus_name="my_rag_corpus")
report = profiler.profile_directory("./documents/")
print(report.summary())
```

```
=== DOC-QUALITY -- CORPUS QUALITY REPORT ===
  Documents:      47
  Pass:           34
  Warn:           9
  Fail:           4
  Pass rate:      72%
  Avg score:      74.2/100
  Duplicates:     6 pairs
  Recommended chunking: recursive
```

## Single document

```python
from doc_quality import DocumentProfiler

profiler = DocumentProfiler()
profile = profiler.profile(text, name="annual_report.pdf")

print(profile.quality_score)      # 83.4
print(profile.quality_level)      # QualityLevel.PASS
print(profile.boilerplate_ratio)  # 0.18
print(profile.chunk_risk_score)   # 0.22
print(profile.issues)
```

## Near-duplicate detection (no embeddings needed)

```python
from doc_quality import find_duplicates

pairs = find_duplicates({"doc_a.txt": text_a, "doc_b.txt": text_b}, threshold=0.85)
for pair in pairs:
    print(pair)
# DuplicatePair('policy_v1.txt' <-> 'policy_v1_copy.txt', EXACT)
```

## Data card for model cards and papers

```python
print(report.data_card.to_markdown())
```

## CLI

```bash
doc-quality profile report.txt
doc-quality corpus ./documents/ --name "my_corpus"
doc-quality corpus ./documents/ --data-card
doc-quality deduplicate ./documents/
doc-quality corpus ./documents/ --json
```

## Quality dimensions scored

| Dimension | Weight | Detects |
|---|---|---|
| Encoding | 25% | Replacement chars, mojibake |
| Text density | 20% | Whitespace-heavy extractions |
| Uniqueness | 20% | Repeated lines, template noise |
| Sentences | 15% | List dumps, failed extractions |
| Boilerplate | 10% | Headers, footers, disclaimers |
| Table integrity | 5% | Malformed table structure |
| Chunk boundary risk | 5% | Wall-of-text, no split points |

## Pipeline position

```
doc-quality (profile) -> chunk-bench (benchmark) -> rag-eval-kit (evaluate)
```

---

**Linda Oraegbunam** | [LinkedIn](https://www.linkedin.com/in/linda-oraegbunam/) | [GitHub](https://github.com/obielin)
