Metadata-Version: 2.4
Name: rag-hygiene
Version: 0.1.1
Summary: Find and automatically repair broken RAG chunks before you embed them. Zero dependencies.
Author: OnProject
License-Expression: MIT
Project-URL: Homepage, https://github.com/OnProject/rag-hygiene
Project-URL: Repository, https://github.com/OnProject/rag-hygiene
Project-URL: Issues, https://github.com/OnProject/rag-hygiene/issues
Keywords: rag,chunking,retrieval,llm,embeddings,vector-database,linter,cleanup
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# rag-hygiene

**Find and repair broken RAG chunks before you embed them.**

Most RAG failures are decided before retrieval ever runs. A chunk that opens in
the middle of a sentence embeds two unrelated claims. A table split from its
header embeds a grid of numbers with no column names. A PDF that broke a word
across lines embeds `docu` and `ment` as one token. None of this is visible until
answers start being subtly wrong months later.

`rag-hygiene` sits between your splitter and your vector database. It diagnoses
nine classes of chunk damage and **repairs seven of them automatically** — no
LLM calls, no embeddings, no eval set, no network, no dependencies.

```bash
pip install rag-hygiene
```

## Quick start

```bash
rag-hygiene scan chunks.json          # what is wrong?
rag-hygiene fix  chunks.json          # repair it → chunks.clean.json
rag-hygiene fix  chunks.json --dry-run -v   # preview the diff first
```

```
2 chunks scanned

mid-sentence-start  Chunk resumes a cut sentence
  [high] c2  opens mid-sentence ('40% year over year, beat'); its embedding mixes two unrelated claims ·fixable

2 finding(s)  1 high  1 low  · 1 auto-fixable
run `rag-hygiene fix` to repair, or add --dry-run to preview
```

## The repair that matters

```
before                                    after
────────────────────────────────────      ────────────────────────────────────
[chunk 7]  …因此公司营收同比增长            [chunk 7]  …因此公司营收同比增长 40%，
[chunk 8]  40%，超出市场预期。                        超出市场预期。
                                          ✓ merged  ✓ heading "Q3 财报" injected
```

Every repair is **lossless or a rejoin of text that was already adjacent**.
Nothing is summarised, paraphrased, or invented — which is what lets it run
unattended in a build step.

## Rules

| Rule | Severity | Auto-fix | What it catches |
|---|---|---|---|
| `empty-chunk` | high | ✅ removed | Whitespace-only chunks; wasted embedding slots |
| `mid-sentence-start` | high | ✅ merged | Chunk resumes a sentence the previous one cut |
| `tiny-chunk` | medium | ✅ merged | Under 20 chars of prose — no meaning to embed |
| `unclosed-fence` | medium | ✅ closed | Code fence opens, never closes — rest of prompt becomes code |
| `orphan-table` | medium | ✅ header restored | Markdown table rows whose header landed in another chunk |
| `near-duplicate` | medium | — | ≥72% word-sequence overlap; dupes split ranking mass |
| `oversized-chunk` | low | — | Over 4000 chars — retrieval dilutes |
| `pdf-noise` | low | ✅ normalised | Hyphenated line breaks, column padding, page gaps |
| `missing-heading` | low | ✅ injected | A heading exists in the document but the chunk can't see it |

`near-duplicate` and `oversized-chunk` are deliberately **not** auto-repaired:
dropping a duplicate or truncating text changes what the retriever can find, and
that is an editorial decision, not a mechanical one. The tool reports them and
stays out of the way.

## Python API

```python
from rag_hygiene import check, repair

chunks = [
    {"id": "a", "text": "Revenue grew by",        "source": "q3.md", "order": 0},
    {"id": "b", "text": "40% year over year.",    "source": "q3.md", "order": 1},
    {"id": "c", "text": "   ",                    "source": "q3.md", "order": 2},
]

report = check(chunks)
report.count("high")          # 2
report.by_rule()              # {'empty-chunk': 1, 'mid-sentence-start': 1, ...}
report.fails("medium")        # True

clean, report = repair(chunks)
clean[0].text                 # 'Revenue grew by 40% year over year.'
clean[0].metadata['merged_ids']   # ['b']  ← provenance is never lost
```

Accepts `Chunk` objects, dicts (`text` / `page_content` / `content`), or bare
strings. Works with LangChain and LlamaIndex document exports unchanged.

## CI gate

The exit code follows the linter convention, so it drops into any pipeline:

```yaml
- run: pip install rag-hygiene
- run: rag-hygiene scan chunks.json --fail-on high
```

| exit | meaning |
|---|---|
| 0 | clean, or the `--fail-on` gate was not tripped |
| 1 | a finding at or above `--fail-on` remains |
| 2 | bad input (missing file, unsupported format, unknown rule) |
| 3 | internal error |

## Formats

Reads `.json`, `.jsonl`, and `.txt` / `.md`. A JSON payload may be a bare list or
an object with a `chunks` / `documents` / `data` / `items` key. Output preserves
the input format.

## Design notes

Three decisions shape the behaviour:

**It joins only definite continuations.** A chunk opening with `40% …`, `, and
then`, or a lowercase word is provably a continuation, so it gets merged. A chunk
opening with `Revenue grew` *might* have lost its predecessor — that is reported
at high severity but **not** auto-merged, because guessing would rewrite evidence.
`rag-hygiene scan --fail-on high` still catches it; you decide.

**Repairs are idempotent.** Running `fix` twice produces the same bytes as running
it once. That is asserted in the test suite (`test_repair_converges`), because a
formatter that keeps drifting is worse than no formatter.

**`missing-heading` converges.** A document that carries no heading anywhere gets
one `info`-level finding, not one per chunk — there is nothing to repair, and
flooding the report would make the gate useless. A document with a heading but
unlabelled chunks gets per-chunk findings that `fix` actually resolves.

## Comparison

`chunklint` (May 2026) performs overlapping static checks and depends on
pydantic, rich, typer, rapidfuzz and pyyaml. `rag-hygiene` differs in three ways:
it **repairs** rather than only reporting, it has **zero dependencies**, and it
ships CJK sentence segmentation (Chinese/Japanese/Korean text has no spaces after
`。`, so the usual whitespace heuristics silently do nothing on it).

## License

MIT

## Changelog

### 0.1.1

- **Fix:** Chinese chunks resuming a clause after a numeral were reported as
  auto-fixable but never repaired. `is_definite_continuation` now recognises CJK
  measure words (`3 个工作日`, `2024 年`), which carry the same "this resumed a
  sentence" signal that capitalisation carries in Latin text.
- **Test:** added `test_fixable_findings_are_actually_fixed`, asserting the
  contract that every finding marked `fixable` is gone after `repair()`.

### 0.1.0

- Initial release: 9 rules, 7 auto-repairing, zero dependencies, CJK segmentation.
