Metadata-Version: 2.4
Name: chunklab
Version: 0.2.0
Summary: Find out which chunking strategy actually retrieves your answers best - on your own documents - in 60 seconds.
Project-URL: Homepage, https://github.com/Ludovico-Chieffallo/chunklab
Project-URL: Repository, https://github.com/Ludovico-Chieffallo/chunklab
Project-URL: Issues, https://github.com/Ludovico-Chieffallo/chunklab/issues
Project-URL: Changelog, https://github.com/Ludovico-Chieffallo/chunklab/blob/main/CHANGELOG.md
Author: Ludovico Chieffallo
License-Expression: MIT
License-File: LICENSE
Keywords: chunking,embeddings,evaluation,rag,retrieval
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Requires-Dist: jinja2>=3.1
Requires-Dist: langchain-text-splitters>=0.2
Requires-Dist: numpy>=1.26
Requires-Dist: pydantic>=2.5
Requires-Dist: pymupdf4llm>=0.0.17
Requires-Dist: python-docx>=1.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: rapidfuzz>=3.0
Requires-Dist: rich>=13.0
Requires-Dist: sentence-transformers>=3.0
Requires-Dist: tiktoken>=0.7
Requires-Dist: typer>=0.12
Provides-Extra: demo
Requires-Dist: gradio>=4.0; extra == 'demo'
Provides-Extra: dev
Requires-Dist: pre-commit>=3.5; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# chunklab

> **Find out which chunking strategy actually retrieves your answers best — on your own documents — in 60 seconds.**

[![CI](https://github.com/Ludovico-Chieffallo/chunklab/actions/workflows/ci.yml/badge.svg)](https://github.com/Ludovico-Chieffallo/chunklab/actions)
![Python](https://img.shields.io/badge/python-3.11%2B-blue)
![License](https://img.shields.io/badge/license-MIT-green)

When a RAG system gives a wrong answer, the failure is often in **retrieval** — the passage that holds the answer never reaches the model — not in the LLM's reasoning over what it was given. And one of the most overlooked variables in retrieval is **chunking**: how you split documents before embedding them. Independent benchmarks that hold the embedding model and retriever fixed and vary *only* the chunking find that it moves retrieval quality on the same corpus (see, e.g., Chroma's [Evaluating Chunking Strategies for Retrieval](https://www.trychroma.com/research/evaluating-chunking)).

On the example corpus bundled with this repo, the choice of strategy moves per-document recall@5 by up to **28 points** (API-reference document: 0.84 best vs 0.56 worst strategy), and **three different strategies win depending on the document** — reproduce both claims with `chunklab run --docs examples/corpus --questions examples/questions.yaml` (enforced in CI by `tests/test_corpus_discriminates.py`).

There is no universal best strategy: which one wins depends on **your** documents and **your** questions. `chunklab` makes that a one-command, fully-local, evidence-based decision instead of a guess.

## What it does

You give it your documents and a handful of questions (each tagged with the "gold" passage that answers it). It runs several chunking strategies, indexes and retrieves for each, and tells you **which strategy actually retrieves your answer-bearing text best — and why**.

```
docs + questions ─▶ [fixed · recursive · semantic · structure] ─▶ ranked report + diagnostics
```

- **Runs fully locally.** Default embeddings are a small local model (`BAAI/bge-small-en-v1.5`) — no API key, your documents never leave your machine.
- **Explains itself.** Per-strategy diagnostics (token distribution, % tiny fragments, boundary health, table integrity) tell you *why* a strategy won or lost.
- **Three outputs:** a console table, a standalone HTML report (per-question drill-down + chunk-boundary visualization), and a machine-readable JSON report for CI.

## Install

```bash
pip install chunklab
```

The first run downloads the embedding model (~130 MB) once and caches it.

## 60-second quickstart

```bash
# clone the repo to get the example corpus, or point --docs at your own files
chunklab run --docs examples/corpus --questions examples/questions.yaml
```

Console output of exactly that command (regenerated from a real run, never hand-edited):

<!-- BEGIN GENERATED EXAMPLE (scripts/regen_readme_example.py) -->
```
ChunkLab — 5 document(s), 129 scored questions, top_k=5, model=BAAI/bge-small-en-v1.5

 Strategy            ┃ recall@5 ┃ hit@5 ┃  MRR ┃ #chunks ┃ med_tok ┃ %tiny ┃ boundary 
━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━
 ▶ structure         │     0.81 │  0.81 │ 0.63 │      71 │     346 │   25% │     100% 
   fixed             │     0.79 │  0.79 │ 0.61 │      61 │     512 │    2% │       2% 
   recursive         │     0.78 │  0.78 │ 0.59 │      62 │     458 │    5% │     100% 
   semantic          │     0.72 │  0.72 │ 0.55 │      47 │     592 │    2% │     100% 
   semantic_no_floor │     0.70 │  0.70 │ 0.53 │      59 │     470 │   24% │     100% 

Recommendation:
  Use STRUCTURE chunking (max_tokens=800). It gave the best retrieval on your corpus (recall@5 = 
0.81). Note: 27% of structure chunks exceed the embedding model's max sequence length and may be 
truncated at embed time.
```
<!-- END GENERATED EXAMPLE -->

Open `chunklab_report/report.html` for the full drill-down and the chunk-boundary visualization. The example corpus itself is documented in [`examples/CORPUS.md`](examples/CORPUS.md) — five documents designed so that different strategies win on different documents.

## Writing your `questions.yaml`

The tool scores retrieval offline by checking whether a **gold snippet** — a verbatim (or near-verbatim) passage from your document that answers the question — lands inside a retrieved chunk. Aim for 10–20 questions.

```yaml
questions:
  - id: q1
    query: "What is the termination notice period?"
    gold_snippets:
      - "written notice at least 30 days prior to termination"
    tags: [contracts]
  - id: q2
    query: "How is overtime compensated?"
    gold_snippets:
      - "Overtime is paid at 1.5x the regular hourly rate"
  - id: q3
    query: "What is the dress code?"
    # no gold_snippets -> skipped (with a warning), so scoring stays deterministic
```

Tips:
- Copy the gold snippet **verbatim** from the source so matching is reliable (small drift is absorbed by fuzzy matching, threshold configurable).
- A question with no `gold_snippets` is skipped in the MVP — add the passage to include it.

## Configuration

`chunklab run` works with no config. To customize, pass `--config config.yaml`:

```yaml
embedding:
  backend: local                 # local (default) | openai (coming soon)
  model: BAAI/bge-small-en-v1.5
retrieval:
  mode: dense                    # dense (bm25/hybrid coming soon)
  top_k: 5
eval:
  fuzzy_threshold: 0.90
  ranking_metric: recall_at_k    # recall_at_k | mrr | hit_rate_at_k
  min_floor_tokens: 200
strategies:
  - { name: fixed,     params: { chunk_size: 512, overlap: 64 } }
  - { name: recursive, params: { chunk_size: 512, overlap: 64 } }
  - { name: semantic,  params: { breakpoint_percentile: 95, min_tokens: 200, max_tokens: 1000 } }
  - { name: structure, params: { max_tokens: 800 } }
output:
  formats: [console, html, json]
  dir: ./chunklab_report
```

See [`examples/config.example.yaml`](examples/config.example.yaml) for the full default.

### The chunking strategies

| Strategy | What it does |
|---|---|
| `fixed` | Fixed-size token windows with overlap. |
| `recursive` | Recursive split on paragraph → sentence → word separators. |
| `semantic` | Embedding-based boundaries **with a minimum-size floor** that merges tiny fragments — the fix for the "fragment trap." |
| `semantic_no_floor` | The naive version, included on purpose to show what the floor prevents. |
| `structure` | Heading-aware: one chunk per section, sub-split only when oversized. |

Run `chunklab strategies` to list them.

## Python API

```python
from chunklab import evaluate

report = evaluate(
    docs="./docs",
    questions="./questions.yaml",
    config=None,  # or a path / a Config object
)
print(report.recommendation)
for r in report.strategy_results:      # ranked best-first
    print(r.strategy, r.recall_at_k, r.mrr)
```

The `EvalReport` schema is a versioned public contract (`report.schema_version`, documented field-by-field in [`docs/schema.md`](docs/schema.md)) — serialize it with `report.model_dump_json()` for CI or dashboards.

## Web demo

```bash
pip install "chunklab[demo]"
chunklab demo          # launches a local Gradio app
```

Upload a document, type a few questions with gold snippets, pick strategies, and see the ranked comparison plus the chunk visualization.

## What this is (and isn't)

**It is** a fast, focused, local pre-flight utility that answers exactly one question: *which chunking strategy retrieves best on my corpus, and why?*

**It is not** a production RAG framework, a vector database, a document parser, or a general LLM-eval platform. It deliberately does one thing well.

## Supported inputs

Documents: **PDF, DOCX, TXT, MD**. Corpora of tens to low-hundreds of documents (this is a pre-flight tool, not a batch pipeline).

## License

[MIT](LICENSE) — permissive on purpose. Runs locally, no telemetry, no phone-home.
