Metadata-Version: 2.4
Name: arara-rag
Version: 0.5.0
Summary: Portuguese-first, CPU-only retrieval: tinyzchunk chunking, static embeddings, BM25, rank fusion and CXM25 reranking. numpy only, out-of-core, with metadata filtering and CRUD.
Author: Carlo Moro
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/cnmoro/Arara-RAG
Project-URL: Repository, https://github.com/cnmoro/Arara-RAG
Keywords: rag,retrieval,portuguese,pt-br,embeddings,bm25,cpu,numpy,out-of-core
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Indexing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: tokenizers>=0.20
Requires-Dist: model2vec>=0.9
Requires-Dist: tinyzchunk>=0.3.3
Requires-Dist: cxm25>=0.1.0
Requires-Dist: threadpoolctl>=3.1
Provides-Extra: nanoe5
Requires-Dist: nanoe5>=0.3; extra == "nanoe5"
Provides-Extra: bench
Requires-Dist: datasets>=2.19; extra == "bench"
Requires-Dist: pyarrow; extra == "bench"
Requires-Dist: huggingface_hub>=0.23; extra == "bench"
Requires-Dist: matplotlib>=3.8; extra == "bench"
Provides-Extra: validate
Requires-Dist: pytrec_eval-terrier>=0.5.6; extra == "validate"
Requires-Dist: scikit-learn>=1.3; extra == "validate"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# arara-rag

**Portuguese-first retrieval that runs entirely on CPU.** Chunking, dense and
lexical retrieval, rank fusion and reranking — numpy only. No PyTorch, no ONNX
Runtime, no FAISS. The whole install is **200 MB**.

```bash
pip install arara-rag
```

```python
from arara_rag import Arara

arara = Arara(path="./indice")            # out-of-core and persistent
arara.add_documents(
    {"lei_1234": open("lei.txt").read()},
    metadata={"ano": 2024, "tipo": "lei", "uf": "BR"},
)
hits = arara.search("qual a alíquota?", top_k=5, where={"ano": {"$gte": 2020}})
print(hits[0].doc_id, arara.resolve(hits[0]))     # exact source span
```

## What's in it

| Stage | Component | Size |
|---|---|---|
| Chunking | [`tinyzchunk`](https://github.com/cnmoro/tinyzchunk) — tokenizer-free, distilled from an LLM teacher | 2.1 MB |
| Dense | [`static-nomic-384-pten-v2`](https://huggingface.co/cnmoro/static-nomic-384-pten-v2) — Model2Vec static embeddings | 62 MB |
| Lexical | BM25 over a numpy inverted index | — |
| Rerank | [`CXM25`](https://github.com/cnmoro/CXM25) — PT-BR lexical scoring | bundled |
| Fusion | Reciprocal Rank Fusion | — |

An index can be **out-of-core**: vectors live in a memory-mapped file and
documents, metadata and offsets in SQLite, so the index is about **1 KB per
document** and cold pages can be evicted by the OS instead of being pinned on
the heap. Same API either way — `Arara()` keeps everything in memory.

## Speed and memory

One CPU core, no GPU. Measured end to end with `python -m bench.profile`.

| documents | index build | query p50 | query p95 | index size | peak RSS to serve |
|---|---|---|---|---|---|
| 1,000 | 3.1 s | **0.45 ms** | 0.54 ms | 1.8 MB | 472 MB |
| 10,000 | 8.8 s | **0.86 ms** | 6.8 ms | 18 MB | 481 MB |
| 50,000 | 33.8 s | **7.0 ms** | 8.0 ms | 91 MB | 553 MB |

- **~1,300–1,500 documents/second** to chunk, embed, tokenise and index —
  chunking and embedding are per-document, so they run across processes.
  A single process manages ~300/second.
- **~1.8 KB per document** of index: 1.5 KB of vectors plus BM25 postings.
- Query latency scales with corpus size because both retrievers score the whole
  corpus per query — that is what makes the ranking exact rather than
  approximate.
- Peak RSS is dominated by a **~470 MB fixed cost** (Python, numpy, the
  embedding model and the tokenizer tables); the corpus adds ~1.7 KB per
  document on top. Vectors are memory-mapped, so cold pages can be evicted.

![Speed and memory](docs/scaling.png)

## Retrieval quality

MTEB-BR, the Brazilian Portuguese benchmark with a
[public leaderboard](https://huggingface.co/spaces/MTEB-BR/leaderboard).
nDCG@10, fixed-window chunking. Metrics are computed by `bench/metrics.py`,
which `bench/validate_metrics.py` checks against `pytrec_eval` to **0.0e+00**,
and against scikit-learn on binary relevance.

| Task | docs | rel./query | MRR@10 | dense | lexical | hybrid |
|---|---|---|---|---|---|---|
| BRTaxQAR (capped) | 478 | 2.92 | 0.516 | 0.2934 | **0.4051** | 0.3486 |
| FaQuADIR | 244 | 1.0 | 0.873 | 0.7139 | **0.8961** | 0.8304 |
| FaqBacenRetrieval | 1,673 | 1.0 | 0.433 | 0.3744 | **0.4881** | 0.4526 |
| JurisTCU | 16,045 | 15.0 | 0.821 | 0.3887 | **0.5378** | 0.4890 |
| Quati | 50,000 | 38.66 | 0.676 | 0.3268 | **0.4067** | 0.4046 |

**Read `rel./query` before comparing across rows.** nDCG@10 measures very
different things on these tasks, and that is a property of the benchmarks rather
than of the retrieval:

- **FaQuADIR and FaqBacen have exactly one relevant document per query.** There
  nDCG@10 is a transform of the rank of that one document, so 0.90 means it is
  usually first and 1.0 is the ceiling.
- **Quati has 38.66 relevant documents per query.** With ten slots, a perfect
  score requires all ten to be relevant, so 0.41 means roughly four of the top
  ten are relevant — much closer to precision at 10 than the FaQuADIR number is.
- JurisTCU (15.0) and BRTaxQAR (2.92) sit in between.

A retriever scoring 0.33 on Quati is therefore not "worse" than one scoring 0.71
on FaQuADIR — the numbers are not comparable across rows, only within one.

### Choosing a dense encoder

Two backends ship. The default is a Model2Vec lookup table; [nanoE5.c](https://github.com/cnmoro/nanoe5.c)
is an opt-in alternative — a 4-bit `multilingual-e5-small` in C, with no PyTorch,
no ONNX and no BLAS. It is a real transformer forward pass, so it is slower, and
it is a separate dependency:

```bash
pip install "arara-rag[nanoe5]"
```
```python
Arara(dense_backend="nanoe5")                              # or "static", the default
Arara(dense_backend="nanoe5", dense_variant="original")    # English-first build
```

It is meaningfully better, and it changes the shape of the results:

| Task | static dense | nanoE5 dense | static hybrid | **nanoE5 hybrid** | lexical |
|---|---|---|---|---|---|
| BRTaxQAR (capped) | 0.2934 | 0.3423 | 0.3486 | **0.4180** | 0.4051 |
| FaQuADIR | 0.7139 | 0.8314 | 0.8304 | 0.8906 | 0.8961 |
| FaqBacenRetrieval | 0.3744 | 0.5858 | 0.4526 | **0.5659** | 0.4881 |
| JurisTCU | 0.3887 | 0.4906 | 0.4890 | **0.5685** | 0.5378 |

Two things follow. **Dense retrieval improves by 0.05–0.21 nDCG@10** — on
FaqBacen that is a larger jump than the whole distance from BM25 to the
leaderboard median. And **hybrid finally beats BM25**: with the static model,
lexical alone won every task and the dense half was dead weight; with a real
encoder, fusion wins three of four and ties FaQuADIR. That is the argument for
hybrid retrieval, and it needed a dense model that carries its weight.

The cost is indexing — nanoE5 windows anything past 512 tokens, so the penalty
tracks chunk length: ~4× for short chunks, ~30× for 16k of them, and far more
for a corpus of 2.5 kB chunks. Query latency is ~8 ms against 0.5–7 ms, since a
query is one short sequence either way.

CXM25 reranking on top of the hybrid adds **+0.018 to +0.077 nDCG@10** across
these tasks for 1–6 ms per query (FaQuADIR: 0.8304 → **0.9078**,
BRTaxQAR full documents: 0.4801 → **0.5091**).

## Why chunking matters most

Legal documents in BR-TaxQA-R average 32,000 characters and reach 1.17M. MTEB-BR
truncates them at 32k because transformer encoders cannot fit more. arara
chunks, so it indexes the whole statute.

| Configuration | chunks | nDCG@10 | R@100 |
|---|---|---|---|
| capped at 32k, one vector per doc *(the leaderboard's setting)* | 478 | 0.1496 | 0.4351 |
| capped at 32k, fixed windows | 2,552 | 0.2934 | 0.6300 |
| capped at 32k, **tinyzchunk** | 23,319 | 0.3088 | 0.6225 |
| **full documents**, fixed windows | 6,439 | 0.4041 | 0.7497 |
| **full documents**, paragraph splits | 6,439 | 0.4041 | 0.7497 |
| **full documents**, tinyzchunk | 60,927 | 0.4287 | 0.7486 |
| **full documents**, tinyzchunk + BM25 | 60,927 | 0.4801 | 0.8496 |
| **full documents**, + CXM25 rerank | 60,927 | **0.5091** | 0.8102 |

![Chunking a legal corpus beats truncating it by 3.4×](docs/ablation.png)

## Against the leaderboard

**On FaQuADIR, arara's best configuration outranks all 96 models on the board**
— above `voyage-context-4`, `gemini-embedding-2` and `Qwen3-Embedding-8B` — on
one CPU core. On BR-TaxQA-R it beats 90 of 95.

The honest caveat: the leaderboard evaluates *embedding* models, and there is no
BM25 entry on it. arara's strongest modes are lexical, and lexical retrieval is
simply very good on short, high-overlap PT-BR documents — part of that gap is a
missing baseline on their side, not a transformer-killing dense model on ours.

![arara against the MTEB-BR leaderboard](docs/leaderboard.png)

## Reranking

MTEB-BR reranking hands you a fixed candidate list and scores only the order
(MAP@1000), so `identity` is the baseline the benchmark ships with.

| Task | identity | dense | lexical | hybrid | **CXM25** |
|---|---|---|---|---|---|
| QuatiReranking | 0.2839 | 0.2798 | 0.2939 | 0.3066 | **0.3100** |
| JurisTCUReranking | 0.4150 | 0.3609 | 0.4279 | 0.4129 | **0.4845** |

## Out-of-core, metadata, CRUD

An index is read far more than it is written, so deletes are tombstones and
freed slots are recycled on the next write.

```python
arara = Arara(path="./indice", max_chunk_chars=2000)

arara.add_documents(docs, metadata={"ano": 2024})        # insert / replace
arara.update_metadata("lei_1234", {"revisado": True})    # no re-embedding
arara.delete_document("lei_1234")                        # tombstone + slot reuse
arara.get_document("lei_1234")                           # (text, metadata)
arara.compact()                                          # reclaim file space

arara.search(q, where={"tipo": {"$in": ["lei", "decreto"]}, "ano": {"$gte": 2020}})
arara.search(q, where={"$or": [{"uf": "SP"}, {"uf": "RJ"}]})
```

Supported per field: `$eq` (bare value), `$ne`, `$gt`, `$gte`, `$lt`, `$lte`,
`$in`, `$nin`, `$exists`, `$contains`, `$startswith`, `$endswith`; plus
top-level `$and` / `$or`. Field names are validated and values are bound as SQL
parameters, so a filter cannot inject SQL.

## Guarantees

Enforced by 88 tests, not asserted in prose:

- every chunk is an **exact substring** of the canonical document, ordered and
  non-overlapping, with only whitespace between chunks — nothing is dropped;
- **no chunk exceeds `max_chunk_chars`**, including on a 24,000-character line;
- CRLF and LF inputs chunk **identically** and offsets still resolve;
- the in-memory and on-disk paths return **identical rankings**;
- importing the package never imports `torch` or `onnxruntime`.

## Reproduce

```bash
python -m venv .venv && .venv/bin/pip install -e ".[bench,validate,dev]"
python -m pytest tests/                 # 88 tests
python bench/validate_metrics.py        # metrics vs pytrec_eval
./bench/run_all.sh                      # every suite -> bench/results/
python -m bench.profile                 # speed and memory -> docs/scaling.png
python -m bench.charts                  # regenerate the figures
python -m bench.leaderboard             # compare against MTEB-BR
```

## Layout

```
arara_rag/
  chunk.py      chunking and the losslessness contract
  dense.py      static encoder
  lexical.py    BM25 inverted index + CXM25 reranker
  store.py      memory-mapped vectors, SQLite catalog, filters
  pipeline.py   Arara: add / search / rerank / CRUD
bench/          task loaders, metrics, suites, profiling, charts
tests/          88 contract and correctness tests
space/          Gradio demo
```

## Limitations

- **PT-BR and English.** The tokenizer, stemmer and stopwords are Portuguese.
- **The dense model is small** and static; lexical retrieval carries the stack
  on short, high-overlap documents.
- **Per-chunk bookkeeping stays resident** (16 bytes/chunk); vectors, text and
  metadata do not.
- **CXM25 reranking is ~71 µs/document**, so it runs over a candidate set.
- **Index build forks worker processes** to parallelise chunking and embedding.
  Set `workers=1` where forking is unsafe or unavailable; the result is
  byte-identical, only slower.

## License

Apache-2.0.
