Metadata-Version: 2.5
Name: parallax-bench
Version: 0.0.2
Summary: Measuring language-induced retrieval displacement in RAG systems using parallel corpora
Project-URL: Homepage, https://github.com/ondrejbruha/parallax-bench
Project-URL: Repository, https://github.com/ondrejbruha/parallax-bench
Author-email: Ondřej Brůha <ondrej.bruha@alphacodes.eu>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: benchmark,czech,evaluation,multilingual,rag,retrieval
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: httpx[http2]>=0.27
Requires-Dist: pandas>=2.2
Requires-Dist: pydantic>=2.7
Requires-Dist: scipy>=1.13
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: tenacity>=8.3
Requires-Dist: typer>=0.12
Provides-Extra: baseline
Requires-Dist: sentence-transformers; extra == 'baseline'
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytrec-eval-terrier; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: web
Requires-Dist: fastapi; extra == 'web'
Requires-Dist: uvicorn; extra == 'web'
Description-Content-Type: text/markdown

# parallax-bench

*Measuring language-induced retrieval displacement in RAG systems using
parallel corpora.*

Parallax is the effect where **the same object appears at a different position
when viewed from a different vantage point**. This benchmark measures that
**the same document lands at a different rank when you ask about it in a
different language** — with everything else (content, models, configuration,
hardware) held constant.

The PARALLAX benchmark uses parallel EU legislative corpora: the same document
exists in many languages under a stable CELEX identifier. Ingesting each
language into its own index and running every query variant against every
index yields a fully controlled X×X matrix:

- **diagonal** — monolingual RAG (query language = document language)
- **off-diagonal** — cross-lingual retrieval
- **all indexes at once** — MultiRAG

Because ground truth is at *document* level and the document keeps the same ID
across languages, **a single TREC-format qrels file is valid for every cell of
the matrix** — monolingual and cross-lingual runs are scored by identical code
over identical judgements.

## Quickstart (two minutes, offline)

```bash
pip install parallax-bench
parallax-bench run --system baseline-local --subset smoke
parallax-bench score
parallax-bench report
```

No GPU, no API key, no server, no Postgres — the `smoke` subset ships with
document texts and the local baseline (BM25, optionally + dense via
`pip install 'parallax-bench[baseline]'`) runs anywhere. CI runs exactly these
commands on every PR.

## Benchmarking a real system

```bash
pip install parallax-bench
parallax-bench fetch    --data-version v1        # download ONCE into corpus-store/, verify sha256_text
parallax-bench ingest   --system my-system       # reads only the frozen snapshot — never the network
parallax-bench run      --system my-system --phase retrieval
parallax-bench run      --system my-system --phase generation
parallax-bench score    --run <run_id>
parallax-bench report   --run <run_id>
parallax-bench verify   --data-version v1        # measure source drift; fixes nothing
```

Systems are defined in `systems.toml` (see `systems.example.toml`) as an
adapter class plus configuration. Ablations are expressed as two configured
instances of the same adapter — never as extra protocol parameters. The
adapter protocol has exactly four methods (`describe`, `index`, `search`,
`generate`); see [docs/adding-a-system.md](docs/adding-a-system.md).

`run` is resumable: the task queue lives in a database (SQLite by default,
any SQLAlchemy URL — e.g. Heroku Postgres — via `--db` or `$DATABASE_URL`).
Interrupt at any point; `run --resume <id>` picks up where it stopped.
`score` is separate from `run`, so metrics can be recomputed under different
definitions without repeating a forty-hour collection.

## Results & reproducibility

Aggregated metrics for our runs live in [`runs/`](runs/) as small, diffable
CSVs; raw rankings are attached to GitHub releases. Every run directory
carries full provenance: `config.json` (resolved configuration, secrets
visibly redacted), `system.json` (read from the running system, never
assumed), and `manifest.lock` (the binding `sha256_text` of every document
that went into the index).

Results are **verifiable**: every published number carries the full
provenance needed to check it (`runs/<id>/`). The corpus snapshot used for
these runs is **not yet publicly archived**, so third-party re-execution on
the identical documents is not possible at this time. The manifest records
`sha256_text` for every document, so an archive published later can be
verified against the numbers already reported. Rebuilding the corpus from
EUR-Lex is best-effort; `parallax-bench verify` measures source drift
rather than assuming its absence.

To add your own system's results, open a PR adding a `runs/<id>/` directory —
CI validates the schema. There is no submission leaderboard by design.

## Dataset

`benchmark/v1/` contains queries, TREC qrels and a corpus **manifest** —
`(celex, lang) → URL + sha256`. Document texts are **not redistributed**
(licensing, size, and trust: it is immediately visible that nobody touched
the documents). `parallax-bench fetch` downloads them from EUR-Lex and
verifies checksums; if a source document changes, the run fails loudly.

The single exception is `benchmark/smoke/` (10 documents × 3 languages,
including texts) which exists so the quickstart works offline.

## Licensing

Dual-licensed on purpose: **code** (`src/`, `tests/`, `benchmark/build/`)
under [Apache-2.0](LICENSE); **queries + qrels** (`benchmark/*/`) under
[CC BY 4.0](benchmark/LICENSE-DATA). Corpus documents are not redistributed;
EUR-Lex content is reusable under Decision 2011/833/EU with attribution.

## Citation

See [CITATION.cff](CITATION.cff). Code versions follow semver; dataset
versions (`v1`, `v2`, …) are independent, each with its own Zenodo DOI, and a
released dataset version is never modified in place.
