Metadata-Version: 2.4
Name: retrieval-eval-gate
Version: 0.1.0
Summary: Retrieval evaluation that reports what it cannot resolve — intervals, significance, anytime-valid CI gates, and distribution-free guarantees. Zero dependencies, any retriever.
Author-email: Jinwoo Kwon <kownjinwoo@gmail.com>
License: MIT License
        
        Copyright (c) 2026 jinwovo
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/jinwovo/rag-eval-gate
Project-URL: Source, https://github.com/jinwovo/rag-eval-gate
Project-URL: Issues, https://github.com/jinwovo/rag-eval-gate/issues
Keywords: rag,retrieval,evaluation,information-retrieval,benchmark,conformal-prediction,anytime-valid,confidence-sequence,ci-gate,beir
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: scipy>=1.10; extra == "dev"
Dynamic: license-file

# rag-eval-gate

**Retrieval evaluation that tells you what it can't resolve.**

[![ci](https://github.com/jinwovo/rag-eval-gate/actions/workflows/ci.yml/badge.svg)](https://github.com/jinwovo/rag-eval-gate/actions/workflows/ci.yml)
[![python](https://img.shields.io/badge/python-3.10%20%E2%80%93%203.13-blue)](https://github.com/jinwovo/rag-eval-gate/actions/workflows/ci.yml)
[![dependencies](https://img.shields.io/badge/dependencies-none-brightgreen)](#why-zero-dependencies)
[![license](https://img.shields.io/badge/license-MIT-lightgrey)](LICENSE)

> `Recall@5 = 1.00` on 10 queries is really **`[0.69, 1.00]`**.
> That `+0.30` lift over BM25 has a p-value floor of **0.25** — no outcome of that
> comparison could ever have been significant.
> Both of those are computable **from the published table alone.**

## → [Check a number right now, nothing to install](https://jinwovo.github.io/rag-eval-gate/)

Type in the figure and the query count. It returns the interval that figure supports and,
for a claimed improvement, whether *any* outcome of that comparison could have reached
significance. The arithmetic runs in the page; nothing is sent anywhere.

Or from a terminal:

```bash
pip install retrieval-eval-gate      # the PyPI name; `rag-eval-gate` there is someone else's

rag-eval-gate audit -n 10 -v 1.00 -b 0.70     # audit a table you are reading
rag-eval-gate power gold.jsonl                # what your own gold set can resolve
```

Zero dependencies. Works with any retriever in any language. Runs as a GitHub Action.
The command is `rag-eval-gate`; the distribution is `retrieval-eval-gate` because the shorter
name on PyPI belongs to an unrelated project. Both spellings work as the command.

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="docs/anytime-dark.png">
  <img src="docs/anytime-light.png" alt="Left: a fixed-N 95% confidence interval, checked after every query, misses the true score in 17% of runs at 50 queries and 33% at 300, against 0.5-1.7% for an anytime-valid confidence sequence. Right: a sequential gate spends 31 of its 300-query budget on a clearly broken system and 54 on a clearly good one, but the full budget on one sitting exactly at the threshold.">
</picture>

---

## Three things your eval is probably not telling you

**A perfect score is not a perfect score.** Ten queries, one relevant document each,
`Recall@5 = 1.00` — that is ten successes out of ten, and the exact binomial interval
bottoms out at `(α/2)^(1/n) = 0.69`. The headline is 31 points wide.

**Some comparisons cannot be won.** Your hybrid retriever beats BM25 by `+0.30 Recall@5`.
On ten queries that is three queries flipping, seven tying — and a paired randomization
test on three differing pairs has a smallest attainable two-sided p of `2/2³ = 0.25`. Not
"not significant": *unable* to be, whatever the result.

**Your interval breaks when you look at it.** A 95% interval promises that **one** look at
a **finished** sample contains the truth 95% of the time. Watching a CI log is not that.
Over 800 simulated evaluation streams, inspection starting only at n = 30 where the normal
approximation is defensible:

| metric | queries | fixed-N interval | confidence sequence |
|---|:---:|:---:|:---:|
| Recall@5 (0/1) | 300 | **34.0%** | 1.9% |
| reciprocal rank (discrete, skewed) | 300 | **32.8%** | 2.6% |
| nDCG@10 (graded) | 300 | **35.1%** | 1.6% |
| groundedness judge (skewed high) | 300 | **30.0%** | 0.5% |

*Runs where the true mean escapes a nominal 95% interval at some point while it is watched.*
The fixed-N column grows with stream length — the longer you look, the more chances to be
wrong. That is the law of the iterated logarithm, and it is why the right-hand column exists.

## Audit a table in five seconds

No gold set, no retriever, no data — the reported means and the query count are enough:

```console
$ rag-eval-gate audit -n 10 -m recall@5 -v 1.00 -b 0.70

recall@5 = 1.000 on 10 queries
  that is 10/10 successes

  95% interval   [0.692, 1.000]   width 0.308   (exact binomial)
  The reported figure is 31 points wide. Quoting it to two decimal places implies a
  precision 10 queries cannot deliver.

Claimed lift +0.300 over 0.700
  note: the two means determine it exactly: 3 of 10 queries differ

  smallest attainable p   0.2500

  UNRESOLVABLE. Under every possible overlap of the two systems' successes,
  a paired test on this many differing queries has a floor above 0.05, so no
  outcome of that comparison could have reached significance. This is a fact
  about the sample size, not about the systems.
```

It exits non-zero when the comparison could not have been significant, so it works as a
check and not only as something to read. The interval is exact for a 0/1 metric; for a
graded one the per-query spread is unknown, so it reports the widest interval the mean
allows — a `[0,1]` variable with mean *m* has variance at most *m*(1−*m*).

The same thing runs in a browser at
[jinwovo.github.io/rag-eval-gate](https://jinwovo.github.io/rag-eval-gate/), with a
shareable link for the result.

## Quickstart

```bash
pip install retrieval-eval-gate

# 1. What can this gold set even measure?  (no retriever needed)
rag-eval-gate power gold.jsonl

# 2. Score it, with intervals and significance
rag-eval-gate eval gold.jsonl --modes bm25,vector,hybrid

# 3. Gate a pull request
rag-eval-gate eval gold.jsonl --gate --min-mrr10 0.85
```

A gold set is JSONL:

```json
{"query": "how do I rotate a signing key", "relevant_doc_ids": ["kms-rotation"]}
```

…or the TREC qrels you already have (`--gold qrels/test.tsv --queries-file queries.jsonl`).

## Works with whatever you built

Three ways in, so the language and shape of your retriever are irrelevant.

**HTTP** — a URL template and a small field map. No assumption about the route or schema:

```bash
rag-eval-gate eval gold.jsonl \
  --url-template 'https://search.internal/v2/query?text={query}&strategy={mode}' \
  --results-path 'data.hits' --id-field 'document_id' --score-field 'relevance'
```

**A command** — anything that prints JSON to stdout. Go, Rust, a notebook, a shell script:

```bash
rag-eval-gate eval gold.jsonl \
  --backend command --command './retrieve --q {query} --mode {mode} --json'
```

**A TREC run file** — rank the corpus once, offline, on whatever hardware; evaluate as often
as you like with nothing running:

```bash
rag-eval-gate eval qrels/test.tsv --backend run-file --run-file runs/bm25.trec
```

## Four gate policies

```bash
rag-eval-gate eval gold.jsonl --gate --gate-policy sequential
```

| policy | fails when | use it when |
|---|---|---|
| `point` *(default)* | the mean falls below the threshold | always — the absolute floor |
| `ci-lower` | the 95% lower bound falls below it | your gold set is big enough that the interval is narrower than the safety margin |
| `regression` | *also* on a significant paired drop against a recorded run | you have a green baseline to compare against |
| `sequential` | the anytime-valid verdict is `fail`, or the budget runs out `undecided` | queries cost money or minutes |

`regression` is the sensitive one. An absolute threshold only notices a regression once the
mean crosses a line someone guessed; a paired test compares the **same queries** before and
after, so a real drop on two of them is caught while the mean still clears the line.

`sequential` is the cheap one. Because a confidence sequence is valid at every sample size,
stopping as soon as the verdict is settled is a decision rather than a peek:

| the system's true Recall@5 | verdict | queries scored (of 300) | saved |
|:---:|:---:|:---:|:---:|
| 0.98 | pass | 54 | **82%** |
| 0.95 | pass | 89 | **70%** |
| 0.85 (exactly at the line) | `undecided` | 293 | 2% |
| 0.60 | fail | 31 | **90%** |

`undecided` fails. The budget ran out before the evidence arrived, which is a fact about the
gold set rather than a clean bill of health — and a gate that treats "we could not tell" as
a pass has quietly stopped gating.

## As a GitHub Action

```yaml
- uses: jinwovo/rag-eval-gate@v1
  with:
    gold-file: eval/gold.jsonl
    url-template: 'http://localhost:8080/api/search?q={query}&mode={mode}'
    gate-policy: sequential
    min-mrr10: "0.85"
```

The step summary gets the interval table, the significance verdicts, what the gold set can
resolve, and a ✅/❌.

## What comes out

```
queries=300  depth=10  intervals=95%

mode                      recall@5             recall@10                mrr@10               ndcg@10
--------------------------------------------------------------------------------------------------
bm25              0.700 [0.35,0.93]    0.800 [0.44,0.97]    0.670 [0.35,0.90]    0.702 [0.38,0.90]
hybrid            1.000 [0.69,1.00]    1.000 [0.69,1.00]    0.950 [0.70,1.00]    0.963 [0.78,1.00]

paired randomization test vs bm25 (Holm-corrected within each metric)
  recall@5   hybrid    delta=+0.300 p=0.2500 holm=0.2500  unresolvable (3/10 queries differ; floor p = 0.250)
  mrr@10     hybrid    delta=+0.280 p=0.1250 holm=0.2500  unresolvable (4/10 queries differ; floor p = 0.125)

resolution of this gold set (hybrid vs bm25, mrr@10):
  n = 10 queries, per-query difference sd = 0.388
  smallest detectable effect at 80% power: +0.344
  queries needed to resolve +0.020: 2,957
```

`unresolvable` is the word this tool exists to print. It is not a softer way of saying "not
significant" — it means the queries that differ are too few for *any* outcome to reach
p < 0.05, which is a property of your labels and no amount of rerunning will change it.

## The statistics, and why each one

| | what | why not the obvious thing |
|---|---|---|
| **Exact binomial** (Clopper–Pearson) | intervals for Recall@k when each query has one relevant document | a bootstrap resamples a constant at 10/10 and returns `[1.00, 1.00]`, which reads as certainty |
| **BCa bootstrap** | intervals for graded metrics | percentile intervals under-cover on skewed samples, and reciprocal rank is very skewed |
| **Paired randomization test** <br><sub>Smucker, Allan & Carterette, CIKM 2007</sub> | mode comparisons | reciprocal rank is discrete, bounded and skewed; a t-test at n = 10 is not measuring what it claims |
| **Holm–Bonferroni** | correction across a mode sweep | five challengers against one baseline is five chances at a false win; uncorrected 0.05 is about 0.23 |
| **Betting confidence sequences** <br><sub>Waudby-Smith & Ramdas, JRSS-B 2024</sub> | the sequential gate | alpha spending needs the looks planned in advance; a CI log is inspected continuously |
| **Conformal prediction, adaptive sets** <br><sub>Romano, Sesia & Candès, NeurIPS 2020</sub> | `calibrate`: how many passages to send an LLM | a fixed top-K over-spends on easy queries and truncates the hard ones |
| **Risk-controlling prediction sets** <br><sub>Bates et al., JACM 2021</sub> | `calibrate`: when to abstain | a hand-picked threshold implies a claim about hallucination rate that was never checked |
| **Prediction-powered inference** <br><sub>Angelopoulos et al., Science 2023</sub> | `ppi_mean`: a valid interval when only a biased LLM judge scored your data | averaging the judge gives a tight interval around the judge's belief, which covers the truth ~0% of the time |

Every one is seeded, so a gate cannot flap because a resample drew differently.

## Certified thresholds

```bash
rag-eval-gate calibrate gold.jsonl --alpha 0.10 --risk-alpha 0.05
```

Turns the two constants that decide what a RAG system costs and how often it can make things
up into thresholds with finite-sample, distribution-free guarantees:

```
coverage   P(the passages sent to the LLM contain a relevant document)  >= 1 - alpha
risk       P( P(answering from context with no relevant document) <= risk-alpha ) >= 1 - delta
```

Measured on 800 queries against a stand-in cross-encoder: **5.0 passages against a fixed 9
carrying the identical promise — 44% fewer prompt tokens**, because the budget moves to
where it is needed (3.2 passages when the answer ranked first, 8.4 when it ranked fourth or
lower). Three splits — one picks the softmax temperature, one fits the quantile, one scores
both having informed neither — because choosing the temperature on the data the quantile is
fitted on breaks exchangeability, and the resulting under-coverage is invisible in every
in-sample number.

A certificate that abstains on everything is valid, useless, and labelled **degenerate**. A
threshold search that stalls names the calibration size that would settle it.

## When only an LLM has judged your data

Your groundedness score is a judge model's opinion of the system it belongs to. Your
relevance labels, at any real scale, were written by a model too. Both are cheap and neither
is trustworthy on its own, and averaging thousands of them does not produce a 95% interval —
it produces a very narrow interval around whatever the model believes.

`ppi_mean` implements **prediction-powered inference** ([Angelopoulos et al., *Science*
2023](https://www.science.org/doi/10.1126/science.adi6000), power-tuned as PPI++): hand-label
a small sample, let the model predict everything, and subtract the bias measured on the
labelled part.

```python
from rag_eval_gate import ppi_mean

estimate = ppi_mean(hand_labels, judge_on_those, judge_on_everything_else)
print(estimate.statement())
# The judge reports 0.886. Measured against 50 hand labels it is optimistic by 0.081,
# so the true value is 0.805 [0.781, 0.829] — an interval worth about 474 hand labels,
# from 50.
```

Over 1,500 repetitions of the whole label-and-estimate cycle, 50 hand labels against 2,050
judge-scored items:

| judge behaviour | coverage: judge only | hand labels only | **PPI** | interval width | effective labels *(from 50)* |
|---|:---:|:---:|:---:|:---:|:---:|
| flattering, accurate | **0.0%** | 94.0% | **94.3%** | −67% | **474** |
| flattering, noisy | **0.0%** | 94.2% | **93.5%** | −22% | 84 |
| unbiased, noisy | 14.5% | 94.7% | **93.7%** | −22% | 85 |
| uninformative | **0.0%** | 94.8% | **94.3%** | −1% | 51 |

**Validity never depends on the judge being good** — the bias is measured, not assumed. And
**a useless judge costs nothing**: λ is tuned to minimise variance, so noise gets λ ≈ 0 and
the estimator falls back to the hand-label mean. That last row is the safety property, not a
failure.

## Why zero dependencies

This runs inside a CI job that should not need a resolver, a wheel build or a lockfile to
gate a pull request. The machinery it needs — a regularized incomplete beta, a BCa
bootstrap, a martingale — is a few hundred readable lines, and `pip install retrieval-eval-gate`
installs exactly one thing. The test suite cross-checks the hand-rolled special functions
against SciPy, and skips those checks when SciPy is not installed.

## Tests

```bash
pip install -e ".[dev]" && pytest
```

200 tests. Against closed forms (`(α/2)^(1/n)` at `k = n`, `2^(1-k)` for a uniform improvement, R's
`p.adjust` worked example, the `(n+1)` conformal quantile correction), against an
independent brute-force permutation reference written inside the test, and — for the
guarantees — by simulating the whole calibrate-then-deploy cycle and counting how often each
promise actually breaks.

## Provenance

Extracted from [jinwovo/recall](https://github.com/jinwovo/recall), a hybrid-search and
grounded-RAG system, after its own eval harness was pointed at its own README and three of
the headline claims failed. The design notes live there as ADRs 0011–0014.

## License

MIT.
