Metadata-Version: 2.5
Name: rubricon-eval
Version: 0.1.3
Summary: The eval reliability platform — every score ships with an error bar and a calibration receipt.
Project-URL: Homepage, https://github.com/sachinn854/rubricon
Project-URL: Documentation, https://github.com/sachinn854/rubricon#readme
Project-URL: Issues, https://github.com/sachinn854/rubricon/issues
Project-URL: Changelog, https://github.com/sachinn854/rubricon/blob/main/ROADMAP.md
Author: Rubricon contributors
License: Apache-2.0
License-File: LICENSE
Keywords: agents,calibration,evaluation,langchain,llm,llm-as-a-judge,observability,rag
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: numpy>=1.26
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: scipy>=1.11
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: fastapi>=0.115; extra == 'all'
Requires-Dist: langchain-core>=0.3; extra == 'all'
Requires-Dist: openai>=1.40; extra == 'all'
Requires-Dist: psycopg[binary]>=3.2; extra == 'all'
Requires-Dist: uvicorn[standard]>=0.30; extra == 'all'
Provides-Extra: judge
Requires-Dist: openai>=1.40; extra == 'judge'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
Provides-Extra: server
Requires-Dist: fastapi>=0.115; extra == 'server'
Requires-Dist: uvicorn[standard]>=0.30; extra == 'server'
Description-Content-Type: text/markdown

<div align="center">

# Rubricon

**The eval reliability platform.**

Every other tool tells you how good your model is.
Rubricon tells you whether your eval can be trusted.

*Every score ships with an error bar and a calibration receipt.*

[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org)

</div>

---

## The problem

Your dashboard says **75% pass rate**. You ship.

That number is wrong, and more data will not fix it:

- **Your judge is biased.** LLM judges have imperfect sensitivity and specificity, and they
  drift lenient — prompt-iterating on "is this response acceptable?" is a process that
  converges on yes. A naive judge score is a *biased estimator*, and the bias does not shrink
  as you collect more samples. More data just tightens the interval around the wrong number.
- **Your confidence interval is too narrow.** Variance from prompt phrasing, judge choice and
  temperature is not captured by a naive binomial CI.
- **You never checked for bias modes.** Position bias, verbosity bias and self-preference bias
  are measurable in an afternoon. Almost nobody measures them.
- **Your sample is too small to see the delta you care about.** "We went from 72% to 75%" is
  usually noise at n=200.

The eval SDK layer is a commodity in 2026. Tracing, judge templates, scorer registries —
everything exists. **Statistical trustworthiness of the resulting number does not.**

That is the entire product.

## The hook

```console
$ rubricon align examples/support_bot/suite.yaml \
      --judge helpfulness --labels examples/support_bot/labels.jsonl

  Judge "helpfulness" · 9ce49933373f  ·  n=40 scored  ·  24 human-labelled
==============================================================================

   Raw score (what your dashboard shows)          0.750
   Bias-corrected (stratified PPV/NPV)            0.561   95% CI [0.364, 0.708]  <-
   Bias-corrected (Rogan-Gladen)                  0.735   95% CI [0.411, 1.000]

   Judge precision (PPV)                          0.714   [0.454, 0.883]
   Judge NPV                                      0.900   [0.596, 0.982]
   Agreement with humans (kappa)                  0.589   (moderate)

   ✓  Verbosity bias    -0.079   controlling for true quality   (p=0.722)

   → the labels were sampled by the judge's verdict rather than at random. That makes
     sensitivity and specificity biased — and so makes Rogan-Gladen biased — but leaves
     PPV and NPV valid, so prevalence is recovered from those instead.

   ➜  Your "75% pass rate" is really 36% to 71%.
   ➜  When your judge says pass it is right 71% of the time; when it says fail, 90%.
   ➜  Agreement with humans is only 0.59 (moderate). Below 0.60 the judge and the humans
      are measuring different things — fix the rubric before reading anything else here.
   ➜  n=40 is too small to see +3.0 pp. The smallest detectable improvement here is
      +21.0 pp; you would need n=2,137.
```

**The true rate is 0.600.** That is not a claim — it is knowable, because
[`examples/support_bot`](examples/support_bot) is forty responses whose helpfulness was
decided when they were written. The raw score is 15 points out. The corrected interval
contains the truth. You can run it yourself and check.

Three things in that output are the reason this tool exists:

- **It picked a different estimator, and said why.** The labels were drawn evenly from the
  judge's passes and fails — the right way to label, and the only way to get enough of the
  rarer class. It also breaks Rogan-Gladen. Rubricon detects the design, switches to the
  estimator that survives it, and marks the other one biased rather than quietly reporting it.
- **It refused to be confident.** kappa 0.589 is below the line where a judge and its humans
  are measuring the same thing, and the report says so instead of printing the number and
  moving on.
- **It told you the run could not answer your question.** +3 pp was never detectable at n=40.

## Quickstart

```bash
pip install rubricon-eval            # core: stats, scorers, CLI, storage
pip install "rubricon-eval[all]"     # + judges via OpenRouter, dashboard, Postgres
```

The **distribution** is `rubricon-eval`; the import and the command are both `rubricon`. Plain
`rubricon` on PyPI belongs to an unrelated project, so `pip install rubricon` gets you someone
else's package — check the hyphen.

Not published yet. Until it is, install from source:

```bash
git clone https://github.com/sachinn854/rubricon && cd rubricon
uv sync --all-extras                 # or: pip install -e ".[all]"
```

```bash
rubricon init my-eval                       # scaffold a project
export OPENROUTER_API_KEY=sk-or-...         # or a .env file; code scorers need no key

rubricon eval suites/example.yaml --plan    # what will this cost? spends nothing
rubricon eval suites/example.yaml           # run it

rubricon annotate suites/example.yaml --judge helpfulness   # label, verdict-stratified
rubricon align suites/example.yaml --judge helpfulness \
    --labels labels/labels.jsonl            # ← the calibration receipt

rubricon experiment base.yaml candidate.yaml --gate   # A/B, paired, with an MDE
rubricon eval suites/example.yaml --gate              # CI threshold gates
rubricon serve                                        # dashboard on :6006
```

### What it costs

```
LLM calls = samples x judges
```

Every code and retrieval scorer is free, the whole statistics layer is free, and calibration
re-reads scores you already paid for rather than making new calls. Only judges bill, and each
one makes exactly one call per sample — no self-consistency, no majority voting.

Two things cut that further. The rubric is identical on every call, so it is sent as a cache
breakpoint and the run is scheduled to make caching actually pay: one call goes first and
writes the cache entry, then the rest fan out and read it. Fire all 500 at once and every one
of them misses.

And because an eval runs offline against traffic that has already been served, nothing is
waiting on the answer — which is what batch pricing is for:

```yaml
judge:
  model: google/gemini-2.5-flash:batch   # half price, queued rather than immediate
```

`rubricon eval --plan` prices both and shows the difference before you spend anything.

The stats engine has **no LLM dependency**. `rubricon.stats` imports numpy and scipy and
nothing else — including nothing else from Rubricon — so it works on any array of scores and
labels, from any tool:

```python
from rubricon.stats import ConfusionMatrix, stratified_prevalence

cm = ConfusionMatrix.from_labels(human_labels, judge_labels)
print(
    stratified_prevalence(
        0.75,
        cm.precision().value,
        cm.npv().value,
        n_test=1000,
        n_labelled_positive=150,
        n_labelled_negative=150,
    )
)
```

For the full report with no API key at all, `uv run python examples/hero_report.py` builds one
from simulated data with a known truth of 0.710 and prints exactly what `align` prints.

## Already have an app? Record it

The quickstart above starts from a dataset. Most people start from an application that is
already running, and the fastest eval set is the traffic it already serves.

```python
import rubricon

rubricon.instrument()  # once, at startup, before anything is handled
```

That is the whole change. It returns the integrations it attached — `['langchain']` for
LangChain or LangGraph — and switches recording on either way. Requests are appended to
`.rubricon/traces.jsonl`, one line each: the question, the answer, retrieved documents with
their ranks and scores, model calls with token usage, and per-step timings.

```bash
rubricon traces                      # what was captured, and what was not
rubricon dataset from-traces         # turn it into an eval set
```

### If your retriever is not a `BaseRetriever`

Rubricon sees retrieval and model calls through LangChain's callback system. A hand-rolled
retriever class, or the provider SDK called directly, is invisible to it — the graph is
captured, and nothing inside it is. This is common, and it is not a failure to be discovered
later: the trace carries a `capture_gap` message saying exactly this, and `rubricon traces`
prints it.

Mark those two points yourself:

```python
import rubricon

rubricon.instrument()

with rubricon.record(question=question) as run:
    chunks = my_retriever.search(question)
    rubricon.record_retrieval(question, chunks)

    run.answer = my_llm(question, chunks)
```

`record_retrieval` accepts whatever your retriever returns — dicts, dataclasses, LangChain
`Document`s, plain strings — reading each document's id and text from whichever of these it
has, attribute or metadata:

| | keys, in priority order |
|---|---|
| id | `id` `doc_id` `document_id` `chunk_id` `_id` `source` `file_path` |
| text | `text` `content` `page_content` `chunk` `body` `passage` |

**The id is the part that matters.** Recall and precision are computed by comparing these ids
against your dataset's `expected.relevant_doc_ids`, so they have to be written against the
same thing. Ids that do not distinguish anything — every chunk arriving as `doc` — make every
retrieval metric meaningless, and
`python scripts/verify_capture.py .rubricon/traces.jsonl` will say so before you build a
dataset from them.

Recording is sampled and bounded, so leaving it on is affordable:

```python
rubricon.instrument(sample_rate=0.05, path="/var/log/rubricon/traces.jsonl")
```

Nothing is sent anywhere. The file is local, and so is everything that reads it.

## Two examples, both with known ground truth

Neither is a toy demo — they are the only way to check that an estimator is right, because on
your own data you can never compare a correction against the answer.

**[`examples/support_bot`](examples/support_bot)** — 40 responses, true helpfulness 0.600.
Shows the calibration story end to end, and includes `bakeoff.py`, which is how Rubricon's own
default judge model was chosen: run four candidates against known truth and read the result.
Every one of them overstates the truth by 15 to 20 points; every corrected interval contains
it. Correction is what makes a cheap judge usable.

**[`examples/rag_bot`](examples/rag_bot)** — 30 questions over a 12-document corpus, each
tagged with *why* it is hard. This is the component-level argument in one table:

```
                        context_recall   faithfulness   answer_relevance
clean            n= 8            1.000          1.000              0.875
hallucination    n= 6            1.000          0.000              1.000
off_topic        n= 4            1.000          1.000              0.250
retrieval_miss   n= 6            0.000          0.000              0.833
```

A hallucination and a retrieval miss produce equally wrong answers and are indistinguishable
downstream — but recall is 1.000 for one and 0.000 for the other, so the two rows point at
different teams. And a hallucination (right question, false facts) and an off-topic answer
(true facts, wrong question) are each caught by exactly one judge and invisible to the other.

## What is in the box

**Working today:**

| | |
|---|---|
| **Statistics core** | Wilson and BCa bootstrap intervals, Rogan-Gladen with a delta-method CI, PPI++, verdict-stratified PPV/NPV, Cohen's kappa and Krippendorff's alpha, `pass@k` / `pass^k`, power and MDE, four bias probes, variance decomposition, drift tests, Benjamini-Hochberg. Standalone. |
| **Judges** | YAML rubrics, content-hash versioning, structured-output ladder, any model family via OpenRouter, response caching, cost estimated before you spend it |
| **Code scorers** | 17 exact and free: match, regex, JSON schema, latency, cost, and six retrieval metrics for RAG |
| **Calibration** | `rubricon align` — the report above, with automatic estimator selection |
| **Experiments** | `rubricon experiment` — paired bootstrap and permutation test, MDE beside every delta, slice analysis corrected for multiplicity |
| **CI gates** | `--gate` on both commands. Fires only on a *statistically significant* regression; a gate that blocks on noise gets switched off |
| **Dashboard** | `rubricon serve` — run history, per-scorer breakdown, per-sample verdicts, judge versions. JSON twin of every page |

**Not built yet:** production trace ingest, error-analysis clustering, agent trajectory eval,
red teaming, online/drift monitoring, governance packs. See [ROADMAP.md](ROADMAP.md).

## Design principles

1. **No naked numbers.** Every score carries an interval and the assumptions behind it.
2. **The judge is a measuring instrument, so calibrate it.** You would not trust a thermometer
   you never checked against boiling water.
3. **Fail wide, not narrow.** An interval that is too wide is honest; one that is too narrow is
   a lie. Where the two trade off, Rubricon takes the conservative side.
4. **Say when you cannot answer.** "Not computable", "underpowered" and "this scorer never
   varies" are all real outputs. Silence would be worse.
5. **Local-first.** `pip install` → value in five minutes. No account, no cloud, no telemetry.
6. **The stats layer is standalone.** Use it with Langfuse, Phoenix, Braintrust or a CSV.

## How the numbers are checked

Every estimator is tested against hand-computed answers. That is not enough, and the file that
says so is [`tests/simulation/`](tests/simulation): data generated from a known ground truth
thousands of times, asserting that a 95% interval really covers 95% of the time. An interval
that is too narrow returns plausible numbers, passes every known-answer test, and quietly makes
every decision built on it wrong. Simulation is the only thing that catches it.

It has caught real bugs here, most recently a nominal 95% interval delivering **83%** in the
regime the estimator was written for. [`ISSUES.md`](ISSUES.md) records what is still open,
what was found, and — where an investigation refuted its own hypothesis — that too.

## Status

Pre-alpha, under active development. `rubricon align`, `eval`, `experiment`, `serve` and the
statistics core all work end to end against real models. Interfaces may still move.

## License

Apache-2.0
