Metadata-Version: 2.5
Name: rubricon-eval
Version: 0.3.0
Summary: The eval reliability platform — every score ships with an error bar and a calibration receipt.
Project-URL: Homepage, https://github.com/sachinn854/rubricon
Project-URL: Documentation, https://github.com/sachinn854/rubricon#readme
Project-URL: Issues, https://github.com/sachinn854/rubricon/issues
Project-URL: Changelog, https://github.com/sachinn854/rubricon/blob/main/CHANGELOG.md
Author: Rubricon contributors
License: Apache-2.0
License-File: LICENSE
Keywords: agents,calibration,evaluation,langchain,llm,llm-as-a-judge,observability,rag
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: numpy>=1.26
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: scipy>=1.11
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: fastapi>=0.115; extra == 'all'
Requires-Dist: langchain-core>=0.3; extra == 'all'
Requires-Dist: openai>=1.40; extra == 'all'
Requires-Dist: psycopg[binary]>=3.2; extra == 'all'
Requires-Dist: uvicorn[standard]>=0.30; extra == 'all'
Provides-Extra: judge
Requires-Dist: openai>=1.40; extra == 'judge'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
Provides-Extra: server
Requires-Dist: fastapi>=0.115; extra == 'server'
Requires-Dist: uvicorn[standard]>=0.30; extra == 'server'
Description-Content-Type: text/markdown

<div align="center">

<img src="docs/assets/logo-full.png" alt="Rubricon-eval logo: a fused R/E monogram with an open-book icon in the counter." width="420">

**The eval reliability platform.**

Every other tool tells you how good your model is.
rubricon-eval tells you whether your eval can be trusted.

*Every score ships with an error bar and a calibration receipt.*

[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org)
[![Status](https://img.shields.io/badge/status-pre--alpha-orange.svg)](#status)
[![No telemetry](https://img.shields.io/badge/telemetry-none-brightgreen.svg)](#design-principles)

[Installation](#installation) ·
[Commands](#command-reference) ·
[Quickstart](#quickstart) ·
[Dashboard](#dashboard) ·
[Examples](#two-examples-both-with-known-ground-truth) ·
[FAQ](#faq)

</div>

<br>

<div align="center">
  <img src="docs/assets/calibration.png" alt="The rubricon dashboard showing a calibration receipt: a raw score of 0.820, a corrected score of 0.741 with a 95% confidence interval, and a table of judge precision, recall and agreement statistics." width="880">
  <p><em>The calibration receipt — a raw score is a point; the honest answer is an interval.</em></p>
</div>

---

## Table of contents

- [The problem](#the-problem)
- [The hook](#the-hook)
- [Installation](#installation)
- [Command reference](#command-reference)
- [Quickstart](#quickstart)
- [Is the number the measurement, or the system?](#is-the-number-the-measurement-or-the-system)
- [Already have an app? Record it](#already-have-an-app-record-it)
- [Dashboard](#dashboard)
- [Two examples, both with known ground truth](#two-examples-both-with-known-ground-truth)
- [What is in the box](#what-is-in-the-box)
- [Design principles](#design-principles)
- [How the numbers are checked](#how-the-numbers-are-checked)
- [FAQ](#faq)
- [Status](#status)
- [License](#license)

## The problem

Your dashboard says **75% pass rate**. You ship.

That number is wrong, and more data will not fix it:

- **Your judge is biased.** LLM judges have imperfect sensitivity and specificity, and they
  drift lenient — prompt-iterating on "is this response acceptable?" is a process that
  converges on yes. A naive judge score is a *biased estimator*, and the bias does not shrink
  as you collect more samples. More data just tightens the interval around the wrong number.
- **Your confidence interval is too narrow.** Variance from prompt phrasing, judge choice and
  temperature is not captured by a naive binomial CI.
- **You never checked for bias modes.** Position bias, verbosity bias and self-preference bias
  are measurable in an afternoon. Almost nobody measures them.
- **Your sample is too small to see the delta you care about.** "We went from 72% to 75%" is
  usually noise at n=200.
- **You never checked whether the score is even the judge you meant to keep.** A different
  seed, a different model, a rubric with its worked examples switched off — each one moves
  the score, and nobody records which was in effect when a number got published.

The eval SDK layer is a commodity in 2026. Tracing, judge templates, scorer registries —
everything exists. **Statistical trustworthiness of the resulting number does not.**

That is the entire product.

## The hook

```console
$ rubricon align examples/support_bot/suite.yaml \
      --judge helpfulness --labels examples/support_bot/labels.jsonl

  Judge "helpfulness" · 9ce49933373f  ·  n=40 scored  ·  24 human-labelled
==============================================================================

   Raw score (what your dashboard shows)          0.750
   Bias-corrected (stratified PPV/NPV)            0.561   95% CI [0.364, 0.708]  <-
   Bias-corrected (Rogan-Gladen)                  0.735   95% CI [0.411, 1.000]

   Judge precision (PPV)                          0.714   [0.454, 0.883]
   Judge NPV                                      0.900   [0.596, 0.982]
   Agreement with humans (kappa)                  0.589   (moderate)

   ✓  Verbosity bias    -0.079   controlling for true quality   (p=0.722)

   → the labels were sampled by the judge's verdict rather than at random. That makes
     sensitivity and specificity biased — and so makes Rogan-Gladen biased — but leaves
     PPV and NPV valid, so prevalence is recovered from those instead.

   ➜  Your "75% pass rate" is really 36% to 71%.
   ➜  When your judge says pass it is right 71% of the time; when it says fail, 90%.
   ➜  Agreement with humans is only 0.59 (moderate). Below 0.60 the judge and the humans
      are measuring different things — fix the rubric before reading anything else here.
   ➜  n=40 is too small to see +3.0 pp. The smallest detectable improvement here is
      +21.0 pp; you would need n=2,137.
```

**The true rate is 0.600.** That is not a claim — it is knowable, because
[`examples/support_bot`](examples/support_bot) is forty responses whose helpfulness was
decided when they were written. The raw score is 15 points out. The corrected interval
contains the truth. You can run it yourself and check — see [Installation](#installation).

Three things in that output are the reason this tool exists:

- **It picked a different estimator, and said why.** The labels were drawn evenly from the
  judge's passes and fails — the right way to label, and the only way to get enough of the
  rarer class. It also breaks Rogan-Gladen. rubricon-eval detects the design, switches to the
  estimator that survives it, and marks the other one biased rather than quietly reporting it.
- **It refused to be confident.** kappa 0.589 is below the line where a judge and its humans
  are measuring the same thing, and the report says so instead of printing the number and
  moving on.
- **It told you the run could not answer your question.** +3 pp was never detectable at n=40.

## Installation

### Requirements

- Python **3.11+**
- An [OpenRouter](https://openrouter.ai) API key — only if you use an LLM judge. The
  statistics layer and the 17 code scorers need no key at all.

### From PyPI

```bash
pip install "rubricon-eval[all]"
```

> [!IMPORTANT]
> The **distribution** on PyPI is `rubricon-eval`; the import and the command are both
> `rubricon`. Plain `rubricon` on PyPI belongs to an unrelated project in a confusingly
> adjacent domain — `pip install rubricon` (no hyphen, no `-eval`) gets you someone else's
> package. Check the hyphen.

`[all]` is what every command in this README needs. The bare `pip install rubricon-eval` is
deliberately small — statistics, code scorers, storage and the CLI, on `numpy` and `scipy`
only — because `rubricon.stats` is useful against a CSV from any other tool and should not
drag in a web server to get there. Install only the pieces you need instead:

| Extra | Adds | Needed for |
|---|---|---|
| *(none)* | `numpy`, `scipy` | `rubricon.stats` standalone, code scorers, `rubricon eval` on a target with no judge |
| `judge` | `openai` (OpenRouter speaks its API) | `rubricon eval`, `align`, `harness` — anything with an LLM judge |
| `server` | `fastapi`, `uvicorn` | `rubricon serve` (the dashboard) |
| `langchain` | `langchain-core` | automatic capture from LangChain / LangGraph via `rubricon.instrument()` |
| `postgres` | `psycopg` | a shared Postgres store instead of local SQLite |
| `all` | every extra above | everything in this README |

```bash
# pick only what you need
pip install "rubricon-eval[judge,server]"
```

### From source

```bash
git clone https://github.com/sachinn854/rubricon && cd rubricon

# with uv (recommended — this is what the project itself uses)
uv sync --all-extras

# or with plain pip
pip install -e ".[all]"
```

### Verify it worked

```bash
rubricon version
rubricon --help
```

### Set your API key

Only needed for LLM judges — code scorers and the statistics layer work with no key.

```bash
export OPENROUTER_API_KEY=sk-or-...
```

or drop it in a `.env` file at your project root; rubricon-eval reads it automatically.

## Command reference

Every command supports `--help` for the full flag list; this is the one-line map of what
exists and when you would reach for it.

| Command | What it does |
|---|---|
| `rubricon init [path]` | Scaffold a project: config, rubrics to fork, an example suite and dataset. `--rag` scaffolds RAG rubrics instead of chat ones. `--quick` scaffolds a target that calls a model directly — no code of yours to wire up, `rubricon eval` runs as soon as an API key is set. |
| `rubricon eval <suite>` | Run a suite and report the results. `--plan` prices it first; `--gate` fails the command on a threshold breach; `--json` for machine-readable output. |
| `rubricon annotate <suite> --judge <name>` | Label responses by hand, to calibrate a judge against. Draws samples stratified by the judge's own verdict, so the rarer class is actually reachable. |
| `rubricon align <suite> --judge <name>` | Calibrate a judge and report what its raw score is actually worth — the receipt shown above. This is the flagship command. |
| `rubricon harness <suite>` | Ask how much of your score is the measurement rather than the system — sweep judge model, seed, effort, anchors and see the spread. See [below](#is-the-number-the-measurement-or-the-system). |
| `rubricon matrix <suite> --target-model a,b,c` | Which model should answer this prompt? Runs the same suite once per candidate target model, judges held fixed, every scorer's mean reported side by side. The mirror image of `harness` — that sweeps the judge and never the target, this sweeps the target and never the judge. Needs a `kind: prompt` target. |
| `rubricon drift <reference> <current>` | Check whether a dataset still looks like the one you evaluated on — PSI, KS, MMD or energy-distance test on question length. Costs nothing, needs no API key. Point `<current>` at `dataset from-traces` output to check real traffic against your golden set. |
| `rubricon scan <suite> --metric <name>` | Check a judge for bias: does it reward length (verbosity), or favour its own model family (self-preference, needs a `--matrix-report`)? Position, sycophancy and formatting bias are named as not-yet-built rather than silently skipped. |
| `rubricon experiment <baseline> <candidate>` | Compare two suites and say whether the difference is real: paired bootstrap, sign-flip permutation test, MDE beside every delta. `--gate` for CI. |
| `rubricon serve` | Serve the local dashboard (`:6006` by default) over every run, calibration, experiment and harness sweep recorded in this project. |
| `rubricon traces` | Show what `rubricon.instrument()` has recorded — and what it could not capture. |
| `rubricon dataset from-traces` | Turn recorded traffic into an eval dataset. |
| `rubricon dataset generate` | No dataset yet? Draft one from a description and/or a few real examples. Generates questions only — never a fabricated answer or label. |
| `rubricon rubrics list` / `show <name>` / `copy <name>` | Inspect and copy the built-in judge rubrics into your own project so you can edit them. |
| `rubricon scorers` | List every registered code scorer — 17 of them, all free, no API key required. |
| `rubricon version` | Print the installed version. |

Flags that show up on almost every command: `--limit -n` (only the first N samples, for a fast
smoke test), `--concurrency -c` (samples in flight, default 8), `--model -m` (force a
different judge model without editing YAML), `--json` (machine-readable output for scripting).

## Quickstart

```bash
rubricon init my-eval --quick && cd my-eval  # scaffold a project — runs with no code of yours
export OPENROUTER_API_KEY=sk-or-...          # or a .env file; code scorers need no key

rubricon eval suites/example.yaml --plan     # what will this cost? spends nothing
rubricon eval suites/example.yaml            # run it

rubricon annotate suites/example.yaml --judge helpfulness   # label, verdict-stratified
rubricon align suites/example.yaml --judge helpfulness \
    --labels labels/labels.jsonl             # ← the calibration receipt

rubricon experiment base.yaml candidate.yaml --gate    # A/B, paired, with an MDE
rubricon eval suites/example.yaml --gate               # CI threshold gates

rubricon harness suites/example.yaml \
    --judge-model a,b --seed 1,2 --detect 0.03         # how much of the score is the setup?

rubricon serve                                         # dashboard on :6006
```

Already running `pytest` in CI? The same gate works as an assertion — no new tool to wire in:

```python
from rubricon.testing import assert_gate


def test_helpfulness_holds():
    assert_gate("suites/example.yaml")  # runs the suite for real, raises if a gate fails
```

### What it costs

```
LLM calls = samples x judges
```

Every code and retrieval scorer is free, the whole statistics layer is free, and calibration
re-reads scores you already paid for rather than making new calls. Only judges bill, and each
one makes exactly one call per sample — no self-consistency, no majority voting.

Two things cut that further. The rubric is identical on every call, so it is sent as a cache
breakpoint and the run is scheduled to make caching actually pay: one call goes first and
writes the cache entry, then the rest fan out and read it. Fire all 500 at once and every one
of them misses.

And because an eval runs offline against traffic that has already been served, nothing is
waiting on the answer — which is what batch pricing is for:

```yaml
judge:
  model: google/gemini-2.5-flash:batch   # half price, queued rather than immediate
```

`rubricon eval --plan` prices both and shows the difference before you spend anything.

The stats engine has **no LLM dependency**. `rubricon.stats` imports numpy and scipy and
nothing else — including nothing else from rubricon-eval — so it works on any array of scores and
labels, from any tool:

```python
from rubricon.stats import ConfusionMatrix, stratified_prevalence

cm = ConfusionMatrix.from_labels(human_labels, judge_labels)
print(
    stratified_prevalence(
        0.75,
        cm.precision().value,
        cm.npv().value,
        n_test=1000,
        n_labelled_positive=150,
        n_labelled_negative=150,
    )
)
```

For the full report with no API key at all, `uv run python examples/hero_report.py` builds one
from simulated data with a known truth of 0.710 and prints exactly what `align` prints.

## Is the number the measurement, or the system?

<div align="center">
  <img src="docs/assets/harness.png" alt="The rubricon dashboard's harness detail page, showing a spread of +12.4 percentage points across six judge setups, a warning that the setup swamps the change being detected, and a table decomposing the movement by factor." width="880">
</div>

`align` corrects one judge, set up one way. It cannot tell you whether a published improvement
would survive someone else picking a different judge model, a different seed, or leaving the
rubric's worked examples out. Nobody records those choices, and each of them moves the score.

`rubricon harness` scores the *same* samples across a grid of measurement setups and reports
how far the number travels, and which factor is doing the travelling. Only measurement factors
are varied — the judge's model, its seed, its reasoning effort, whether its anchors are shown —
never the target or the dataset, because that is a different question (`experiment` answers
it), and mixing the two would produce a spread that means neither.

```bash
rubricon harness suites/example.yaml \
    --judge-model google/gemini-2.5-flash,openai/gpt-5-mini \
    --seed 1,2,3 \
    --detect 0.03            # the improvement you are trying to publish
```

The four raw scores in the [judge bake-off](examples/support_bot) — one model varied, nothing
else — are already a one-factor sweep, and the real report over them reads:

```console
Harness sweep · helpfulness · 4 cells · 40 samples · $0.0758

  judge_model    4 levels   google/gemini-2.5-flash, google/gemini-2.5-flash-lite, openai/gpt-5-mini, openai/gpt-oss-120b

   0.750   judge_model=google/gemini-2.5-flash
   0.750   judge_model=google/gemini-2.5-flash-lite
   0.775   judge_model=openai/gpt-5-mini
   0.800   judge_model=openai/gpt-oss-120b

  spread     5.0 pp

  ➜ Your score moves 5.0 pp depending on how you measured, not on what you measured: 0.750 at [judge_model=google/gemini-2.5-flash] and 0.800 at [judge_model=openai/gpt-oss-120b].
  ➜ That is larger than the 3.0 pp you are trying to detect. A change that size cannot be told apart from a different judge setup, so pin the setup and report it before comparing anything.
```

Add `seed` or `effort` to the grid and the report also decomposes the spread by factor — `41%
of your score's variance comes from judge_model, not from the system under test` is a real
line it can print, using the same variance-decomposition machinery `align` uses for prompt and
temperature nuisance factors. The target runs once and is cached; only judge calls multiply
with the grid, and `--plan` prices the whole thing before anything is spent. A grid where
nothing varies is refused rather than run, because one factor pinned to one level is a normal
`eval` wearing a sweep's clothes.

## Already have an app? Record it

The quickstart above starts from a dataset. Most people start from an application that is
already running, and the fastest eval set is the traffic it already serves.

```python
import rubricon

rubricon.instrument()  # once, at startup, before anything is handled
```

That is the whole change. It returns the integrations it attached — `['langchain']` for
LangChain or LangGraph — and switches recording on either way. Requests are appended to
`.rubricon/traces.jsonl`, one line each: the question, the answer, retrieved documents with
their ranks and scores, model calls with token usage, and per-step timings.

```bash
rubricon traces                      # what was captured, and what was not
rubricon dataset from-traces         # turn it into an eval set
```

### If your retriever is not a `BaseRetriever`

rubricon-eval sees retrieval and model calls through LangChain's callback system. A hand-rolled
retriever class, or the provider SDK called directly, is invisible to it — the graph is
captured, and nothing inside it is. This is common, and it is not a failure to be discovered
later: the trace carries a `capture_gap` message saying exactly this, and `rubricon traces`
prints it.

Mark those two points yourself:

```python
import rubricon

rubricon.instrument()

with rubricon.record(question=question) as run:
    chunks = my_retriever.search(question)
    rubricon.record_retrieval(question, chunks)

    response = my_llm(question, chunks)
    rubricon.record_llm(model="gpt-5", usage=response.usage, output=response.text)

    run.answer = response.text
```

**Both calls have to be inside the `record()` block.** That block is the request they attach
to; called outside one they have nothing to attach to, the span is dropped, and you get a
warning saying so.

`record_llm(usage=...)` takes the usage object your provider SDK already handed you — OpenAI's
`CompletionUsage`, Anthropic's `Usage`, Gemini's `UsageMetadata`, a plain dict, or rubricon-eval's
own `rubricon.TokenUsage`. Cached prompt tokens are recognised and not double-counted. A shape
it cannot read costs the token numbers and nothing else. Usage is what the `cost` and `latency`
scorers read, so a pipeline that never threads it out of its generator simply does without
those two — everything else still scores.

`record_retrieval` accepts whatever your retriever returns — dicts, dataclasses, LangChain
`Document`s, plain strings — reading each document's id and text from whichever of these it
has, attribute or metadata:

| | keys, in priority order |
|---|---|
| id | `id` `doc_id` `document_id` `chunk_id` `_id` `source` `file_path` |
| text | `text` `content` `page_content` `chunk` `body` `passage` |

**The id is the part that matters.** Recall and precision are computed by comparing these ids
against your dataset's `expected.relevant_doc_ids`, so they have to be written against the
same thing. Ids that do not distinguish anything — every chunk arriving as `doc` — make every
retrieval metric meaningless, and
`python scripts/verify_capture.py .rubricon/traces.jsonl` will say so before you build a
dataset from them.

### When retrieval happens somewhere your route cannot reach

The example above assumes the code holding the request also holds the retriever. In an
orchestrated app it usually does not: retrieval runs inside a graph node several layers below
`graph.ainvoke()`, and the route never sees it. Do not run retrieval a second time to record it
— that is a different retrieval from the one the answer came from. Record it from the final
state instead:

```python
with rubricon.record(question=question) as run:
    result = await graph.ainvoke({"question": question})
    rubricon.record_retrieval(question, result["chunks"], duration_ms=None)
    run.answer = result["answer"]
```

Ids and text come through correctly, and two things do not: the span carries no duration, and a
request that fails *after* retrieval records no retrieval at all — even though retrieval
succeeded, which is exactly what you would want to see while debugging that failure. If either
matters, pass the node's own timing through the graph state and record from inside the node.

### Where `instrument()` goes

Recording is process-wide and has to be on before the first request is handled, so it belongs
at **module scope**, not in a startup hook:

```python
# main.py
import rubricon

rubricon.instrument(path="traces.jsonl")  # module scope: runs at import, before any request

app = FastAPI(lifespan=lifespan)
```

FastAPI's `lifespan` is the idiomatic place for startup work and the wrong place for this: it
runs per worker process after the app object exists, and anything imported and handled before
it is missed.

**Under multiple workers, give each process its own file.** The write lock is a
`threading.Lock`, which does not coordinate across processes, and a trace carrying eight
documents is a single line well past any atomic-append guarantee:

```python
rubricon.instrument(path=f"traces-{os.getpid()}.jsonl")
```

`rubricon dataset from-traces --path` reads one file; concatenate them first, or point it at
each in turn.

Recording is sampled and bounded, so leaving it on is affordable:

```python
rubricon.instrument(sample_rate=0.05, path="/var/log/rubricon/traces.jsonl")
```

Nothing is sent anywhere. The file is local, and so is everything that reads it.

## Dashboard

<div align="center">
  <img src="docs/assets/runs.png" alt="The rubricon dashboard's runs page, showing store-wide statistics — runs, samples, errors, spend — and a table of recent runs with status, sample count, error rate and cost." width="880">
</div>

```bash
rubricon serve            # :6006 by default
rubricon serve --port 6017 --host 0.0.0.0   # a different port, or exposed off localhost
```

A local React application, compiled into the package — `pip install` gets a working dashboard
with no separate `npm install`. Every page has a JSON twin under `/api/*`, so anything the
dashboard shows is also a URL you can `curl` or script against. It reads the same SQLite file
every CLI command writes to, so there is nothing separate to keep in sync: run `align`, then
`serve`, and the receipt is already there.

Run history, per-scorer breakdowns, per-sample verdicts with the judge's reasoning, judge
versions, calibration receipts, experiment comparisons, and harness sweeps — every object this
README talks about has a page. Command-K (or Ctrl-K) opens a search palette over all of them.

## Two examples, both with known ground truth

Neither is a toy demo — they are the only way to check that an estimator is right, because on
your own data you can never compare a correction against the answer.

**[`examples/support_bot`](examples/support_bot)** — 40 responses, true helpfulness 0.600.
Shows the calibration story end to end, and includes `bakeoff.py`, which is how rubricon-eval's own
default judge model was chosen: run four candidates against known truth and read the result.
Every one of them overstates the truth by 15 to 20 points; every corrected interval contains
it. Correction is what makes a cheap judge usable.

**[`examples/rag_bot`](examples/rag_bot)** — 30 questions over a 12-document corpus, each
tagged with *why* it is hard. This is the component-level argument in one table:

```
                        context_recall   faithfulness   answer_relevance
clean            n= 8            1.000          1.000              0.875
hallucination    n= 6            1.000          0.000              1.000
off_topic        n= 4            1.000          1.000              0.250
retrieval_miss   n= 6            0.000          0.000              0.833
```

A hallucination and a retrieval miss produce equally wrong answers and are indistinguishable
downstream — but recall is 1.000 for one and 0.000 for the other, so the two rows point at
different teams. And a hallucination (right question, false facts) and an off-topic answer
(true facts, wrong question) are each caught by exactly one judge and invisible to the other.

## What is in the box

**Working today:**

| | |
|---|---|
| **Statistics core** | Wilson and BCa bootstrap intervals, Rogan-Gladen with a delta-method CI, PPI++, verdict-stratified PPV/NPV, Cohen's kappa and Krippendorff's alpha, `pass@k` / `pass^k`, power and MDE, four bias probes, variance decomposition, drift tests, Benjamini-Hochberg. Standalone. |
| **Judges** | YAML rubrics, content-hash versioning, structured-output ladder, any model family via OpenRouter, response caching, cost estimated before you spend it. 12 built-in rubrics across chat, RAG, code and safety (`rubricon rubrics list`) |
| **Code scorers** | 17 exact and free: match, regex, JSON schema, latency, cost, and six retrieval metrics for RAG |
| **Calibration** | `rubricon align` — the report above, with automatic estimator selection |
| **Experiments** | `rubricon experiment` — paired bootstrap and permutation test, MDE beside every delta, slice analysis corrected for multiplicity |
| **Harness sweeps** | `rubricon harness` — scores the same samples across judge model, seed, effort and anchors, reports the spread and decomposes it by factor |
| **Target matrix** | `rubricon matrix` — the same suite run once per candidate target model, judges held fixed, every scorer's mean side by side. Which model should answer this prompt, not how much of the score is the measurement |
| **CI gates** | `--gate` on both `eval` and `experiment`. Fires only on a *statistically significant* regression; a gate that blocks on noise gets switched off. `rubricon.testing.assert_gate` is the same check as a pytest assertion, for teams already running `pytest` in CI |
| **Dashboard** | `rubricon serve` — run history, per-scorer breakdown, per-sample verdicts, judge versions, harness sweeps. JSON twin of every page |
| **Trace capture** | `rubricon.instrument()` records real traffic in-process (no OTel setup needed); `rubricon traces` inspects what was captured; `rubricon dataset from-traces` turns it into an eval dataset with retrieval spans attached |
| **Drift detection** | `rubricon drift` — PSI, KS, MMD or energy-distance test on question length between two datasets. Free, instant, no API key |
| **Dataset bootstrap** | `rubricon dataset generate` — draft questions from a description and/or a few real examples. No fabricated ground truth: it writes `input.question` only, tagged `synthetic` |
| **Bias scan** | `rubricon scan` — verbosity bias from the run itself, self-preference bias from a `rubricon matrix` report. Position/sycophancy/formatting bias are reported not-yet-built, not silently skipped |

**Not built yet:** OTel-standard ingest for production-scale/multi-service traffic (today's
capture is a lightweight in-process recorder, not a receiver), error-analysis clustering, agent
trajectory eval, red teaming, governance packs.

## Design principles

1. **No naked numbers.** Every score carries an interval and the assumptions behind it.
2. **The judge is a measuring instrument, so calibrate it.** You would not trust a thermometer
   you never checked against boiling water.
3. **Fail wide, not narrow.** An interval that is too wide is honest; one that is too narrow is
   a lie. Where the two trade off, rubricon-eval takes the conservative side.
4. **Say when you cannot answer.** "Not computable", "underpowered" and "this scorer never
   varies" are all real outputs. Silence would be worse.
5. **Local-first.** `pip install` → value in five minutes. No account, no cloud, no telemetry.
6. **The stats layer is standalone.** Use it with Langfuse, Phoenix, Braintrust or a CSV.

## How the numbers are checked

Every estimator is tested against hand-computed answers. That is not enough, and the file that
says so is [`tests/simulation/`](tests/simulation): data generated from a known ground truth
thousands of times, asserting that a 95% interval really covers 95% of the time. An interval
that is too narrow returns plausible numbers, passes every known-answer test, and quietly makes
every decision built on it wrong. Simulation is the only thing that catches it.

It has caught real bugs here, most recently a nominal 95% interval delivering **83%** in the
regime the estimator was written for. [`ISSUES.md`](ISSUES.md) records what is still open,
what was found, and — where an investigation refuted its own hypothesis — that too.

## FAQ

**Do I need an API key?**
Not for the statistics layer or the 17 code scorers — they run locally and free. You need one
only for LLM judges, which route through OpenRouter so a single client reaches every model
family. `rubricon eval --plan` prices a run before it spends anything.

**How many human labels does a calibration need?**
Fewer than you'd guess. The `support_bot` example corrects a 40-sample run from 24 labels.
Labels are drawn stratified by the judge's verdict — what makes the rarer class reachable at
all — and Neyman allocation decides the split. The report states the minimum detectable
effect, so you know whether the run could have answered your question before you read the
answer.

**Which judge model does it use by default?**
`google/gemini-2.5-flash`, chosen by the bake-off in `examples/support_bot/bakeoff.py`, not by
preference. `--model` overrides it for any command; the model otherwise resolves per judge,
then per suite, then per project, then from this default.

**Does it send my data anywhere?**
No. Local-first: no account, no cloud, no telemetry. Runs live in a local SQLite file and the
dashboard is a process on your machine. The only egress is the judge call to your own
provider, with your own key — pure code scorers make none at all.

**Can I use it alongside the eval tool I already have?**
That's the intended path. `rubricon.stats` depends on `numpy`/`scipy` and nothing else from
rubricon-eval, so you can point it at scores and labels from Langfuse, Phoenix, Braintrust or a CSV
without adopting anything else.

**Why `pip install rubricon-eval` and not `pip install rubricon`?**
Plain `rubricon` on PyPI already belongs to an unrelated project. The distribution is
`rubricon-eval`; the import package and the CLI command stay `rubricon`.

## Status

Pre-alpha, under active development. `rubricon align`, `eval`, `experiment`, `harness`,
`serve` and the statistics core all work end to end against real models. Interfaces may still
move.

## License

Apache-2.0
