Rubricon
the number nobody checks
Your eval says 75%. The truth is 60%.
Rubricon is the eval reliability platform. Every other tool tells you how good your model is; this one tells you whether the number can be trusted, and what it is really worth once the judge's own bias is taken out of it.
$ pip install rubricon-eval
The distribution isrubricon-eval; the import and the command are both
rubricon. Mind the hyphen: plain rubricon on PyPI is an
unrelated project.
This is checkable. The forty responses behind it were tagged as helpful or not when they were written, so the true rate really is 0.600. The raw score is fifteen points out — and no amount of extra samples would fix it. More data only tightens the interval around the wrong number.
a real run
One command. Everything it took to trust the number.
This is not a mock-up of a report. It is the report: the output of
rubricon align against the bundled example, which you can clone and run
yourself in about a minute.
$ rubricon align examples/support_bot/suite.yaml \ --judge helpfulness --labels examples/support_bot/labels.jsonl Judge "helpfulness" · 9ce49933373f · n=40 scored · 24 human-labelled ══════════════════════════════════════════════════════════════════════════ Raw score (what your dashboard shows) 0.750 Bias-corrected (stratified PPV/NPV) 0.561 95% CI [0.364, 0.708] <- Bias-corrected (Rogan-Gladen) 0.735 95% CI [0.411, 1.000] biased by this design Judge precision (PPV) 0.714 [0.454, 0.883] Judge NPV 0.900 [0.596, 0.982] Agreement with humans (kappa) 0.589 (moderate) ✓ Verbosity bias -0.079 controlling for true quality (p=0.722) → the labels were sampled by the judge's verdict rather than at random. That makes sensitivity and specificity biased — and so makes Rogan-Gladen biased — but leaves PPV and NPV valid, so prevalence is recovered from those instead. ➜ Your "75% pass rate" is really 36% to 71%. ➜ When your judge says pass it is right 71% of the time; when it says fail, 90%. ➜ Agreement with humans is only 0.59 (moderate). Below 0.60 the judge and the humans are measuring different things — fix the rubric before reading anything else here. ➜ n=40 is too small to see +3.0 pp. The smallest detectable improvement here is +21.0 pp; you would need n=2,137.
Three things in that output are the reason the tool exists. It picked a different estimator and said why: labelling evenly from the judge's passes and fails is the right way to label and it breaks Rogan-Gladen, so the biased one is marked rather than quietly printed. It refused to be confident at kappa 0.589. And it said the run could not answer the question that was probably being asked: +3 pp was never detectable at n=40.
the annoying part
A judge is a measuring instrument. Almost nobody calibrates theirs.
An LLM judge has imperfect sensitivity and specificity, and it drifts lenient — prompt-iterating on “is this response acceptable?” is a process that converges on yes. That makes a raw judge score a biased estimator, and the bias does not shrink as you collect more samples.
The bias does not average out
More data tightens the interval around the wrong number. Only a correction moves the number itself, and a correction needs human labels the tool knows how to spend sparingly.
Below this, stop reading
Agreement with humans under 0.60 means the judge and your labellers are measuring different things. Rubricon says so instead of printing the score anyway.
The smallest change the run could see
At n=40, a 3-point improvement was never detectable. “No significant change” and “no change” are not the same finding, and only one of them is a reason to ship.
Bias modes, measurable in an afternoon
Position, verbosity, self-preference and sycophancy probes ship with the tool and run beside every calibration. Almost nobody measures them.
the bake-off
The default judge model was chosen by measurement, not by vibes.
Four candidates, the same forty responses, each calibrated against the known truth of
0.600. examples/support_bot/bakeoff.py is the script that produced this
table, and it is the same script you would point at your own data.
drag the table sideways →
| Candidate | Raw | Corrected | 95% CI | Covers 0.600 | Kappa | Cost /40 |
|---|---|---|---|---|---|---|
| google/gemini-2.5-flashdefault | 0.750 | 0.589 | [0.393, 0.714] | yes | 0.753 | $0.0331 |
| google/gemini-2.5-flash-lite | 0.750 | 0.561 | [0.364, 0.708] | yes | 0.589 | $0.0073 |
| openai/gpt-5-mini | 0.775 | 0.568 | [0.372, 0.708] | yes | 0.673 | $0.0331 |
| openai/gpt-oss-120b | 0.800 | 0.550 | [0.355, 0.701] | yes | 0.595 | $0.0023 |
Every one of them covers the truth once corrected, and every raw score overstates it by 15 to 20 points in the same direction. That is the argument in one table: correction is what makes a cheap judge usable, not a better model. The corrected estimates barely separate the candidates; kappa does, and two of the four land below the 0.60 line at which this tool's own report tells you to stop reading. One honest caveat: 0.753 against 0.589 on 24 labels is about one standard error, so the ranking is not statistically established. What is not a question of significance is that one point estimate sits below the usability threshold and the other does not.
one level up from align
Your score moves depending on how you measured it.
align corrects one judge, set up one way. It cannot say whether a
published improvement would survive someone else picking a different judge model, a
different seed, or leaving the rubric's worked examples out. Nobody records those
choices, and each of them moves the score. rubricon harness scores the
same samples across a grid of them and reports how far the number travels.
$ rubricon harness suites/example.yaml \ --judge-model gemini-2.5-flash,gpt-5-mini,gpt-oss-120b --detect 0.03 Harness sweep · helpfulness · 4 cells · 40 samples · $0.0758 0.750 judge_model=google/gemini-2.5-flash 0.750 judge_model=google/gemini-2.5-flash-lite 0.775 judge_model=openai/gpt-5-mini 0.800 judge_model=openai/gpt-oss-120b spread 5.0 pp ➤ Your score moves 5.0 pp depending on how you measured, not on what you measured. ➤ That is larger than the 3.0 pp you are trying to detect. Pin the setup and report it before comparing anything measured under it.
Only measurement factors are swept — judge model, seed, reasoning effort, whether
the rubric's anchors are shown — never the target or the dataset, because that is
a different question and experiment already answers it. The target runs
once and is cached, so only judge calls multiply with the grid, and
--plan prices the whole sweep before anything is spent. Add seed
or effort to the grid and the report decomposes the spread by factor, using
the same variance-decomposition machinery align uses for prompt and
temperature nuisance factors.
what you get
What is in the box today.
Everything below works end to end against real models. Interfaces may still move: this is pre-alpha, and the roadmap says what it is not yet.
- Statistics
-
Wilson and BCa bootstrap intervals, Rogan-Gladen with a delta-method CI, PPI++,
verdict-stratified PPV/NPV via Zou & Donner MOVER, Cohen's kappa and
Krippendorff's alpha,
pass@k/pass^k, power and MDE, four bias probes, variance decomposition, drift tests, Benjamini-Hochberg and Holm-Bonferroni. Depends on numpy and scipy only. - Judges
-
YAML rubrics with content-hash versioning, a structured-output ladder that degrades
from
json_schemato tool calls to prompting, any model family through OpenRouter, response caching, and a cost estimate before you spend anything. - Code scorers
- Seventeen exact and free scorers: match, regex, JSON schema, latency, cost, and six retrieval metrics for RAG. No API key required for any of them.
- Calibration
-
rubricon align: the receipt above, with automatic estimator selection, verdict-stratified annotation, and target-output caching so calibrating the same samples never re-runs your pipeline. - Experiments
-
rubricon experiment: paired bootstrap and sign-flip permutation test, a minimum detectable effect beside every delta, and slice analysis corrected for multiplicity so twenty slices do not manufacture one “significant” finding. - CI gates
-
--gateon bothevalandexperiment. A gate fires only on a statistically significant regression, because a gate that blocks on noise is a gate that gets switched off. A skipped gate does not pass. - Storage
- SQLite by default, Postgres-compatible DDL, immutable judge and dataset versions. Every run, score, judge version and per-sample verdict is recorded.
- Dashboard
-
rubricon serve: run history, per-scorer breakdown, per-sample verdicts, judge versions. A local React application, with a JSON twin of every page.
install
Calibrate first. Everything downstream is anchored to that number.
Code scorers need no API key at all, so the first three commands cost nothing. Judges
route through OpenRouter; put the key in the environment or a .env file.
# scaffold a project pip install "rubricon-eval[all]" rubricon init my-eval # what will this cost? spends nothing rubricon eval suites/example.yaml --plan rubricon eval suites/example.yaml # label a sample, stratified by the judge's verdict rubricon annotate suites/example.yaml --judge helpfulness # the calibration receipt rubricon align suites/example.yaml --judge helpfulness \ --labels labels/labels.jsonl # A/B two versions, paired, with a minimum detectable effect rubricon experiment base.yaml candidate.yaml --gate # local dashboard on :6006 rubricon serve
Use the statistics without the rest
rubricon.stats imports numpy and scipy and nothing else, including
nothing else from Rubricon. Point it at scores from Langfuse, Phoenix, Braintrust
or a CSV.
from rubricon.stats import ( ConfusionMatrix, stratified_prevalence, ) cm = ConfusionMatrix.from_labels( human_labels, judge_labels, ) # the honest number, not the raw one print(stratified_prevalence( 0.75, # raw cm.precision().value, # PPV cm.npv().value, n_test=1000, n_labelled_positive=150, n_labelled_negative=150, ))
Block a regression, not a coin flip
The gate fires on a significant paired difference. If the run was underpowered to see the delta you care about, it says that instead of passing you quietly.
# .github/workflows/eval.yml - name: Evaluate the candidate run: | rubricon experiment \ suites/main.yaml \ suites/candidate.yaml \ --gate env: OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }} # exit 1 only on a significant # regression; noise does not # block the merge
before you install it
Questions worth answering first.
Do I need an API key?
Not for the statistics layer or the seventeen code scorers: match, regex, JSON
schema, latency, cost and the retrieval metrics all run locally and free. You need
one only for LLM judges, which route through OpenRouter so a single client reaches
every model family. rubricon eval --plan prices a run before it spends
anything.
How many human labels does a calibration need?
Fewer than you would guess, and the report tells you rather than making you estimate. The example above corrects a 40-sample run from 24 labels. Labels are drawn stratified by the judge's verdict, which is what makes the rarer class reachable at all, and Neyman allocation decides how to split them. The report then states the minimum detectable effect, so you know whether the run could have answered your question before you read the answer.
Which judge model does it use?
Whichever you say. The model resolves per judge, then per suite, then per project,
then from the packaged default, and --model overrides all four. The
default is google/gemini-2.5-flash, and it holds that position because
of the bake-off table above, not because of a preference. Re-run
bakeoff.py on your own data and the answer may well be different.
Does it send my data anywhere?
Local-first: no account, no cloud, no telemetry. Runs are stored in a local SQLite file and the dashboard is a process on your machine. The only egress is the judge call to your own provider, with your own key, and pure code scorers make none at all.
Can I use it alongside the eval tool I already have?
That is the intended path. The eval SDK layer (tracing, judge templates, scorer
registries) is a commodity now; statistical trustworthiness of the resulting number
is not, and that is the only thing Rubricon claims. rubricon.stats
depends on numpy and scipy and nothing else, not even the rest of this package, so
you can point it at an array of scores and labels from Langfuse, Phoenix,
Braintrust or a CSV without adopting anything else.
Why pip install rubricon-eval and not rubricon?
Plain rubricon on PyPI already belongs to an unrelated project in a
confusingly adjacent domain. Publishing under it is not an option and squatting
near it would be worse, so the distribution is rubricon-eval while the
import package and the command stay rubricon.
one minute, three cents
Find out what your 75% is really worth.
Clone the repo, run the bundled example, and compare the corrected interval against a truth you can check by opening the file.