Your eval says 75%. The truth is 60%.

One command. Everything it took to trust the number.

This is not a mock-up of a report. It is the report: the output of rubricon align against the bundled example, which you can clone and run yourself in about a minute.

$ rubricon align examples/support_bot/suite.yaml \
      --judge helpfulness --labels examples/support_bot/labels.jsonl

  Judge "helpfulness" · 9ce49933373f  ·  n=40 scored  ·  24 human-labelled
══════════════════════════════════════════════════════════════════════════

   Raw score (what your dashboard shows)          0.750
   Bias-corrected (stratified PPV/NPV)            0.561   95% CI [0.364, 0.708]  <-
   Bias-corrected (Rogan-Gladen)                  0.735   95% CI [0.411, 1.000]  biased by this design

   Judge precision (PPV)                          0.714   [0.454, 0.883]
   Judge NPV                                      0.900   [0.596, 0.982]
   Agreement with humans (kappa)                  0.589   (moderate)

   ✓  Verbosity bias    -0.079   controlling for true quality   (p=0.722)

   → the labels were sampled by the judge's verdict rather than at random. That
     makes sensitivity and specificity biased — and so makes Rogan-Gladen biased —
     but leaves PPV and NPV valid, so prevalence is recovered from those instead.

   ➜  Your "75% pass rate" is really 36% to 71%.
   ➜  When your judge says pass it is right 71% of the time; when it says fail, 90%.
   ➜  Agreement with humans is only 0.59 (moderate). Below 0.60 the judge and the
      humans are measuring different things — fix the rubric before reading anything
      else here.
   ➜  n=40 is too small to see +3.0 pp. The smallest detectable improvement here is
      +21.0 pp; you would need n=2,137.

Three things in that output are the reason the tool exists. It picked a different estimator and said why: labelling evenly from the judge's passes and fails is the right way to label and it breaks Rogan-Gladen, so the biased one is marked rather than quietly printed. It refused to be confident at kappa 0.589. And it said the run could not answer the question that was probably being asked: +3 pp was never detectable at n=40.

A judge is a measuring instrument. Almost nobody calibrates theirs.

An LLM judge has imperfect sensitivity and specificity, and it drifts lenient — prompt-iterating on “is this response acceptable?” is a process that converges on yes. That makes a raw judge score a biased estimator, and the bias does not shrink as you collect more samples.

+15pp

The bias does not average out More data tightens the interval around the wrong number. Only a correction moves the number itself, and a correction needs human labels the tool knows how to spend sparingly.

0.589

Below this, stop reading Agreement with humans under 0.60 means the judge and your labellers are measuring different things. Rubricon says so instead of printing the score anyway.

+21pp

The smallest change the run could have seen At n=40, a 3-point improvement was never detectable. “No significant change” and “no change” are not the same finding, and only one of them is a reason to ship.

4

Bias modes you can measure in an afternoon Position, verbosity, self-preference and sycophancy probes ship with the tool and run beside every calibration. Almost nobody measures them.

It picks the estimator your labelling design allows — and says which one it ruled out.

• It knows how you labelled Labels drawn evenly from the judge's passes and fails are the right way to label, and they break Rogan-Gladen. Rubricon detects the design, switches to the estimator that survives it, and marks the other one biased rather than quietly reporting it.
• It fails wide, not narrow An interval that is too wide is honest; one that is too narrow is a lie. Where the two trade off, Rubricon takes the conservative side, and simulation tests assert that a 95% interval really covers 95% of the time.
• It says when it cannot answer “Not computable”, “underpowered”, and “this scorer returned the same value on every sample” are all real outputs. A metric that never varies cannot detect a regression, and silence about that would be worse than saying it.
• It costs what it says it will --plan prices a run before it spends anything, and an unpriced model is reported as unknown rather than as free. Responses are cached, so labelling and calibrating the same samples costs money once.
• It never assumes your judge model The model is resolved per judge, then per suite, then per project, then from the default. --model overrides all four. Nothing about a rubric is welded to a vendor.
• It is local-first and standalone No account, no cloud, no telemetry. rubricon.stats imports numpy and scipy and nothing else, not even the rest of Rubricon, so the estimators work on any array of scores and labels, from any tool.

The default judge model was chosen by measurement, not by vibes.

Four candidates, the same forty responses, each calibrated against the known truth of 0.600. examples/support_bot/bakeoff.py is the script that produced this table, and it is the same script you would point at your own data.

Every one of them covers the truth once corrected, and every raw score overstates it by 15 to 20 points in the same direction. That is the argument in one table: correction is what makes a cheap judge usable, not a better model. The corrected estimates barely separate the candidates; kappa does, and two of the four land below the 0.60 line at which this tool's own report tells you to stop reading. One honest caveat: 0.753 against 0.589 on 24 labels is about one standard error, so the ranking is not statistically established. What is not a question of significance is that one point estimate sits below the usability threshold and the other does not.

Two wrong answers, two different teams.

A retrieval miss and a hallucination look identical in the output and need completely different fixes. One number cannot tell them apart. Component metrics can, and the rows below are a real run over a 12-document corpus, not an illustration.

Read recall first. On the retrieval_miss rows the generator was never given material that could support an answer, so “improve the prompt” is the wrong conclusion to draw from them. And notice the middle two rows: a hallucination (right question, false facts) and an off-topic answer (true facts, wrong question) are each caught by exactly one judge and invisible to the other. That is the whole argument for having both.

What is in the box today.

Everything below works end to end against real models. Interfaces may still move: this is pre-alpha, and the roadmap says what it is not yet.

Statistics

Wilson and BCa bootstrap intervals, Rogan-Gladen with a delta-method CI, PPI++, verdict-stratified PPV/NPV via Zou & Donner MOVER, Cohen's kappa and Krippendorff's alpha, pass@k / pass^k, power and MDE, four bias probes, variance decomposition, drift tests, Benjamini-Hochberg and Holm-Bonferroni. Depends on numpy and scipy only.

Judges

YAML rubrics with content-hash versioning, a structured-output ladder that degrades from json_schema to tool calls to prompting, any model family through OpenRouter, response caching, and a cost estimate before you spend anything.

Code scorers

Seventeen exact and free scorers: match, regex, JSON schema, latency, cost, and six retrieval metrics for RAG. No API key required for any of them.

Calibration

rubricon align: the receipt above, with automatic estimator selection, verdict-stratified annotation, and target-output caching so calibrating the same samples never re-runs your pipeline.

Experiments

rubricon experiment: paired bootstrap and sign-flip permutation test, a minimum detectable effect beside every delta, and slice analysis corrected for multiplicity so twenty slices do not manufacture one “significant” finding.

CI gates

--gate on both eval and experiment. A gate fires only on a statistically significant regression, because a gate that blocks on noise is a gate that gets switched off. A skipped gate does not pass.

Storage

SQLite by default, Postgres-compatible DDL, immutable judge and dataset versions. Every run, score, judge version and per-sample verdict is recorded.

Dashboard

rubricon serve: run history, per-scorer breakdown, per-sample verdicts, judge versions. Server-rendered, with a JSON twin of every page.

Calibrate first. Everything downstream is anchored to that number.

Code scorers need no API key at all, so the first three commands cost nothing. Judges route through OpenRouter; put the key in the environment or a .env file.

# scaffold a project
pip install "rubricon-eval[all]"
rubricon init my-eval

# what will this cost? spends nothing
rubricon eval suites/example.yaml --plan
rubricon eval suites/example.yaml

# label a sample, stratified by the judge's verdict
rubricon annotate suites/example.yaml --judge helpfulness

# the calibration receipt
rubricon align suites/example.yaml --judge helpfulness \
    --labels labels/labels.jsonl

# A/B two versions, paired, with a minimum detectable effect
rubricon experiment base.yaml candidate.yaml --gate

# local dashboard on :6006
rubricon serve

from rubricon.stats import (
    ConfusionMatrix, stratified_prevalence,
)

cm = ConfusionMatrix.from_labels(
    human_labels, judge_labels,
)

# the honest number, not the raw one
print(stratified_prevalence(
    0.75,                        # raw
    cm.precision().value,        # PPV
    cm.npv().value,
    n_test=1000,
    n_labelled_positive=150,
    n_labelled_negative=150,
))

# .github/workflows/eval.yml
- name: Evaluate the candidate
  run: |
    rubricon experiment \
      suites/main.yaml \
      suites/candidate.yaml \
      --gate
  env:
    OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}

# exit 1 only on a significant
# regression; noise does not
# block the merge

An interval that is too narrow passes every ordinary test.

It returns plausible numbers, matches every hand-computed answer, and quietly makes every decision built on it wrong. The only thing that catches it is generating data from a known ground truth thousands of times and counting how often the interval actually contains it.

That bug was found by the simulation suite, in exactly the regime the estimator was written for, after it had passed every known-answer test. ISSUES.md records what is still open, what was found, and, where an investigation refuted its own hypothesis, that too.

Shipped, and what comes next.

The wedge shipped first and complete: if the calibration report alone does not make people say “nobody else prints this”, nothing downstream would save it.

• v0.1Calibration: align The statistics core, judges, rubric versioning, the receipt.
• v0.2Evaluation, experiments, CI gates Paired A/B with an MDE, slice analysis, threshold and regression gates.
• v0.3Local dashboard Run history, per-sample verdicts, a JSON twin of every page.
• v0.4Trace ingest, error analysis, dataset foundry Turn production traces into labelled datasets, and cluster the failures.
• v0.5Agent and trajectory evaluation Scoring the path, not only the final answer.
• v0.6Red team Adversarial suites with the same statistical treatment as everything else.
• v0.7Online evals and drift Sequential, always-valid testing for numbers that are watched as they arrive.
• v1.0Governance and hardening Evidence packs, reproducible reports, audit trails.

Questions worth answering before you install it.