Metadata-Version: 2.5
Name: kunko-judge-audit
Version: 0.6.3
Summary: Independent calibration audits for AI judges: is the confidence honest, what can you automate, what does it cost.
Project-URL: Homepage, https://github.com/kunko-ai-labs/judge-audit
Project-URL: Repository, https://github.com/kunko-ai-labs/judge-audit
Project-URL: Documentation, https://github.com/kunko-ai-labs/judge-audit/tree/main/docs
Project-URL: Changelog, https://github.com/kunko-ai-labs/judge-audit/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/kunko-ai-labs/judge-audit/issues
Project-URL: Security, https://github.com/kunko-ai-labs/judge-audit/blob/main/SECURITY.md
Author: Kunko AI Labs
License: Apache-2.0
License-File: LICENSE
Keywords: ai-act,ai-judge,calibration,ece,evaluation,governance,jev,llm-as-judge,selective-prediction,shadow-mode,typesafe
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Provides-Extra: anthropic
Requires-Dist: anthropic>=1.0; extra == 'anthropic'
Provides-Extra: charts
Requires-Dist: matplotlib>=3.7; extra == 'charts'
Provides-Extra: clef
Requires-Dist: huggingface-hub; extra == 'clef'
Requires-Dist: pillow; extra == 'clef'
Requires-Dist: torch>=2.2; extra == 'clef'
Requires-Dist: transformers>=5.10; extra == 'clef'
Provides-Extra: decider
Requires-Dist: huggingface-hub; extra == 'decider'
Requires-Dist: torch>=2.2; extra == 'decider'
Requires-Dist: transformers>=5.17; extra == 'decider'
Provides-Extra: decision2
Requires-Dist: safetensors; extra == 'decision2'
Requires-Dist: tokenizers; extra == 'decision2'
Requires-Dist: torch>=2.2; extra == 'decision2'
Requires-Dist: transformers>=5.17; extra == 'decision2'
Provides-Extra: dev
Requires-Dist: matplotlib>=3.7; extra == 'dev'
Requires-Dist: mcp>=2.0; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: pyyaml>=6.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: laya
Requires-Dist: laya<0.4,>=0.3.20; extra == 'laya'
Provides-Extra: mcp
Requires-Dist: mcp>=2.0; extra == 'mcp'
Provides-Extra: mlx
Requires-Dist: mlx-lm>=0.31; extra == 'mlx'
Requires-Dist: mlx-lm[cpu]>=0.31; (platform_system == 'Linux') and extra == 'mlx'
Provides-Extra: nli
Requires-Dist: torch>=2.2; extra == 'nli'
Requires-Dist: transformers>=4.40; extra == 'nli'
Provides-Extra: strands
Requires-Dist: strands-decider==0.1.0; extra == 'strands'
Description-Content-Type: text/markdown

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/kunko-ai-labs/judge-audit/v0.6.3/docs/assets/logo-dark.png">
  <img src="https://raw.githubusercontent.com/kunko-ai-labs/judge-audit/v0.6.3/docs/assets/logo.png" alt="judge-audit — calibration audits for AI judges" width="720">
</picture>

[![CI](https://github.com/kunko-ai-labs/judge-audit/actions/workflows/ci.yml/badge.svg)](https://github.com/kunko-ai-labs/judge-audit/actions/workflows/ci.yml) [![Release](https://github.com/kunko-ai-labs/judge-audit/actions/workflows/release.yml/badge.svg)](https://github.com/kunko-ai-labs/judge-audit/actions/workflows/release.yml) [![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/LICENSE) [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/pyproject.toml)

## Turn AI confidence into a defensible decision boundary

**Before an AI judge acts alone, show with evidence where autonomy is earned.** judge-audit replays decisions your people already made and tells you which share an AI judge can take over at the error you accept, with the confidence threshold to deploy, a held-out check and the bound attached (approximate, for traffic like the decisions you labelled). Open source, runs in CI, no LLM inside the measurement, and every published number recomputes from committed evidence.

**What we contribute.**
- **An independent audit, not a model.** judge-audit measures the confidence an AI judge declares against decisions people already made. It judges nothing itself.
- **Evidence anyone can recompute.** Every published figure regenerates from committed checkpoints, and CI fails when a report or this README drifts from its evidence (`scripts/verify_published.py`, `scripts/verify_readme.py`).
- **Pre-registered studies, separate reviews.** The v0.5 and v0.6 studies were pre-registered before their first model call (v0.6: the plan's sha256 committed at the freeze; amendments A1–A4 hashed before the calls or the analysis they govern, A5, an erratum on wording, at publication). Each pull request goes through a separate review and QA pass before it merges, by AI review agents that did not write the change, with the verdicts posted on each pull request. No external human reviewer has read the study plans.
- **Our own security advisories.** We publish the weaknesses we find in our own tools: [GHSA-52xj-j738-jpm3](https://github.com/kunko-ai-labs/judge-audit/security/advisories/GHSA-52xj-j738-jpm3) (judge-audit: an API key in error output, or sent on a redirect) and [GHSA-f7jx-f5h6-pmx7](https://github.com/kunko-ai-labs/agent-assurance/security/advisories/GHSA-f7jx-f5h6-pmx7) (agent-assurance).

![judge-audit in 25 seconds: the question, the audit run, the share each judge can decide alone on 3,080 real banking queries, the CI gate failing below the minimum](https://raw.githubusercontent.com/kunko-ai-labs/judge-audit/v0.6.3/docs/demo.gif)

| | |
|---|---|
| 🎯 **Safe automation rate** | The share of decisions a judge can take alone at the error you accept (1, 2, 5, 10 %), with the confidence threshold to deploy and an out-of-sample check. |
| 🌡️ **Calibration** | When it says 90 %, is it right 90 % of the time? ECE, Brier score, reliability diagram, with intervals. |
| 🚦 **CI gate** | `judge-audit check` fails the build when the judge drifts or the rate falls below your minimum. |
| 🤖 **Where you work** | CLI, GitHub Action with a PR comment, and an MCP server for Claude Code, Cursor or any agent. |

**[Run the quickstart](#quickstart)** · **[Audit your own decisions](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/real-audits.md)** · **[What you get](#what-you-get)** · **[v0.6 findings](#v06-findings)** · **[v0.5 findings](#v05-findings)** · ▶ [launch video](https://github.com/kunko-ai-labs/judge-audit/releases/download/v0.5.1/judge-audit-launch.mp4)


## v0.6 findings

**v0.6 in short.** 12 AI judges, the same 2,974 items of MASSIVE 1.1 in English, Spanish and Catalan, pre-registered before the first model call ([plan](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/v06-plan.md), sha256 committed at the freeze; amendments A1–A5). Six pre-registered transfer tests asked whether a threshold validated (approximate) on English rows at a 5 % error target lets more than 5 % errors through on the same items in Spanish or Catalan: none was resolved at its Holm-corrected level, so no breach was shown; that is not a pass: the approximate one-sided 95 % upper bounds on the pooled target-language error run from 3.30 % (T6) to 8.65 % (T2), above 5 % in 5 of 6 tests, and 3 of the 6 came out as predicted. Demonstrated on this sample, and only on it (MASSIVE is translated public data, probably seen in pretraining): declared confidence does not always match accuracy; mean confidence minus accuracy is +0.080 to +0.085 for jev (Jev 1.13.0), −0.052 to −0.067 for gpt-6-luna and +0.063 to +0.099 for claude-sonnet-4.5 across the three languages. The differences between languages are 95 % intervals with no correction for multiplicity, so they are not called demonstrated ([§2](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/v06/README.md#2-overconfidence-paired-across-locales)). The number to act on is the same audit on your own decisions.

**Jev version.** Every Jev figure of v0.6 was measured with Jev 1.13.0, the model the replies say they served. The alias `jev-latest` has pointed to `jev-1.14.0` since 2026-10-22, so a new run under the alias measures a different model and its figures are not comparable with these. Pin the version with `JEV_MODEL` (for example `JEV_MODEL=jev-1.13.0`; [how](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/real-audits.md#pin-the-model-version)).

**At a glance: v0.6, MASSIVE 1.1, 2,974 translated public items in English, Spanish and Catalan (not your traffic), 12 judges: the held-out share automated by thresholds chosen at a 5 % error target (the threshold chosen on one half of the items, applied to the other), with its approximate bound; met when that bound is within 5 %, at the study's split (seed 2026).**

*Met is per split.* gpt-6-luna en-US is met at the study's split (seed 2026), and its share moves over split seeds 2027–2036 (21.8–39.4 %); on another split a met cell can miss the target. The bounds of the other splits are not computed here.

| Judge | en-US | es-ES | ca-ES |
|---|---|---|---|
| jev (Jev 1.13.0) | 29.0 % (bound 5.8 %, not met; splits 30.7–55.6 %) | 25.3 % (bound 7.4 %, not met; splits 0.0–23.6 %) | none (splits 0.0–19.5 %) |
| gpt-6-luna | 18.2 % (bound 4.6 %, met; splits 21.8–39.4 %) | none (splits 0.0–13.3 %) | 5.3 % (bound 9.8 %, not met; splits 0.0–0.0 %) |
| claude-sonnet-4.5 | 21.5 % (bound 2.9 %, met; splits 21.5–21.5 %) | 8.5 % (bound 5.7 %, not met; splits 0.0–8.8 %) | none (splits 0.0–6.9 %) |
| clef-flash (exploratory) | 40.6 % (bound 5.0 %, met; splits 0.0–44.4 %) | 38.5 % (bound 5.7 %, not met; splits 4.0–44.2 %) | 39.0 % (bound 4.2 %, met; splits 18.0–60.0 %) |
| decider-2b (exploratory) | 31.1 % (bound 7.7 %, not met; splits 0.0–9.7 %) | 10.4 % (bound 7.4 %, not met; splits 0.0–4.5 %) | none (splits 0.0–0.0 %) |
| decision2-kai-0.6b (exploratory) | 4.1 % (bound 7.3 %, not met; splits 0.0–7.4 %) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) |
| deepseek-r1 (exploratory) | none (splits 0.0–28.7 %) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) |
| gemini-3.6-flash (exploratory) | 55.1 % (bound 3.4 %, met; splits 55.1–55.1 %) | 26.6 % (bound 5.3 %, not met; splits 26.0–52.8 %) | 27.1 % (bound 5.1 %, not met; splits 44.6–53.6 %) |
| laya-multilingual (exploratory) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) |
| microsoft-decision-1 (exploratory) | 80.7 % (bound 4.6 %, met; splits 80.2–82.7 %) | 72.3 % (bound 4.6 %, met; splits 40.5–73.7 %) | 30.1 % (bound 4.9 %, met; splits 31.5–60.6 %) |
| strands-decider-2b (exploratory) | 3.9 % (bound 14.4 %, not met; splits 0.0–0.0 %) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) |
| unsloth-decision-4b (exploratory) | none (splits 0.0–4.8 %) | none (splits 0.0–0.0 %) | none (splits 0.0–0.0 %) |

*Bound:* the one-sided 95 % upper bound on the held-out errors of both halves pooled, rounded up; approximate, since each half's threshold depends on the other half; computed after the study from its published counts, not pre-registered (#213). *Met:* that bound is within 5 %; only a share that meets it is called validated. Each met cell is a separate statement, each at 95 %, approximate; read together they do not all hold at 95 %, so no claim is made about all of them at once, and none should be picked as the best. *None:* no threshold was validated on either half at seed 2026. *Splits:* the same share over split seeds 2027–2036; that range need not contain the seed-2026 share, and a share far outside it rests on one split. Labels are carried over by localisation, and no share says anything about future traffic; judges marked exploratory are in no pre-registered test. No ranking: rows follow the study's roster. The shares, their ranges, every other figure with its intervals and the cost of each run are in the **[v0.6 findings in full](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/v06/README.md)**; the bounds are computed here from their counts.

## v0.5 findings

**In short.** Asking Qwen3-8B and gemini-3.6-flash several times and counting the votes ranked their errors worse than their own verbalized number in 3 of 4 tests under the pre-registered rule (4 of 4 under the re-reading), and one verdict (H2 for gemini-3.6-flash) changes with how an answer that copies an option's description is scored. On BANKING77 at a 5 % error target, validated on held-out texts (approximate), Jev native probability can decide 41.0 % (27.0 %–45.1 % over split seeds 2027–2036) of the texts alone, gemini-3.6-flash verbalized 14.3 % (7.6 %–30.0 %), Qwen3-8B token log-probability none (0.0 %–4.4 %). Whether the bound is within the target depends on the split: at split seed 2026 Jev's 41.0 % has a bound of 3.9 %, within 5 %; on another split of the same checkpoints (split seeds 2033 and 2036) the same judge at the same 5 % target automates fewer texts with a bound above 5 %, not met; the range over split seeds shows how far the share moves. *Caveats:* public datasets, probably seen in pretraining; one run each, except Jev and Qwen3-8B verbalized on BANKING77 (3 runs); label noise not measured; the held-out slice not run (#106). The number to act on is the same audit on your own decisions.

**At a glance: BANKING77, 3,080 human-labelled banking queries: the share each judge decides alone at a 5 % error target, validated on held-out texts (approximate) at split seed 2026; another split can change the result.**

| Judge and confidence | Validated share at a 5 % error target (approximate) | 95 % upper bound on the error rate | Range over split seeds | At a 10 % error target |
|---|---|---|---|---|
| Jev 1.13.0, native probability | **41.0 %** | 3.9 % | 27.0–45.1 % | 72.3 % (bound 9.2 %) |
| gemini-3.6-flash, verbalized | **14.3 %** | 2.7 % | 7.6–30.0 % | 45.5 % (bound 5.3 %) |
| Qwen3-8B, token log-probability | **none** | — | 0.0–4.4 % | 12.5 % (bound 7.2 %) |

*Validated:* the threshold is chosen on one half of the texts and applied to the other, both ways, and the 95 % upper bound on the held-out error rate is within the target. The bound is the one-sided Clopper–Pearson bound on the held-out errors of both halves pooled, rounded up; it treats the two held-out halves as independent draws, an approximation since each half's threshold depends on the other half. A share observed on all rows without a bound is not a basis for deployment.

**Jev version (v0.5).** The Jev rows of v0.5 were also measured with Jev 1.13.0: every one of the 3,080 BANKING77 replies in each of the three Jev runs says it was served by `jev-1.13.0`. Figures of another Jev version are not comparable with them.

Teams are shipping judgment models — TypeSafe's Jev, LLM-as-judge, guardrails, routers — that return a confidence with each decision. The literature studies calibration (Guo et al. 2017; Shao 2026; Huang et al. 2026); what a team needs before automating is the same measurement on *its own* decisions. judge-audit runs any judge in **shadow mode** against decisions your humans already made and answers the four questions that matter before you automate:

| Question | Metric | Why a buyer cares |
|---|---|---|
| When it says 80 % confident, is it right 80 % of the time? | ECE, reliability diagram | A confident-and-wrong judge automates its own mistakes |
| What share of the work can it decide alone at a bounded error? | safe automation rate, accuracy-coverage curve | The ROI number — with its error bound, the threshold to deploy, and the labels behind it |
| What does it really cost, and how bad is the latency tail? | $ per decision, p50 / p99 | The demo is cheap; the tail is what pages you |
| Has it drifted since last week? | `judge-audit check` CI gate | Vendors update models without telling you |

The study's design, the eight confirmatory tests and every caveat: **[v0.5 findings in full](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/results/v0.5.md)**.

## Quickstart

```bash
pip install kunko-judge-audit           # every release ships Sigstore-signed build provenance
git clone --branch v0.6.3 https://github.com/kunko-ai-labs/judge-audit && cd judge-audit   # the example data lives in the repo, not in the pip package
judge-audit run examples/email-routing/labels.jsonl --judge simulated --target 0.10   # no API key needed
judge-audit check examples/email-routing/labels.jsonl --judge simulated --baseline audit-result.json --target 0.10 --min-safe-rate 0.10:0.60
```

The start of the `run` line, **SIMULATED**:

```text
SIMULATED — not a real vendor audit · judge=simulated n=200 accuracy=85.5% [75.1, 93.7] safe_automation@10%=not_met(75.0% bound_95_heldout_approx=16.6%) all_rows=65.5% …
```

`simulated` is a seeded simulator so you can see the whole pipeline in ten seconds; every report it touches is stamped **SIMULATED**, and its figures say nothing about any real judge. `safe_automation@10%` leads with the held-out check at a 10 % error target: `not_met`, because the one-sided 95 % upper bound on the held-out error rate (16.6 %, approximate: it pools the two held-out halves) is above the target, so the 75.0 % held out is not validated. A share is shown as validated only when that bound is within the target. `all_rows` (65.5 %) is the share the threshold to deploy covers on every labelled decision, the share `check --min-safe-rate` gates. At the default target, 5 %, the simulator reaches no threshold and the line reads `safe_automation@5%=none`. `check --min-safe-rate 0.10:0.60` exits 0 here and exits 1 if that share falls below 60 %. To audit a real vendor, see [docs/real-audits.md](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/real-audits.md).

In the [GitHub Action](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/integrations.md#github-action) the same controls are inputs: `target` (the error rate the safe automation rate is headlined at, default `0.05`), `segment-by` (where to look for the worst segment: `label`, `meta.FIELD` or `none`) and `min-safe-rate` (`RISK:SHARE` minimums that fail the job). It returns `safe-automation-rate` (at the target) and `safe-automation-rates` (every target) as outputs.

## What you get

<img src="https://raw.githubusercontent.com/kunko-ai-labs/judge-audit/v0.6.3/docs/assets/report-preview.png" alt="The HTML report: key figures (safe automation rate, accuracy, calibration error, cost and latency), data and ground truth, then the answer to Can I automate this?" width="640">

One self-contained report per run (Markdown or HTML, prints to A4): the answer first: the share validated on held-out decisions, with its approximate bound and its range over split seeds, then the threshold to deploy; calibration with intervals; accuracy against coverage; cost and latency; the provenance of every number. Shown here on the seeded simulator, stamped **SIMULATED**. `audit-result.json` carries the same numbers for your pipeline.

## How it works

1. **Labeled dataset** — JSONL rows `{state, questions, labels}`. Your humans already decided; the judge must reproduce them. An optional first line declares where the labels come from — their [ground-truth tier](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/ground-truth.md), from `GT-1 constructed` to `GT-6 production outcome`, and [what each tier lets you claim](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/ground-truth.md#what-each-tier-lets-you-claim).
2. **Shadow run** — the judge scores every row. Nothing is automated; everything is recorded: decision, confidence, latency, cost, raw response.
3. **Audit report** — calibration (ECE with equal-width and with equal-mass bins, Brier score, reliability bins — three numbers, plus log loss next to Brier, never combined), selective prediction (accuracy at every coverage level), cost, latency percentiles, plus provenance: model, backend, timestamp, dataset hash. Accuracy, the three calibration numbers and zero-error coverage carry a 95 % bootstrap interval over the dataset's distinct texts (Jev on the bare-label router: `66.7% [52.5, 80.3]`; 2,000 seeded resamples, pure Python, `--no-ci` to skip), or the exact binomial interval where the bootstrap cannot move so a number on 120 rows reads as the noisy estimate it is. Every report header carries a `Ground truth: GT-1 constructed — …` line with the tier's meaning and the dataset's caveats, so two accuracies are never read as the same evidence; a dataset that declares nothing is reported as `GT-0 unknown`, never silently.
4. **CI gate** — `judge-audit check --baseline baseline.json --max-ece-drift 0.02` fails the build when the judge degrades.

Exit codes: `0` ok · `1` drift detected · `2` usage or configuration error, or a judge that could not answer (an HTTP error, an expired key, a server that cannot be reached; one line, `judge request failed: …`, no traceback). The message says what to fix. An output that cannot be written exits `2` even when drift was also found: fix it before reading the drift.

**In CI:** the [GitHub Action](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/integrations.md#github-action) runs the audit on every push or pull request and fails the build on drift:

```yaml
- uses: kunko-ai-labs/judge-audit@v0.6      # or pin the release's commit SHA
  with: { labels: audits/labels.jsonl, judge: jev, baseline: audits/baseline.json,
          target: "0.05", segment-by: label, min-safe-rate: "0.05:0.40" }
  env: { AI_GATEWAY_API_KEY: "${{ secrets.AI_GATEWAY_API_KEY }}" }
```

One sticky PR comment that opens with the safe automation rate validated on held-out decisions (approximate), then the audit table (n, accuracy, ECE, cost, p99, verdict vs baseline) and, labelled as observed on all rows without a bound, the zero-error coverage; the report in the job summary, the report and result JSON as an artifact. The per-decision judgments stay on the runner unless you ask for them (`upload-evidence: "true"`). `min-safe-rate: "0.05:0.40"` fails the job unless at least 40 % of the labelled decisions is automated at a 5 % error target (inputs in the [Quickstart](#quickstart)). Outputs `accuracy`, `ece`, `zero-error-coverage`, `safe-automation-rate` (at the target), `safe-automation-rates` (every target), `drift` for anything downstream. Details in [docs/integrations.md](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/integrations.md#github-action).

**From inside an agent:** `pip install "kunko-judge-audit[mcp]"` then `claude mcp add judge-audit -- judge-audit-mcp` (or the equivalent in Cursor). The agent gets `run_audit`, `check_drift` and `list_judges` and can audit the judge it is about to rely on without leaving the session. See [docs/integrations.md](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/integrations.md).

## Judge interface

Anything that maps `(state, questions) -> (decision, confidence)` plugs in:

```python
from judge_audit import Judge, Question, Judgment

class MyJudge(Judge):
    name = "my-judge"

    def decide(self, state: str, questions: list[Question]) -> list[Judgment]:
        ...  # return Judgment(question=q.name, decision="spam", confidence=0.93)
```

Ships with fourteen: `jev` (TypeSafe Jev — and, via `JEV_ENDPOINT`, any Jev-compatible server such as OpenJev), `llm` (any chat model with a confidence prompt: Claude through the official SDK, anything OpenAI-compatible — OpenAI, Gemini, Ollama, vLLM — or your own transport), `nli` (a local zero-shot encoder, the control), `finetuned` (your own classifier, trained on your labels with `scripts/train_classifier.py`), `laya` (Laya, an open-weight judgment model run locally), `decision2` (Decision 2.0, an open decision model run locally at a pinned revision), `decider` (decider, likewise), `strands` (Strands Decider, likewise), `clef` (Clef and Clef-flash, likewise), `openai-decisions` (the OpenAI Decisions API, hosted), `clef-hosted` (Clef and Clef-flash on Workers AI, hosted), `microsoft-decision` (Microsoft-Decision-1 through OpenRouter, hosted), `logprob` (an open model's own probability of each option, run locally with MLX) and `simulated`. Details in [docs/judges.md](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/judges.md).

## Learn more

| | |
|---|---|
| **Results by version** | [v0.5: pre-registered study on real data](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/results/v0.5.md) · [v0.4: emails under attack, Jev audits, the Arena, the consensus panel](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/results/v0.4.md) |
| **How the metrics work** | [docs/judges.md](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/judges.md): calibration, the safe automation rate, intervals, what each number can and cannot say |
| **Integrations** | [GitHub Action, MCP server, Python API](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/integrations.md) · [auditing a real vendor](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/real-audits.md) |
| **Roadmap and changes** | [ROADMAP](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/ROADMAP.md) · [CHANGELOG](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/CHANGELOG.md) · [releasing and supply chain](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/RELEASING.md) |
| **Contributing** | [CONTRIBUTING](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/CONTRIBUTING.md): what helps most and how review works |

**Why calibration, not accuracy?** Accuracy tells you who wins a benchmark. Calibration tells you what you can automate safely. A judge that is confident on the decisions it gets wrong automates its own mistakes; one that flags its own doubt hands those to people. **Audit the honesty, not the score.**

## Prior art

The statistical method here is not new, and this project does not claim it. Choosing a confidence threshold so that the error among the automated decisions stays within a stated tolerance, with a stated probability, is selective prediction and risk control:

- **Calibration:** Guo, Pleiss, Sun & Weinberger, *On Calibration of Modern Neural Networks*, ICML 2017 ([arXiv:1706.04599](https://arxiv.org/abs/1706.04599)).
- **Selective prediction:** Geifman & El-Yaniv, *Selective Classification for Deep Neural Networks*, NeurIPS 2017 ([arXiv:1705.08500](https://arxiv.org/abs/1705.08500)): a threshold whose risk bound holds with high probability.
- **Distribution-free risk control:** Angelopoulos, Bates, Candès, Jordan & Lei, *Learn then Test*, Annals of Applied Statistics 2025 ([arXiv:2110.01052](https://arxiv.org/abs/2110.01052)); Angelopoulos, Bates, Fisch, Lei & Schuster, *Conformal Risk Control*, ICLR 2024 ([arXiv:2208.02814](https://arxiv.org/abs/2208.02814)); Angelopoulos & Bates, *A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification*, 2021 ([arXiv:2107.07511](https://arxiv.org/abs/2107.07511)).
- **The same idea for AI judges:** Jung, Brahman & Choi, *Trust or Escalate* (LLM judges and agreement with humans), ICLR 2025 ([arXiv:2407.18370](https://arxiv.org/abs/2407.18370)): automate when the judge is confident, escalate otherwise.
- **Confidence of language models:** Kadavath et al., *Language Models (Mostly) Know What They Know*, 2022 ([arXiv:2207.05221](https://arxiv.org/abs/2207.05221)); Tian et al., *Just Ask for Calibration*, EMNLP 2023 ([arXiv:2305.14975](https://arxiv.org/abs/2305.14975)); Xiong et al., *Can LLMs Express Their Uncertainty?*, ICLR 2024 ([arXiv:2306.13063](https://arxiv.org/abs/2306.13063)).
- **LLM-as-judge:** Zheng et al., *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*, 2023 ([arXiv:2306.05685](https://arxiv.org/abs/2306.05685)).

What this tool adds is not the statistics: it is the audit on your own decisions, the evidence trail (every figure recomputes from a committed checkpoint) and the reproducibility (pinned model versions, recorded provenance, a published protocol). The full list, with what each source was used for, is the [reading list](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/v05/reading-list.md); the metrics are described in [docs/judges.md](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/docs/judges.md#related-work).

## FAQ

**Is this a benchmark?** No. A benchmark ranks judges on a fixed test. An audit checks one judge on *your* decisions and tells you what you can automate. The datasets here are worked examples, not a leaderboard — yet.

**Why was Jev audited first?** Its vendor sells calibrated confidence as the headline feature. A claim that specific deserves an independent check.

**Can I trust a 100 % result?** Only as far as the dataset. The clean-email audit says the judge handles templated business email; it says nothing about your inbox. That is why the adversarial and routing audits exist.

**Does it send my data anywhere?** Only to the judge endpoint you configure. Reports and checkpoints are local files; commit them or not. In CI the Action uploads the report and the metrics JSON as a run artifact; the per-decision judgments are uploaded only if you set `upload-evidence: "true"`.

## License

Apache-2.0 — see [LICENSE](https://github.com/kunko-ai-labs/judge-audit/blob/v0.6.3/LICENSE). Sister project: [agent-assurance](https://github.com/kunko-ai-labs/agent-assurance) (deterministic checks on what an agent *may* do; this repo is the probabilistic half: whether its judgment can be trusted).
