Metadata-Version: 2.4
Name: vernier
Version: 0.3.0
Classifier: Development Status :: 1 - Planning
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Dist: numpy>=2.0
Requires-Dist: pyarrow>=15
Requires-Dist: rfdetr==1.6.5.post0 ; extra == 'real-models'
Requires-Dist: platformdirs>=4 ; extra == 'real-models'
Requires-Dist: torch>=2.4 ; extra == 'real-models'
Requires-Dist: transformers>=5.1 ; extra == 'real-models'
Requires-Dist: huggingface-hub>=0.27 ; extra == 'real-models'
Requires-Dist: pillow>=10 ; extra == 'real-models'
Requires-Dist: timm>=1.0 ; extra == 'real-models'
Requires-Dist: polars>=1.0 ; extra == 'tables'
Requires-Dist: torch>=2.4 ; extra == 'torch'
Requires-Dist: plotly>=6.0 ; extra == 'viz'
Provides-Extra: real-models
Provides-Extra: tables
Provides-Extra: torch
Provides-Extra: viz
License-File: LICENSE-APACHE
License-File: LICENSE-MIT
Summary: High-performance, parity-preserving COCO-style evaluation
Keywords: evaluation,metrics,computer-vision,detection,coco,object-detection
Author: The vernier authors
License: MIT OR Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Documentation, https://github.com/NoeFontana/vernier#readme
Project-URL: Homepage, https://github.com/NoeFontana/vernier
Project-URL: Issues, https://github.com/NoeFontana/vernier/issues
Project-URL: Repository, https://github.com/NoeFontana/vernier

# vernier

[![CI](https://github.com/NoeFontana/vernier/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/NoeFontana/vernier/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/vernier.svg?label=pypi%20%7C%20vernier)](https://pypi.org/project/vernier/)
[![crates.io vernier](https://img.shields.io/crates/v/vernier.svg?label=crates.io%20%7C%20vernier)](https://crates.io/crates/vernier)
[![crates.io vernier-core](https://img.shields.io/crates/v/vernier-core.svg?label=crates.io%20%7C%20vernier-core)](https://crates.io/crates/vernier-core)
[![crates.io vernier-mask](https://img.shields.io/crates/v/vernier-mask.svg?label=crates.io%20%7C%20vernier-mask)](https://crates.io/crates/vernier-mask)
[![crates.io vernier-cli](https://img.shields.io/crates/v/vernier-cli.svg?label=crates.io%20%7C%20vernier-cli)](https://crates.io/crates/vernier-cli)
[![License: MIT OR Apache-2.0](https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue.svg)](#license)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NoeFontana/vernier/blob/main/docs/tutorials/notebooks/colab_smoke.ipynb)

Fast, parity-preserving evaluation for object detection, instance / panoptic / semantic segmentation, boundary IoU, OKS keypoints, LVIS federated, LRP / oLRP error decomposition, and detection-family calibration (ECE / MCE / reliability). Rust core, Python frontend, optional CLI.

## 60-second example

Post training, if your predictions are already serialized to JSON (CI gate, post-training inspection):

```python
from pathlib import Path
from vernier.instance import Bbox, CocoDataset, Evaluator

gt_bytes = Path("instances_val2017.json").read_bytes()
dt_bytes = Path("detections.json").read_bytes()

dataset = CocoDataset.from_json(gt_bytes)
summary = Evaluator(iou=Bbox()).evaluate(dataset, dt_bytes)
for line in summary.pretty_lines():
    print(line)
```

In a training loop, vernier supports overlapping eval with the data loading and inference. The matching kernel runs on a worker thread, so `submit(...)` returns immediately and the main thread keeps moving. Passing a `CocoDataset` reuses the parsed-once GT and its per-kernel derivation cache across every epoch (ADR-0020). On a dedicated validation pass (no trainer competing for cores), pass `num_threads=N` to parallelise the matching kernel inside the worker (ADR-0047):

```python
from pathlib import Path
from vernier.instance import Bbox, CocoDataset, Evaluator

gt = CocoDataset.from_json(Path("instances_val2017.json").read_bytes())
evaluator = Evaluator(iou=Bbox())
with evaluator.background(gt, num_threads=8) as bg:  # default: single core
    for images, targets in val_loader:
        # torchvision detection API shape: list[dict] of length batch_size,
        # each with "boxes" (N,4 xywh), "scores" (N,), "labels" (N,) as
        # torch.Tensor. vernier consumes any DLPack-producing array library
        # (torch, jax, cupy, numpy) zero-copy.
        predictions = model(images)
        bg.submit([
            {"image_id": int(t["image_id"]), **p}
            for t, p in zip(targets, predictions)
        ])
    summary = bg.finalize()
print("AP =", summary.stats[0])
```

Both end in the same 12-line `pycocotools`-shaped Summary;
[`docs/tutorials/first-evaluation.md`](docs/tutorials/first-evaluation.md)
walks each end-to-end.

## Benchmarks

<!-- The headline table and pinned-baselines block below are hand-mirrored from
     docs/benchmarks.md (which is auto-generated by tools/render_benchmarks.py).
     After a fresh bench round, refresh both: re-run the renderer, then update
     the numbers + version pins here. -->

| Workload | vernier median | Speedup vs alternatives |
| --- | ---: | --- |
| Instance — bbox AP (val2017) | 354 ms | **1.6×** hotcoco · **4.7×** faster-coco-eval · **16.0×** pycocotools |
| Instance — segm AP (val2017) | 968 ms | **1.4×** hotcoco · **3.5×** faster-coco-eval · **6.7×** pycocotools |
| Instance — boundary AP (val2017) | 3.2 s | **16.7×** faster-coco-eval · **19.5×** boundary-iou-api |
| Instance — keypoints AP (val2017, OKS) | 136 ms | **1.6×** hotcoco · **5.7×** faster-coco-eval · **16.9×** pycocotools |
| Panoptic — PQ (val2017) | 10.6 s | **3.3×** panopticapi |
| Semantic — mIoU (val2017) | 2.9 s | **14.0×** mmsegmentation |
| Instance — LVIS bbox AP (v1 val, perfect-DT) | 2.6 s | **1.4×** hotcoco · **73.1×** lvis-api · 10× lower peak RSS (1.45 GiB vs 15.01 GiB) |
| Instance — bbox AP (Objects365 val, 1.06M dets) <sup>†</sup> | 8.8 s | **1.7×** hotcoco · **41.9×** pycocotools · faster-coco-eval did not finish (OOM at ~30 GiB) |

**Thread scaling** (`num_threads`, each cell pinned to that many CPUs —
`nt=8` is every library's out-of-the-box configuration on this 8-vCPU host):

| Workload | `nt=1` | `nt=2` | `nt=4` | `nt=8` | vs hotcoco / faster-coco-eval at `nt=8` |
| --- | ---: | ---: | ---: | ---: | --- |
| bbox (val2017) | 354 ms | 267 ms | 229 ms | **226 ms** | **1.6×** hotcoco · **6.5×** faster-coco-eval |
| segm (val2017) | 983 ms | 569 ms | 375 ms | **319 ms** | **1.7×** hotcoco · **11.0×** faster-coco-eval |
| boundary (val2017) | 3.20 s | 1.70 s | 937 ms | **790 ms** | **21.5×** faster-coco-eval |
| keypoints (val2017) | 136 ms | 118 ms | 109 ms | **104 ms** | **1.5×** hotcoco · **7.3×** faster-coco-eval |
| bbox (Objects365) | 8.8 s | — | — | **4.7 s** | **1.7×** hotcoco |

faster-coco-eval ≥1.8 and hotcoco are multi-threaded too, so every column
compares equal CPU budgets. Its parallelism pays off on boundary IoU but
is flat on segm and keypoints, which is why vernier's lead widens with
cores there.

<sup>†</sup> The Objects365 row is harness mode `dev` (one measurement rep per impl, no IQR gate) — pycocotools alone needs ~6 minutes per rep at that size. Every other row is release mode.

Median total-stage wall time on a KVM VPS (AMD EPYC-Milan, 4 cores ×
2 threads = 8 logical CPUs, `x86_64` — not a bare-metal Milan box),
harness mode `release` (N=10 measurement reps + 2 warmup, randomised
impl order, 5% relative-IQR gate per impl), build profile = cargo
release defaults (`opt-level=3`, `lto=thin`, `codegen-units=1`, no
`target-cpu`) — same as the PyPI wheel. **Every impl gets one CPU**:
faster-coco-eval ≥1.8, hotcoco and mmsegmentation are multi-threaded by
default, so each runner is pinned to the cell's CPU budget and each
result records its CPU/wall ratio as evidence (ADR-0049). vernier
scales with `num_threads` — the thread-scaling tables on the benchmarks
page cover 1/2/4/8. Full per-cell breakdown
(including IQRs), peak and eval-delta memory, and methodology in
[`docs/benchmarks.md`](docs/benchmarks.md); per-library comparison of
when to pick which in [`docs/comparison.md`](docs/comparison.md).

**Baselines pinned for these numbers** —
[`pycocotools==2.0.11`](https://pypi.org/project/pycocotools/2.0.11/),
[`hotcoco==1.0.1`](https://pypi.org/project/hotcoco/1.0.1/),
[`faster-coco-eval==1.8.0`](https://pypi.org/project/faster-coco-eval/1.8.0/),
[`panopticapi` @ `7bb4655`](https://github.com/cocodataset/panopticapi/commit/7bb4655548f9),
[`boundary-iou-api` @ `37d2558`](https://github.com/bowenc0221/boundary-iou-api/commit/37d25586a677),
[`mmsegmentation` @ `c685fe6`](https://github.com/open-mmlab/mmsegmentation/commit/c685fe6767c4cadf6b051983ca6208f1b9d1ccb8) (vendored),
[`lvis-api` @ `031ac21`](https://github.com/lvis-dataset/lvis-api/commit/031ac21f939b)
(PyPI `lvis==0.5.3`).
All cells were measured at HEAD `e361050ef582` (machine fingerprint
`59aab88b17f4`). That is a different host from the 2026-05 snapshot's
`37652a58e939`, and the CPU budget above changed what a "single-thread"
cell means, so absolute numbers are not comparable with earlier
snapshots — ratios within this one are. Each baseline is locked in its own uv-managed venv per
[ADR-0017](docs/adr/0017-local-bench-harness.md).

## Install

```bash
pip install vernier                  # Python wheel
cargo add vernier                    # Rust library (all paradigms)
cargo install vernier-cli            # `vernier` CLI binary
```

Wheels ship for linux x86_64 / aarch64 (glibc + musl), macOS
x86_64 / arm64, and windows x64.

On the Rust side, `vernier` is a facade
([ADR-0048](docs/adr/0048-vernier-facade-crate.md)): it re-exports the
paradigm crates under one dependency and one module map —
`vernier::{instance, mask, panoptic, semantic, partial}` — mirroring the
Python namespace. It holds no code of its own, so depending on a leaf
crate directly (`cargo add vernier-core` for bbox / segm / boundary /
keypoints AP alone) is equally supported; the three optional paradigms
can also be trimmed in place with `default-features = false`. Note the
asymmetry: `cargo add vernier` gets the library, `cargo install
vernier-cli` gets the binary — the CLI stays out of the library's
dependency tree so `clap` never lands in a consumer's build.

## Status & validation

Pre-1.0; public API is unstable. See [`docs/adr/`](docs/adr/) for the design decisions shaping it.

`pycocotools==2.0.11` is the de-facto reference for COCO evaluation — slow, unmaintained, and full of edge-case quirks. Faster reimplementations exist, but each silently fixes some quirks and not others, so you discover the divergences empirically. vernier takes a third path:

- **Auditable parity.** Every divergence from pycocotools is filed in the quirks survey under
  [ADR-0002](docs/adr/0002-three-tier-parity-model.md) as either
  `strict` (bit-equal output, even when vernier's implementation is
  structurally different) or `corrected` (opt-in opinionated fix).
  Strict is the default; corrected fixes are itemized so you always
  know when your numbers diverge from a reference run. A drop-in shim
  (`vernier.patch_pycocotools()`) keeps existing pycocotools-based
  scripts working with one line.
- **Rust core, Python frontend.** The matching kernel is pure Rust
  with runtime SIMD dispatch; the FFI layer is data conversion only.
  The CLI ships as a static binary, so CI pipelines call vernier
  without provisioning a Python interpreter.
- **One toolkit instead of five.** bbox / segm / boundary / keypoints
  AP, panoptic PQ, semantic mIoU, LVIS federated, oLRP error
  decomposition, and detection-family calibration all live behind
  one Python API and one CLI — folded over a single matching pass.
  Per-paradigm migration guides under
  [`docs/migrate/`](docs/migrate/) show how to replace `pycocotools`,
  `faster-coco-eval`, `panopticapi`, `lvis-api`, and
  `mmsegmentation` one at a time.
- **Scenario slicing + cross-run aggregation.** A partition manifest
  (`weather`, `time_of_day`, …) feeds `vernier eval --manifest` for
  per-slice headline metrics and `vernier aggregate` for cross-run
  corruption tables (mPC / rPC) — one matching pass, N slices
  ([ADR-0046](docs/adr/0046-slice-and-aggregate.md)).

Per-paradigm parity status:

| Paradigm / metric | Oracle | Parity tier | Open caveat |
| --- | --- | --- | --- |
| Instance bbox / segm / keypoints AP | `pycocotools==2.0.11` | strict bit-equal | none |
| Instance boundary IoU | `boundary-iou-api` | strict bit-equal | none |
| Segm + boundary TIDE thresholds (`t_b`) | none yet | corrected-only | [ADR-0022](docs/adr/0022-tide-thresholds.md) still `proposed`; defaults extrapolated, not measured |
| Panoptic PQ | `panopticapi` (single-core path) | strict bit-equal | none |
| Panoptic boundary PQ | `bowenc0221/boundary-iou-api` (single-core path, same SHA as the instance vendor) | strict bit-equal | [ADR-0025 §Z1/Z2 amendment](docs/adr/0025-panoptic-api.md); Cityscapes panoptic (Z3) deferred |
| Semantic mIoU / FWIoU / pAcc / mAcc | `mmseg.IoUMetric` vendored at v1.2.2 ([ADR-0036](docs/adr/0036-vendor-mmsegmentation-ioumetric.md), still `proposed`); cityscapesScripts + ADE20K cross-impl bench externally blocked | strict bit-equal on the four per-class u64 marginals at val2017 scale | [ADR-0028](docs/adr/0028-sem-seg.md); ADE20K-scale bench gated on license-cleared cache |
| LVIS federated AP | `lvis-api` (vendored at `031ac21f`, ORACLE_LVIS_COMMIT_SHA) | strict bit-equal on the `(T, R, K, A)` precision tensor at full LVIS v1 val | bench paradigm wired; segm cell waits on `evaluate_segm_grid_with_dataset` |
| LRP / oLRP error decomposition (instance bbox / segm / boundary / keypoints) | pure-NumPy oracle ([ADR-0043](docs/adr/0043-lrp-oracle-and-namespace.md)) | strict against the oracle within 1e-9; `kemaloksuz/LRP-Error` tripwire vendored opt-in | panoptic LRP is a typed `NotImplementedError` stub — panoptic predictions carry no per-segment scores (ADR follow-up) |
| Detection-family calibration — ECE / MCE / reliability (instance bbox / segm / boundary / keypoints) | clean-room NumPy oracle ([ADR-0018](docs/adr/0018-calibration.md)) with isolated P1–P10 quirks survey | strict bit-equal against the oracle (16/16 parity tests) | panoptic (Shape 2) and semantic (Shape 3) calibration deferred on data-model prerequisites; Clopper-Pearson CI documented Phase-2 |

Three-tier parity model: [ADR-0002](docs/adr/0002-three-tier-parity-model.md);
per-library comparison: [`docs/comparison.md`](docs/comparison.md).

## Three evaluation paradigms

They have different data models, different matching rules, and
different parity oracles:

- `vernier.instance` — detections with scores → bbox / segm /
  boundary / keypoints AP.
- `vernier.panoptic` — RGB-encoded panoptic PNGs + `segments_info`
  JSON → PQ.
- `vernier.semantic` — single-channel class-id label maps → mIoU /
  FWIoU / pAcc / mAcc.

See [Three paradigms](docs/explanation/three-paradigms.md).

## Documentation

- **Tutorials** — [`docs/tutorials/`](docs/tutorials/)
- **Migration guides** (from pycocotools, faster-coco-eval,
  panopticapi, lvis-api, mmsegmentation) —
  [`docs/migrate/`](docs/migrate/)
- **How-to** —  [`docs/how-to/`](docs/how-to/)
- **Reference** — [`docs/reference/`](docs/reference/)
- **Design / ADRs** — [`docs/adr/`](docs/adr/)
- **Comparison vs pycocotools / faster-coco-eval / panopticapi /
  boundary-iou-api / lvis-api / mmsegmentation** —
  [`docs/comparison.md`](docs/comparison.md)

## Contributing

Local checks: `just lint && just test && just audit`. The full
contributor workflow (ADR lifecycle, vendoring policy, code style) is
in [`CONTRIBUTING.md`](CONTRIBUTING.md). Repository layout and
common just recipes are in [`CLAUDE.md`](CLAUDE.md).

## License

Dual-licensed under [Apache-2.0](LICENSE-APACHE) or [MIT](LICENSE-MIT)
at your option.

## Third-party code

vernier vendors a small number of test-only reference implementations
to support parity testing. None of this code is included in published
wheels or linked into the Rust binary. See
[`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md) for the full
inventory and license attributions.

