Metadata-Version: 2.5
Name: cozy-eval
Version: 0.2.1
Summary: Quality benchmarking for generative image and video models, with the validity rules enforced by the API.
Project-URL: Homepage, https://github.com/cozy-creator/cozy-eval-v2
Project-URL: Source, https://github.com/cozy-creator/cozy-eval-v2
Project-URL: Issues, https://github.com/cozy-creator/cozy-eval-v2/issues
License-Expression: MIT
License-File: LICENSE
Keywords: audio-quality,av-sync,benchmark,diffusion,image-editing,image-quality,no-reference,quantization,text-to-image,video-quality,vlm-judge
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Scientific/Engineering :: Image Processing
Requires-Python: <3.13,>=3.12
Requires-Dist: msgspec>=0.19
Requires-Dist: numpy>=1.26
Provides-Extra: all
Requires-Dist: cvvdp>=0.5.6; extra == 'all'
Requires-Dist: ffmpeg-quality-metrics>=3.12; extra == 'all'
Requires-Dist: huggingface-hub>=0.23; extra == 'all'
Requires-Dist: lpips>=0.1.4; extra == 'all'
Requires-Dist: numpy>=1.26; extra == 'all'
Requires-Dist: onnxruntime>=1.18; extra == 'all'
Requires-Dist: opencv-python-headless>=4.9; extra == 'all'
Requires-Dist: pillow>=10; extra == 'all'
Requires-Dist: piq>=0.8; extra == 'all'
Requires-Dist: rapidocr>=3.0; extra == 'all'
Requires-Dist: safetensors>=0.4; extra == 'all'
Requires-Dist: scipy>=1.11; extra == 'all'
Requires-Dist: torch<3,>=2.13; extra == 'all'
Requires-Dist: torchmetrics[image]>=1.9; extra == 'all'
Requires-Dist: transformers>=4.53; extra == 'all'
Requires-Dist: transformers>=4.57; extra == 'all'
Provides-Extra: audio
Requires-Dist: numpy>=1.26; extra == 'audio'
Requires-Dist: scipy>=1.11; extra == 'audio'
Provides-Extra: detail
Requires-Dist: numpy>=1.26; extra == 'detail'
Requires-Dist: onnxruntime>=1.18; extra == 'detail'
Requires-Dist: pillow>=10; extra == 'detail'
Requires-Dist: rapidocr>=3.0; extra == 'detail'
Requires-Dist: scipy>=1.11; extra == 'detail'
Requires-Dist: torch<3,>=2.13; extra == 'detail'
Requires-Dist: torchmetrics[image]>=1.9; extra == 'detail'
Requires-Dist: transformers>=4.57; extra == 'detail'
Provides-Extra: dev
Requires-Dist: mypy>=1.13; extra == 'dev'
Requires-Dist: pillow>=10; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: hpsv3
Requires-Dist: huggingface-hub>=0.23; extra == 'hpsv3'
Requires-Dist: pillow>=10; extra == 'hpsv3'
Requires-Dist: safetensors>=0.4; extra == 'hpsv3'
Requires-Dist: torch<3,>=2.13; extra == 'hpsv3'
Requires-Dist: transformers>=4.53; extra == 'hpsv3'
Provides-Extra: judge
Requires-Dist: numpy>=1.26; extra == 'judge'
Requires-Dist: pillow>=10; extra == 'judge'
Requires-Dist: torch<3,>=2.13; extra == 'judge'
Requires-Dist: transformers>=4.57; extra == 'judge'
Provides-Extra: ocr
Requires-Dist: onnxruntime>=1.18; extra == 'ocr'
Requires-Dist: rapidocr>=3.0; extra == 'ocr'
Provides-Extra: preference
Requires-Dist: torch<3,>=2.13; extra == 'preference'
Requires-Dist: transformers>=4.57; extra == 'preference'
Provides-Extra: quality
Requires-Dist: numpy>=1.26; extra == 'quality'
Requires-Dist: pillow>=10; extra == 'quality'
Requires-Dist: piq>=0.8; extra == 'quality'
Requires-Dist: scipy>=1.11; extra == 'quality'
Requires-Dist: torch<3,>=2.13; extra == 'quality'
Requires-Dist: torchmetrics[image]>=1.9; extra == 'quality'
Provides-Extra: reference
Requires-Dist: cvvdp>=0.5.6; extra == 'reference'
Requires-Dist: ffmpeg-quality-metrics>=3.12; extra == 'reference'
Requires-Dist: lpips>=0.1.4; extra == 'reference'
Requires-Dist: torch<3,>=2.13; extra == 'reference'
Provides-Extra: similarity
Requires-Dist: numpy>=1.26; extra == 'similarity'
Requires-Dist: torch<3,>=2.13; extra == 'similarity'
Requires-Dist: torchmetrics[image]>=1.9; extra == 'similarity'
Provides-Extra: video
Requires-Dist: numpy>=1.26; extra == 'video'
Requires-Dist: opencv-python-headless>=4.9; extra == 'video'
Requires-Dist: pillow>=10; extra == 'video'
Description-Content-Type: text/markdown

# cozy-eval

**Quality benchmarking for generative image and video models, with the validity
rules enforced by the API.**

Two questions, one library, one set of validity rules over both:

* *How far did the pixels move, and is that damage or just a different take?*
  Lanes, protocol stamping, null controls, population gates.
* *[Did the model make what was asked for?](#did-it-make-what-was-asked-for)*
  Checklist adherence, preference, standalone quality — MIT reimplementations of
  methods the ecosystem otherwise only ships under non-commercial licences.

Everything is organized by **what it measures**, not by who wrote it. Every
metric lives in `cozy_eval.metrics`, every metric is declared in one
`cozy_eval.registry`, and the same protocol/null-control/population rules govern
all of them.

There are good libraries for computing quality metrics. `ffmpeg-quality-metrics`
and `cvvdp` do full-reference video properly; `torchmetrics` owns FID/KID/IS and
the permissively-packaged learned scorers. This library composes those and
reimplements only what is otherwise locked behind a non-commercial licence.

What no library does is stop you from computing the **wrong metric**. That is what
this one is for.

---

## The problem

You quantized a video model and you want to know if it got worse. The obvious move
— render the same prompt at the same seed on both arms and measure LPIPS or PSNR
between them — is the move essentially every production quantization harness makes.
`torchao`'s Flux benchmark scores same-seed LPIPS against a high-precision baseline.
DeepCompressor/SVDQuant's eval config lists `["psnr", "lpips", "ssim"]` against the
BF16 render directory, and Nunchaku's CI **hard-gates merges** on
`assert lpips < expected_lpips * 1.15`. NVIDIA's TensorRT-ModelOpt diffusers
examples ship no quality eval at all, and where they acknowledge divergence the
advice is *"we suggest to run a few more times and choose the best one."*

That measurement is invalid, and here is the evidence, taken CPU-only on banked
LTX-2.3 renders at 1920×1088:

1. **The no-op consumes the entire budget.** A compile-only control — zero
   quantization change — scores LPIPS **0.196–0.249** against a fleet fp8 budget
   of 0.25.
2. **It is divergence, not drift.** Distance is already **0.29–0.41 at frame 0**
   and flat-to-falling across the clip. Numerical error accumulates; a different
   take starts far apart and stays there.
3. **The ranking inverts.** An fp8-storage-cast arm and an unscaled-w8a8 arm carry
   *identical weight bytes*. The cast arm computes its GEMMs in bf16 and is
   therefore strictly the more accurate path — and it scores **worse**
   (0.3875 vs 0.3052).

A metric that ranks a strictly-better arm below a strictly-worse one cannot be
used to choose between arms. Anything that perturbs the sampling trajectory —
quantization, `torch.compile`, an attention-backend swap, a scheduler change, a
LoRA attach — produces a *different take of the same prompt*, and distance to the
reference render measures how different the take is, not how damaged it is.

The academic side has largely avoided the trap (Q-Diffusion, ViDiT-Q and friends
report FID/FVD/VBench, i.e. population metrics) but states its reason as sample
size rather than validity, so the production side never got the message.

## What this library adds

| | |
|---|---|
| **Two lanes, enforced** | `classify_change()` types every change as post-latent or trajectory-perturbing. `require_reference_lane()` **raises** rather than returning a number. One perturbing change contaminates a mixed set. |
| **Protocol stamping** | Every result carries resolution, frame count, steps, seeds, prompt set, execution lane, hardware and same-pod status. Cross-pod or mismatched-lane arms return `INDETERMINATE`, never `PASS`. |
| **Population semantics** | Trajectory-perturbing comparisons need n ≥ 8 paired prompts. At n = 1 a *shipped-clean* w8a8 recipe scores an imaging index of 0.934 on one prompt and 1.005 over eight. One prompt measures the take. |
| **Three orthogonal benchmarks** | Imaging, temporal, distributional — with a banked degradation each one catches that the other two miss. |
| **Thresholds with provenance** | Every budget names the known-good and known-bad populations that fixed it and its separation margin. Provisional ones say so. |
| **Null-control arms** | A budget is only a budget where a *zero-change* arm sits inside it. `measure_null_control()` measures that per family; budgets its control trips are disregarded, and a run with disregarded budgets cannot rise above `INDETERMINATE`. Measured: null controls trip `population_frechet` at 2.3× and 6.6× the budget with **zero** model change. |
| **NO-SIGNAL, not a confident FAIL** | An arm of constant frames returns `DEGENERATE`, not `FAIL` with an imaging index of 0.0. There is nothing to compare; ranking it would be a category error. |
| **Statistical honesty** | Paired *t* with Holm correction, effect sizes, and a practical-effect floor so significance without magnitude cannot fail an artifact. No package surveyed reports a confidence interval or effect size on a quality delta. |

| **One implementation per name** | Every `MetricSpec` carries a per-metric `impl` version beside the registry-wide `METRIC_SET_VERSION`, every score row carries it, and comparing rows across differing impls is a **typed refusal**. clean-fid measured same-named FID disagreeing by whole points on resize filter alone; a name does not identify a number, `(name, impl)` does. |
| **A cost class per metric, as data** | `cheap-deterministic` / `small-model` / `model-backed`, declared in the registry, because that is the fact a consumer routes on. A model-backed metric with no injection seam is a registration error. |
| **Functional AND accumulator** | Every metric is callable one-shot, and `MetricAccumulator` streams `update()`/`compute()` over plain-array state for corpus-sized or per-frame scoring. Both paths call the same pooling function, so they cannot disagree. |

Composed where a maintained permissive implementation exists: `ffmpeg-quality-metrics`
and `cvvdp` for full-reference video, `lpips` for LPIPS, `torchmetrics` for
ARNIQA/CLIP-IQA. PSNR, SSIM and
MS-SSIM are ours, in numpy, one implementation each — they used to be two or
three each depending on what you had installed, which is the same defect the
`impl` version exists to prevent. The base install is **numpy + msgspec only**;
everything else is an extra, and a fence keeps it that way.

## Install

```bash
pip install cozy-eval                      # numpy + the ffmpeg CLI
pip install "cozy-eval[reference]"         # VMAF / ColorVideoVDP / LPIPS
pip install "cozy-eval[video]"             # optical flow + track stability (opencv)
pip install "cozy-eval[quality]"           # NIQE / MUSIQ / ARNIQA / CLIP-IQA
```

> **Licence note.** No dependency here is non-commercial, and none ever will be.
> `pyiqa` was dropped outright when it relicensed Apache-2.0 →
> PolyForm-Noncommercial-1.0.0 at 0.1.16 (2026-07-08): NIQE and MUSIQ are ours,
> ARNIQA and CLIP-IQA come from torchmetrics, and BRISQUE/MANIQA/TOPIQ are gone
> rather than ported. DOVER and FAST-VQA are deliberately not wrapped — both are
> S-Lab-1.0 (non-commercial) while their `setup.py` files still declare
> MIT/Apache. `parity/` keeps the replacements honest against the banked oracle.

## Use

```python
from cozy_eval import (
    ChangeKind, Protocol, score_pairs, run_population_gate,
)

protocol = Protocol(
    family="ltx-2.3-distilled",
    reference_arm="bf16 compiled",
    candidate_arm="w8a8-pcs compiled",
    changes=(ChangeKind.WEIGHT_QUANTIZATION, ChangeKind.ACTIVATION_QUANTIZATION),
    width=1280, height=704, frames=121, steps=8,
    seeds=(8,),
    prompts=("forge", "chef", "market", "portrait",
             "fabric", "street", "water", "forest"),
    execution_lane="compiled",              # both arms, or it is INDETERMINATE
    hardware="NVIDIA H100 80GB HBM3",
)

pairs = score_pairs(reference_clips, candidate_clips)   # paths, dirs, or arrays
report = run_population_gate(pairs, protocol)
print(report.summary())
```

```
PASS  [population]
ltx-2.3-distilled | w8a8-pcs compiled vs bf16 compiled | 1280x704 x121f | 8 steps |
n=8 prompts | compiled both arms | NVIDIA H100 80GB HBM3 | same pod | lane=population
  PASS  imaging
        ok  0.92 <= imaging_index <= 1.25   measured 1.0050
        ok  imaging_worst_prompt >= 0.85    measured 0.9410
  PASS  temporal
        ok  jerk_excess <= 0.04             measured 0.0132
        ok  flicker_ratio <= 1.25           measured 1.0435
  PASS  distributional
        ok  significant_features <= 0.0     measured 0.0000
        ok  population_frechet <= 0.2       measured 0.0680
```

And the refusal:

```python
>>> run_reference_gate(refs, cands, protocol)   # protocol says WEIGHT_QUANTIZATION
TrajectoryPerturbingError: reference metrics (PSNR/SSIM/LPIPS/VMAF) are invalid for
['weight-quantization']: these change the sampling trajectory, so the reference
render is a different take of the prompt and distance to it measures divergence,
not damage. …
```

Post-latent changes take the reference lane and are valid at n = 1:

```python
run_reference_gate(before, after, protocol_with(ChangeKind.VAE_DECODE_DTYPE))
```

Images are the single-frame case — same lanes, same rules,
`run_image_population_gate` drops the temporal benchmark.

### The promotion ladder (quantization gates)

A derive job's fidelity gate is a three-tier ladder (weight diff → activation
probe → output comparison) with a cost cliff between the tiers, so it is not
one `compare()` call: measurement and policy are separate surfaces, and the
policy is one decision per stage so the cheap tier can spare you the expensive
one. `run_output_gate` is tier 3's measurement in one call — it dispatches on
the protocol's lane and frame count to the right gate above. The policy fold
lives in `cozy_eval.ladder`: a `GateConfig` (K_promote / K_reject as multiples
of the probe's measured ref-vs-ref noise floor, absolute caps, pinned
prompts/seeds — a contract document, pinnable by digest per family) and two
functions returning a typed `GateDecision` (PROMOTE / REJECT / ESCALATE):

```python
from cozy_eval import LadderVerdict, output_decision, probe_decision, run_output_gate

d = probe_decision(probe, config)          # tier 2: strict-K short-circuit
if d.verdict is LadderVerdict.ESCALATE:    # only now pay for generations
    report = run_output_gate(reference_renders, candidate_renders, protocol)
    d = output_decision(report, config, probe=probe)   # FAIL names the layer
```

Every tap under `k_promote x floor` (and under its cap) promotes without
generations; NaN/Inf or any tap over `k_reject x floor` rejects outright,
naming the tap; anything between escalates to tier 3, whose failures read the
tier-2 map to say which layer degraded. `probe` evidence comes from
cozy-runtime's activation probe (`cozy.probe.report/1`); the wrap that folds
that document into `ProbeDelta` lands with the probe itself.

### Null controls: is this family allowed to be judged on these budgets?

The thresholds were calibrated on one family. Render a third arm on the same pod
— the **same checkpoint at different seeds** — and let it decide which budgets
you may believe:

```python
from cozy_eval import ChangeKind, measure_null_control

control = measure_null_control(control_pairs, control_protocol)   # ChangeKind.SEED only
report = run_population_gate(pairs, protocol, null_control=control)
```

```
INDETERMINATE  [population]
  null control n=8: DOES NOT TRANSFER — population_frechet measured 0.4688
  PASS  imaging
        ok  0.92 <= imaging_index <= 1.25   measured 1.0739
  PASS  distributional
        ok  significant_features <= 0.0     measured 0.0000
        --- population_frechet <= 0.2       measured 0.1825
        note: … DISREGARDED: the null control — identical weights, seeds only —
              measured 0.4688 …, so this budget does not transfer to this family.
  ! budgets disregarded on the null control's evidence … the best available
    verdict is INDETERMINATE.
```

Measured on two image families: null controls read `population_frechet` 0.4688
and 1.3275 against a 0.20 budget, and one read `imaging_worst_prompt` 0.6102
against 0.85 — at zero model change. Without the control, one of those families
reads a confident FAIL on budgets its own null arm fails harder. `imaging_index`
and `significant_features` transferred on both. Details: [GATE.md §4b](GATE.md).

### Reports are digest-stable bytes

A report is the product: downstream jobs take it as a digest-verified typed
input, so its BYTES are part of the contract.

```python
from cozy_eval import contract

digest = contract.save("run.json", report, protocol=proto)   # sha256:...
sealed = contract.load_verified("run.json", digest)
sealed.body            # the typed SuiteReport
sealed.impls           # {metric: impl} — what makes two runs subtractable
sealed.require_protocol()   # or a typed refusal, if the run never declared one
```

`contract.dump` writes RFC 8785 canonical bytes: sorted keys, one number
spelling, no whitespace, NaN/Infinity refused. Equal reports are equal bytes, so
`sha256` identifies a result. `contract.load` refuses non-canonical bytes, an
unknown schema version and any unknown field — never a best-effort partial read.
Schema ids live in `contract.SCHEMAS` and bump on ANY change, additive included.

`ClipScore.to_dict()` / `from_dict()` round-trip the gate's per-clip INPUTS
(including the `(frames, 6)` feature matrix) so a lane can re-gate without
re-rendering. That is a cache, not a report: no envelope, no digest.

## CLI

```bash
cozy-eval score clip.mp4 other.mp4
cozy-eval gate --reference a.mp4 --candidate b.mp4 … \
    --change weight-quantization --execution-lane compiled …
cozy-eval compare --reference a.mp4 --candidate b.mp4 --vmaf \
    --change vae-decode-dtype …
```

## Compute budget

An eval is a batch job on somebody else's machine, so **it never takes more than
`min(4, cpu_count)` compute threads** — anywhere, including a 96-core pod. One
knob raises or lowers it, and everything derives from that number: BLAS/OpenMP
pools, torch intra- and inter-op, `cv2.setNumThreads`, every `ffmpeg -threads`
and `libvmaf n_threads`, and the worker pool, which *splits* the budget
(`workers = threads / 2`, each worker taking an equal share) rather than
multiplying it. Every run prints what it took:

```bash
cozy-eval --threads 32 score clip.mp4      # configure(threads=32) as a library
# cozy-eval: 32 compute threads, 16 workers (explicit)
```

**Set it explicitly on a dedicated pod** — the default is deliberately modest and
will leave a big machine idle. Pool variables (`OMP_NUM_THREADS` and friends) you
set yourself are never overwritten. Thread count is invisible to results:
one thread and thirty-two produce bit-identical scores, and a test pins that.

Rough cost, CPU-only at the default budget: the no-reference and integrity
statistics are milliseconds per clip; optical-flow temporal fidelity is a few
seconds per 720p pair (it samples `FLOW_PAIRS` transitions, so a 3-second and a
3-minute clip cost the same); per-frame LPIPS/SSIM/MS-SSIM and VMAF scale with
frame count and dominate everything else; the VLM judge is a model call per
sample and belongs on a GPU.

## The three benchmarks

1. **Imaging** — per-frame, no-reference. Detail (Laplacian variance, spectral
   HF ratio, local contrast) and tone (contrast, saturation, histogram entropy),
   as ratios to the reference arm, median-aggregated across prompts.
2. **Temporal** — frame-to-frame. Jerk ratio (second temporal difference over the
   first), exposure flicker, shimmer, motion energy. Quantization noise often
   shows up here and nowhere else.
3. **Distributional** — a paired test across the whole prompt set, with
   Holm-corrected significance and a practical-effect floor. Not FVD: at n = 8 a
   deep-feature Fréchet distance is dominated by estimator bias. Because the
   prompt set is identical between arms, content is differenced out by
   construction and a paired test has real power.

Each catches a degradation the other two miss — see the validation table in
[GATE.md](GATE.md).

## Did it make what was asked for?

The gate above tells you whether pixels moved for a *valid* reason. The suite
answers the other question: is the render actually what the prompt requested?
It exists because the good evaluation code in this space is locked up —
GenEval2 is CC-BY-NC, pyiqa relicensed to PolyForm-Noncommercial, DOVER and
FAST-VQA are S-Lab non-commercial. This library reimplements the published
methods from the papers under MIT, with the audit trail in
[`PROVENANCE.md`](PROVENANCE.md).

Four mostly-independent dimensions, each with exactly one gated headline
number; everything else is report-only:

| dimension | what it asks | headline | needs a reference? | module |
|---|---|---|---|---|
| **similarity** | how far did the pixels move | `lpips` | yes | `metrics/similarity.py`, `metrics/reference.py` |
| **adherence** | did it contain what was asked for | `element_recall` | no | `metrics/adherence.py`, `geneval.py`, `ocr.py`, `vqascore.py` |
| **preference** | would a human prefer it | `pref_delta` | no | `metrics/preference.py`, `hpsv3.py` |
| **quality** | does it look good on its own terms | `arniqa` | no | `metrics/quality.py`, `musiq.py`, `signal.py` |

The Δ-frame temporal channel and the flow family (`metrics/temporal.py`), the
track-stability family (`metrics/tracks.py`) report into the dimensions above; the
gate's own two-sided budgets over a whole prompt population live in
`cozy_eval.benchmarks`, which is a threshold table, not a metric table.

```python
from cozy_eval import promptset, suite

report = suite.run(samples, candidates,
                   checklists=promptset.checklists_for("hard-eval-v1"))
print(report.summary("element_recall"))
```

The load-bearing design decisions:

* **Authored, versioned checklists.** `element_recall` is the weighted fraction
  of a prompt's authored checklist verified present (`ocr` items read literally;
  `vqa` items answered by a VLM judge, one structured call per image). Checklists
  are versioned with their prompt set — never generated per run, so a score is
  reproducible. Shipped sets: `hard-eval-v1` (t2i + edit), `hard-video-v1`
  (16 frozen t2v prompts with motion/hold dualities).
* **Editing is a duality**: `edit_compliance` (the instructed change happened)
  vs `edit_preservation` (everything else stayed put) — under- and over-editing
  fail on opposite halves.
* **Video** (`cozy_eval.video.run_video`): per-frame aggregation with
  worst-frame tails, a Δ-frame temporal channel per-frame metrics cannot see,
  and motion/hold checklists judged on an ordered frame strip **split into three
  temporal windows, gated on the worst** — a clip that holds its content for
  10 s and drops it for the last 5 is not two-thirds compliant, it is broken,
  and the whole-strip mean passed exactly that arm. `element_recall_drop`
  (first window minus last) reports the direction.
* **Tri-state parity verdict** (`free_win` / `conditional_parity` / `reject`): a
  candidate that fulfilled the request *differently but equally well* is not a
  failure — the case a pixel-distance metric cannot express.
* **Registry as data.** `import cozy_eval` sees the complete metric table
  without importing a single scoring backend; external metrics join through
  `register()`.

A judge does not have to be local. `cozy_eval.wire` implements the same four protocols
against the `quality-judge` package (ev-003), so eval-job code is identical either way and
NO extra is needed — the client speaks to a caller-supplied transport (`stage` + `invoke`)
and this library still contains no network code at all:

```python
from cozy_eval import video, wire

judge = wire.WireJudge(transport)      # anything with stage()/invoke()
judge.prefetch(work)                   # one request per sample batch, replies cached
report = video.run_video(samples, clips, checklists=corpus.checklist_set(), judge=judge)
```

Scoring backends are extras, so the base install stays torch-free:

```bash
pip install "cozy-eval[similarity]"      # LPIPS / SSIM / MS-SSIM / PSNR
pip install "cozy-eval[judge]"           # VLM judge, CLIP fallback, Grounding DINO + SigLIP2
pip install "cozy-eval[ocr]"             # OCR items (rapidocr, Apache-2.0)
pip install "cozy-eval[preference]"      # PickScore and alternates
pip install "cozy-eval[quality]"         # ARNIQA / CLIP-IQA / MUSIQ port / NIQE
pip install "cozy-eval[hpsv3]"           # HPSv3 preference scorer (16 GB weights)
pip install "cozy-eval[video]"           # frame handling, Δ-frame channel, flow + track stability
pip install "cozy-eval[all]"
```

**Model licences — read before you ship.** The library is MIT; the models it
can load are not all MIT. It never defaults to weights that cannot be used
commercially, and every model it touches has a verified row in
[`PROVENANCE.md`](PROVENANCE.md) — including "none stated", where that is the
truth. `parity/` holds the harness that keeps the replacements honest against the
non-commercial oracles — NIQE within 2.4%, MUSIQ within 4.5% with identical
rankings, CLIP-IQA bit-identical under the oracle's own prompt set, ARNIQA
deliberately diverged (antialiased half-scale) with the divergence isolated and
recorded. The oracle NUMBERS, not code, are banked in `parity/oracles/`.

**Stability**: everything re-exported from the `cozy_eval` package root (metric
names, the registry, the report schema, checklist/prompt-set formats, the
verdicts, the Judge protocols, the protocol/lane rules) is locked for 0.x;
everything under `cozy_eval.metrics.*` is experimental.

## Is it a render at all?

Before any of the questions below is worth asking, one has to be settled: is
this output *anything*? A production video model of ours served pure
VAE-decoded **noise** on billed, settled requests and passed every check in
place, because the evidence banked was container metadata and a billing row.
Nobody looked at pixels. Metadata is not pixels.

```python
from cozy_eval import output_integrity

checked = output_integrity(frames)          # (T, H, W, 3), uint8 or float
if not checked.ok:
    raise RuntimeError(checked.summary())   # names NOISE, BLANK or GRID
```

Real video is strongly self-similar frame to frame; noise is correlated with
nothing. The median adjacent-frame grey correlation separates the two with an
empty middle — **noise 0.29, real renders 0.92-0.99, floor 0.60** — and taking
the *median* over pairs spread across the clip is what keeps a hard cut from
reading as noise. A per-frame contrast floor catches blank output, which
correlates with nothing and so cannot be caught by correlation at all. numpy
only, no reference, no model: **8.3 ms** on a 121-frame 1344x768 clip, which is
cheap enough to run on every render including the serve path.

**A third axis: is it divided into little boxes?** The screen once passed four
renders the owner rejected on sight for a visible lattice — at
`adjacent_frame_corr` 0.963-0.988, a comfortable pass. That is not a tuning
miss. A periodic grid is *perfectly self-similar frame to frame*, so a
correlation floor is structurally blind to it and the defect **raises** the
number the screen reads. `grid_peak_ratio` measures what a lattice is and
content is not: energy in a single spatial frequency, at a fixed phase, in the
same place on every frame. Owner-accepted renders measure 1.975-3.804,
owner-rejected ones 7.320-14.179, and the limit is **6.0** in the empty band —
1.54× above the worst known-good, the same relative headroom the noise floor
carries. It takes the whole screen to a median of 30 ms. Derivation and the
per-file table: `GATE.md` §5b; evidence `calibration/grid-blockiness.json`.

> **It is not a quality gate, and must never be quoted as one.** It catches
> noise, blank and gridded output. A melted or over-smoothed render scores
> *higher* than a clean one — smearing removes high-frequency temporal
> variation, so damage looks like stability. Fine detail is `detail_verdict` and
> the VLM rubric; motion is the temporal-fidelity family. Four axes, none
> sufficient alone, and the library ships a test that pins this blind spot
> rather than a sentence claiming it does not exist.

## Do the OBJECTS hold together?

Every number above is a statistic of *frames*. Per-frame detail, whole-frame
optical flow over sampled pairs, frame-mean luma, an ordered strip of stills
shown to a VLM. An object can warble — drift, jitter and reshape itself as the
camera moves through the scene — while every one of those stays clean, and that
is not a hypothetical:

> "Objects lose their coherence across frames. Suppose object-A is at position
> X,Y and we move the camera slightly; the object should move on the frame
> correctly as you would expect when moving through 3-dimensional space. Instead
> it warbles and reshapes itself."

Three metric families **passed** the clips that produced that sentence: the
fine-detail detectors (per frame), the temporal-fidelity family (whole-frame
flow statistics over decimated pairs), and the VLM strip read. The clips were
rejected by the owner's eye. Nothing in the stack followed a *point on an
object* through time, so nothing in the stack could see it.

```python
from cozy_eval import track_verdict

checked = track_verdict(candidate_frames, reference_frames)   # same-seed control
if not checked.ok:
    print(checked.summary())
    # track stability REJECT — OBJECT WARBLE: track_stability_ratio 0.176 <
    # floor 0.9 — the candidate retains 18% of the control's coherent tracks
    # (trajectories jitter: 0.322 vs 0.120; neighbours disagree: rigidity error
    # 0.609 vs 0.366) (track_stability 0.058, ratio 0.176)
```

Corner features are seeded and chased frame to frame with forward-backward
validated pyramidal Lucas-Kanade, and each **trajectory** is asked three
questions: is it *smooth* (a point on a rigid object traces a smooth image-plane
curve whatever the camera does, so warble shows up as second-derivative energy —
normalized by the track's own speed, so a fast pan is not penalized for being
fast), does it *survive* (a surface that reshapes stops matching itself), and do
its *neighbours agree* (points on one surface keep their relative geometry
through parallax and perspective, which are smooth).

| metric | dimension | gates? | what it says |
|---|---|---|---|
| `track_stability_ratio` | similarity | **yes**, 0.90 – 1.40 | fraction of the control's coherent tracks the candidate retains |
| `track_stability` | quality | no | fraction of seeded points that survive *and* move like a real 3D point |
| `track_survival` | quality | no | fraction still tracked at the end of the window |
| `track_jitter` | quality | no | median per-track acceleration energy, camera motion removed |
| `track_rigidity_error` | quality | no | median jerk of neighbour distances — the "reshapes itself" half |

**Both bounds are measured on labeled ground truth** (`calibration/track-stability.json`):

| set | what it is | n | ratio | verdict |
|---|---|---:|---|---|
| **rejected** | sparse-attention k16/k32 arms the owner rejected, vs their own same-cell same-seed dense control | 29 | 0.029 – **0.846**, median 0.366 | 29 reject, 0 pass |
| **identical** | SageAttention-2 fp8 arms the owner reviewed as identical to FA3-exact, plus same-arm re-renders across a pod and a torch-line change | 12 | **0.930** – **1.251**, median 1.050 | 12 pass, 0 reject |
| **bit-exact** | a clip against itself, two independent decodes and two independent scorings | 2 | **exactly 1.0** | pass |
| **over-smoothed** | the 15 s arm that re-rolled into a simpler, slower take and dropped the cargo bike and the parcel it was asked for | 1 | **2.169** | reject *(passed before the ceiling)* |

0.90 sits in an empty middle 8 points wide, and 1.40 in a second one — 1.251 to
2.169 at the shipped budget, and 1.251 to 1.512 taken across every window
budget, which is the middle the bound is actually placed in. The independent negative controls agree: an untrained-selector arm and a
grouped-selector arm that a separate detector already called broken score
**0.000** here, and the oracle top-k arm — itself a k16 sparse render — 0.606.

**The gate is two-sided, and that is not decoration.** A floor-only gate is
walked through by anything that does LESS: the over-smoothed row above scored
2.17× its control and PASSED, because a simpler take is easier to track. The
pathology is general, not ours — CD-FVD measured that sampling motion-free video
*lowers* FVD by 31.6–54.8%, VBench's `motion_smoothness` measures interpolation
predictability (which blur maximizes), and EvalCrafter's fitted composite has
all three motion coefficients negative. The obvious companion instrument does
**not** rescue a one-sided gate either: `motion_mag_ratio` reads 0.920
whole-frame and 1.006 under VBench Dynamic-Degree top-5% pooling on that same
arm, inside the 0.80–1.05 band of pairs the owner called identical, at every
pooling and window tried (`calibration/motion-magnitude.json`). It ships
declared and **report-only**, with the negative recorded so it is not
re-proposed. What an over-smoothed arm actually loses is *content*, and the
instrument for that is the windowed `element_recall` below.

The paired ratio is **valid across a re-rolled take**, which is the whole point:
both arms are scored on their *own* trajectories and never compared pixel to
pixel, so a trajectory-perturbing lane is measured on whether its objects hold
together, not on how far its take drifted. Same property as `warp_error_delta`,
and the reason `lpips` cannot be used here at all.

> **Untrackable content is UNMEASURED, never a pass and never a fail.** Steam,
> water, molten glass and dense repetitive weave defeat *any* sparse tracker:
> on those cells the clean control itself holds 13-18% of its tracks and its
> numbers swing further between two renders of the **same arm** than the whole
> rejected-vs-clean separation. When the control's survival is below 0.25 the
> family refuses — measured consequence: at a 0.15 floor, a loom pair the owner
> judged identical would read 0.07 and be called a catastrophic reject. 7 of 36
> rejected pairs and 3 of 15 clean pairs land here and are reported as
> unmeasured, with the reason.

> **Scope.** This catches object warble and 3D-inconsistency under motion. It is
> blind to per-frame damage (melted faces, pseudo-glyphs, halos — that is
> `detail_verdict`), to content adherence (the checklist and the VLM), and to
> whole-clip shimmer (`warp_error`). A clean ratio is one axis, not a quality
> verdict.

**What one clip of the CPU tier costs** (`calibration/perclip-cost.json`, measured
on the labeled 362-frame 1344×768 pair, four threads): decode 2.2 s per clip,
integrity **0.01 s**, `warp_error` 2.3 s per arm, the paired temporal-fidelity
block 5.0 s, the paired track block 1.6 s — and, until @9, **48.5 s of signal
statistics** to return two luma scalars. That one family was three quarters of
the bill: it computes six per-frame feature families at full resolution (an FFT,
a Laplacian, a 64-bin histogram, a box filter, saturation, luma σ) whose only
consumer is `benchmarks.imaging()`, the *population* lane. The per-clip path now
runs a luma-only pass that reproduces `luma_flicker` and `jerk_ratio` **exactly**
(pinned by a test, not approximately): 48.5 s → **3.0 s**, and the paired
per-clip gate **61.8 s → 16.3 s**. No metric was lost.

**Cost and the decimation pin.** No new dependency — the tracker is the same
BSD-licensed OpenCV the flow family already uses, because the obvious learned
tracker, CoTracker, is CC-BY-NC-4.0 and this library does not ship
non-commercial weights (TAP-Net/TAPIR is Apache-2.0 but adds a checkpoint
download and GPU inference to a CPU tier that costs a second). Four 24-frame
windows at 384-line working height: **1.2 s median per clip on four idle
threads**, 3.1 s median in the banked run, which was taken on a shared box under
a 1-minute load of 21 — the number moves with the machine, so both are quoted.
The window count is the decimation knob, and it is pinned: verdicts on the whole
labeled set are **identical to tracking every frame of every clip** (1.9× the
cost), while halving it to two windows flips one clean pair to reject. Four is
the floor of the ladder, not a default that happens to be cheap.

## Does it SOUND right?

Every video model we serve now emits audio — LTX-2.3 denoises audio latents in
the same loop and muxes AAC, MiniMax-H3 generates 32 kHz stereo jointly with the
picture. A quant lane, a cache sweep or a step-distill LoRA can destroy the
soundtrack while every pixel number stays green: on one banked arm a cache sweep
drove audio SNR **20.67 → 13.72 dB** while the same clip's video SSIM still read
0.85. There is, as far as we can find, **no published audio-degradation
measurement for any caching or quantization technique on a joint audio-video
model** — so this is a first-contact instrument, not a reimplementation.

One call, one verdict:

```python
from cozy_eval import audio_verdict, read_audio

result = audio_verdict(
    read_audio("candidate.mp4"),
    read_audio("bf16_anchor.mp4"),   # optional: enables the faithfulness tier
    frames=candidate_frames, fps=24.0,   # optional: enables AV-sync
)
print(result.summary())
# audio REJECT — audio_stereo_separation_db 98.6 breaches audio_stereo_separation_db <= 60 (11 measured, 6 unmeasured)
```

Three tiers, and a fourth outcome that is not a tier:

| tier | metrics | needs |
|---|---|---|
| **signal, reference-free** | `audio_rms_dbfs` `audio_peak_dbfs` `audio_lufs` `audio_clip_fraction` `audio_silence_fraction` `audio_dc_offset` `audio_spectral_flatness` `audio_side_dbfs` `audio_stereo_separation_db` `audio_channel_correlation` | nothing — always runs |
| **paired fidelity** | `audio_si_sdr` (gated) `audio_snr_db` `audio_lsd_db` `audio_mel_l1` `audio_lufs_delta` `audio_align_lag_ms` | a reference arm |
| **AV-sync** | `av_sync_offset_ms` `av_sync_confidence` `av_sync_drift_ms` (gated) | frames + fps |
| **semantic** | `audio_event_recall` (gated) `audio_speech_exact` `audio_speech_fuzzy` | an authored checklist + a `Transcriber` and/or `AudioJudge` |
| **UNMEASURED** | — | everything the run could not score lands in `result.unmeasured` **with the reason**, and a verdict that measured nothing is `unmeasured`, never `pass` |

**Audio is not a seventh dimension.** The six dimensions are *questions*, not
media: audio SNR is `similarity`, a sound-of-X checklist is `adherence`,
clipping and dual-mono are `quality`, AV-sync is `temporal`. So a quant arm that wrecks the audio
reaches `reject` through the tri-state verdict machinery that already existed.

**The one shipped budget.** `AUDIO_DEFECTS` is this library's only built-in
threshold table, and the exception is deliberate: silence, clipping, dual-mono
and DC are content-*independent* engineering faults, unlike an LPIPS budget that
only means something against the population it was calibrated on. It is
calibrated on **41 real fal MiniMax-H3 generations** — 30 ambience plus the
11-clip transient showcase — with a further 9 of our own H100 renders HELD OUT to
validate rather than fix the limits. Every limit carries its own margin over the
worst known-good clip, and **nothing in it is report-only at @11**.

Getting there meant replacing two statistics rather than re-fitting them. v1
calibrated on the ambience half alone, and `audio_peak_dbfs ≤ -0.1` and
`audio_silence_fraction ≤ 0.50` are exactly the two numbers that separate
ambience from transient content, in opposite directions — they failed four
correct renders (both arms' drummers, a domino chain, a sparse blacksmith) for
containing the impulsive events an AV-sync corpus *must* contain. So:

* **sample peak → true peak (BS.1770-4, 4× oversampled), paired with clip
  fraction.** A loudness-normalized transient sits at full scale by construction
  (a known-good domino chain peaks at **+0.18 dBFS**), and a master clipped
  upstream of a lossy codec decodes *under* full scale — the sample peak is wrong
  in both directions. Known-good tops out at +0.193 dBTP; the limit is +1.0.
* **silent fraction → longest consecutive silent RUN.** A known-good blacksmith
  clip is **61 % silent** and correct: three hammer strikes with gaps. A stopped
  generator is one long stretch. Known-good tops out at a 0.136 run; the limit is
  0.25, and the old fraction limit missed a generator that stopped 65 % of the
  way through entirely.

`calibration/run_audio_defects.py` re-derives all six over the real populations:
50 known-good clips pass, 8 red arms — each made by degrading a real clip at a
known severity — all fail.

**What it does NOT do, stated rather than hidden.** AV-sync here is *event*
sync — an audio onset envelope cross-correlated against a visual onset envelope,
closed form, no weights (Bello et al. 2005; Hershey & Movellan, NIPS 1999). It
does **not** score lip-sync: SyncNet's weights are research-worded CC-BY and
Synchformer's checkpoints carry no weights-specific licence at all, so the gap is
recorded and travels in every report that carries a sync number. And when the
content has no shared audio/visual onsets, sync returns UNMEASURED with the
reason — measured on the fal corpus, **all 18 clips are unmeasurable for event
sync**, because generated ambience carries no transients. Gating sync needs a
prompt subset authored to contain door slams, claps and dialogue — and authoring
one is harder than it sounds, which is what `cozy_eval.corpus` exists for.

## Corpora

A corpus is prompts and their authored checklists as **one versioned unit**: a
row IS its checklist, so there is no representation of a prompt without its
assertions and no way to edit them apart. It loads from a dataset tree
(`corpus.json` + `rows.jsonl` + blobs), and an edit that never re-sealed is
refused at load — `corpus_id@version` has to name exactly one set of rows or a
score taken under it is not reproducible.

```console
$ cozy-eval corpus lint calibration/corpus/av-sync-v1 --snapshot-root ~/cozy/samples
av-sync-v1@1  9 rows / 22 items  av=9  rows_digest sha256:138d7ee181dd53cf...
  WARN  sync-gate-texture-source   anime-courier-10s   names both an impulsive
        event ['landing', 'thud'] and a sustained source ['hum', 'wind']
  2 findings, 0 error(s)
```

**The AV-sync authoring rule is an executable check, not advice.** A sync-gated
prompt must name an *impulsive* event, and the measured trap is that steam hiss
and hail read percussive on paper and generate as untrackable texture. Two of the
four clips originally commissioned as sync gates were exactly that. So the lint
screens the rendered exemplar with the same onset instrument that will later
measure the gate: peak-to-median envelope crest ≥ 17 dB and ≥ 2 prominent onsets.
On the shipped corpus the four surviving gates read 17.6–32.7 dB and the rejected
texture reads 7.0–15.4 dB.

The frozen v1 sets (`hard-eval-v1`, `hard-video-v1`) stay in-package, unchanged,
and load through the same validator via `corpus.packaged(name)`.

## Documentation

* **[GATE.md](GATE.md)** — the protocol a producer lane follows verbatim: lanes,
  render conditions, thresholds with calibration provenance, the full validation
  table with separation margins, and an explicit list of what this gate does
  *not* measure.
* **[PROVENANCE.md](PROVENANCE.md)** — per module: which paper it implements,
  whether the implementation is original, and the real licence of every
  dependency and model weight involved.
* **[calibration/](calibration/)** — the evidence. `run_banked.py` regenerates
  every threshold from real renders plus synthetic single-axis controls.
* **[parity/](parity/)** — banked oracle NUMBERS (never NC code) for every
  method reimplemented here, with the deterministic scenes they were taken on.
* **`scripts/verify.py`** — runs the real metrics on real generated inputs and
  prints the values, wall time and RSS it actually got. This is how the library
  is verified: there is no test suite (see Status).
* **`scripts/fence.py`** — the architecture fences: no serving/storage/catalog
  dependency, base install stays numpy + msgspec, the
  environment is read once (into `cozy_eval.settings.EvalSettings`), registry
  identity holds, and
  no test suite creeps back.

## Status

**v2 line (`2.3.3`, metric set `cozy-eval/metrics@13`; CPython 3.12 only).** Six dimensions —
`similarity`, `adherence`, `preference`, `quality`, `temporal`,
`distributional` — with one headline per dimension and modality. Cut at @10:
video-mode `ssim`/`ms_ssim` (declared image-only; pooled over frames a windowed
structural similarity rewards over-smoothing), the whole Δ-frame PSNR family (a
reference distance taken of a reference distance), and `edge_overshoot` off the
report surface (scene-confounded, and it ranks arms backwards). The runner now
REFUSES reference metrics on a trajectory-perturbing lane instead of printing a
quotable confounded number.

**There is no test suite, deliberately.** Verification is running the real
metrics on real inputs and recording what happened: `python scripts/verify.py`
is 92 invariant checks and 31 metrics benchmarked live, and CI is build plus
fences only.

Alpha. The thresholds are calibrated on LTX-2.3, Wan-2.2 and H3 class video at
720p–1080p, 4–30 steps; a family far outside that should carry a null-control arm
(and, better, re-derive the budgets from its own clean population).
`population_frechet` gained its second clean population at @11 — H3, n=9 pairs the
owner labelled identical, **0.0665** against the banked LTX population's
**0.0680** on a different family — so it gates on video, with an owner-rejected
sparse-attention arm 51× away at 3.4191. It is deliberately absent from
`IMAGE_BUDGETS`: on images the only evidence is two zero-change null controls at
0.4688 and 1.3275, and a budget that does not transfer is not a budget there.

## Licence

MIT. See [`LICENSE`](LICENSE).
