Metadata-Version: 2.4
Name: metrik-eval
Version: 0.5.0
Summary: Metrik evaluation analysis — where a model is strong, where it is weak, and which claims survive the interval.
Project-URL: Homepage, https://github.com/Asmodeus14/Metrik
Project-URL: Repository, https://github.com/Asmodeus14/Metrik
Author: The Metrik Authors
License-Expression: Apache-2.0
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: metrik-bench==0.5.0
Requires-Dist: metrik-sdk==0.5.0
Description-Content-Type: text/markdown

# metrik-eval

Where a model is strong, where it is weak, and which of those claims survive the interval.

Part of [Metrik](https://github.com/Asmodeus14/Metrik).

## The problem this exists for

A benchmark score is one number over a heterogeneous set of tasks. MMLU is 57 subjects; a
single 49.3% says nothing about whether a model is fine at biology and hopeless at law.

Decomposing it is easy, and several tools do. What none of them do is decline to over-read the
decomposition:

- **Ten items per subject gives a 95% interval about 30 points wide.** Ranking 57 subjects by
  point estimate under those conditions is ranking noise.
- **57 tests are not one test.** Testing each subject at α=0.05 with no correction reports
  about three subjects as notable on a model with no subject-level variation at all.

So every subject is measured and reported with a Wilson interval, and only subjects surviving
an exact binomial test with **Holm–Bonferroni correction across the whole family** are
described as notable. On a subsampled run that set is usually empty — and saying so is the
finding, not a failure of the analysis.

## Being wrong is not one thing

For multiple-choice tasks scored by log-probability, the gap between the chosen answer and the
correct one separates a model that was *narrowly* beaten from one that was *confidently*
mistaken. Accuracy cannot see the difference; the per-sample records already contain it.

That breakdown is `null` rather than empty for generative tasks — there are no per-choice
log-probabilities to compare, and an all-zero banding would read as "measured, and every item
was narrow".

## Install

```console
pip install metrik-eval
```

No ML stack. This is a consumer: it reads stored artifacts and their blobs and never loads a
model, which the `consumers-are-read-only` contract enforces rather than merely asks for.

## License

Apache-2.0
