Metadata-Version: 2.4
Name: athena-eval
Version: 0.1.0
Summary: Executable A/B validity gates (G1-G7) for Athena harnesses
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: llm
Requires-Dist: athena-llm>=0.2.0; extra == "llm"

# athena-eval

Executable A/B validity gates. The arithmetic half of "is this result allowed to decide
anything?", as code that can fail instead of prose in a style guide.

It exists because written-down rules did not stop the mistakes they described. Judgement about
*what* to measure stays human; whether a measured gap is allowed to be called a winner is
arithmetic, and arithmetic should not be re-derived by hand each time — least of all by a party
motivated to find an effect.

## What it refuses to let ship

| Mistake | Guard |
|---|---|
| Declaring a winner when the gap is inside the noise | `verdict()` returns `cannot-separate` |
| Counting ties as evidence in a paired comparison | `paired_verdict()` scores only discordant pairs |
| Reporting a mean with no spread | `Stat` has no mean-only form |
| Grading against inputs that silently arrived empty | `assert_inputs()` prints sizes and refuses |
| Scoring a truncated or empty completion as an answer | `complete_checked()` |

```python
from athena_eval import verdict, paired_verdict, assert_inputs

assert_inputs(prompt=prompt, corpus=corpus)      # prints byte counts, refuses empties
v = verdict("baseline", a_scores, "challenger", b_scores, higher_is_better=True)
print(v.cls)     # "decided" | "cannot-separate" | "VOID"
```

**State the metric direction.** `higher_is_better` has no safe default: a gate that guesses will
confidently name the slower provider the latency winner.

Below n=10 a verdict is `VOID`, not weak evidence. For paired comparisons, fewer than 6 discordant
pairs cannot reach p<0.05 at any split, however lopsided the totals look.

## Install

```
pip install athena-eval                  # gates only, no dependencies
pip install "athena-eval[llm]"           # adds complete_checked, via athena-llm
```

`complete_checked` imports `athena-llm` lazily inside the function, so the gates themselves pull in
nothing.
