Metadata-Version: 2.4
Name: eval-flake-detector
Version: 0.1.0
Summary: Detect flaky LLM eval cases across repeated runs. Pass-rate + standard-deviation per case, with per-case severity. Python port of @mukundakatta/eval-flake-detector.
Project-URL: Homepage, https://github.com/MukundaKatta/eval-flake-detector-py
Project-URL: Issues, https://github.com/MukundaKatta/eval-flake-detector-py/issues
Project-URL: Source, https://github.com/MukundaKatta/eval-flake-detector-py
Project-URL: JS sibling, https://github.com/MukundaKatta/eval-flake-detector
Author-email: Mukunda Katta <mukunda.vjcs6@gmail.com>
License: MIT
License-File: LICENSE
Keywords: agents,ai,claude,evals,evaluation,flaky-tests,llm,openai,rag
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# eval-flake-detector

[![PyPI](https://img.shields.io/pypi/v/eval-flake-detector.svg)](https://pypi.org/project/eval-flake-detector/)
[![Python](https://img.shields.io/pypi/pyversions/eval-flake-detector.svg)](https://pypi.org/project/eval-flake-detector/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)

**Detect flaky LLM eval cases across repeated runs.** Computes per-case pass rate, standard deviation, and a flakiness severity ("stable" / "low" / "medium" / "high"). Pure stdlib, zero runtime dependencies.

Python port of [@mukundakatta/eval-flake-detector](https://github.com/MukundaKatta/eval-flake-detector). The JS-compatible shape is exposed as `detect_eval_flakes` so existing dashboards keep working.

## Install

```bash
pip install eval-flake-detector
```

## Usage

```python
from eval_flake_detector import detect

# One inner list per case; entries are repeat results.
runs = [
    [True, True, True, True, True],            # case 0: stable pass
    [False, False, False, False, False],       # case 1: stable fail
    [True, False, True, False, True, False],   # case 2: flaky 50/50
]

report = detect(runs, flaky_low=0.1, flaky_high=0.9, min_runs=3)

report.stable          # False -- case 2 is flaky
for case in report.flakes:
    print(case.case_id, case.pass_rate, case.std_dev, case.severity)
# case-2  0.5  0.5  high
```

Already have a flat list of result dicts (one row per repeat)? Pass that
directly -- it's auto-grouped by `id`:

```python
runs = [
    {"id": "math-1",   "passed": True},
    {"id": "math-1",   "passed": False},
    {"id": "math-1",   "passed": True},
    {"id": "lookup-2", "score":  0.95},
    {"id": "lookup-2", "score":  1.00},
]
detect(runs)
```

## Severity bands

| Band | Meaning |
|---|---|
| `stable` | Pass rate is outside the ambiguous band -- consistently passes or consistently fails. |
| `low`    | In band, std_dev < 0.30. Probably noise; revisit prompt. |
| `medium` | In band, 0.30 <= std_dev < 0.45. Real flakiness; investigate. |
| `high`   | In band, std_dev >= 0.45. Behaves randomly; consider redesigning the eval. |

## JS-compat output

```python
from eval_flake_detector import detect_eval_flakes

# Same shape as the JS sibling: {"stable": bool, "flakes": [{"id", "count", "stddev"}]}
detect_eval_flakes(runs, max_stddev=0.2)
```

See the JS sibling's [README](https://github.com/MukundaKatta/eval-flake-detector) for the original design notes.
