Metadata-Version: 2.4
Name: evaloheval
Version: 0.1.0
Summary: Minimal research-grade evaluation library for ML outputs
Author: Abdulloh
License-Expression: MIT
Project-URL: Homepage, https://github.com/abdullohndm/evaloheval
Project-URL: Repository, https://github.com/abdullohndm/evaloheval
Project-URL: Issues, https://github.com/abdullohndm/evaloheval/issues
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Dynamic: license-file

# evaloheval


[![PyPI Version](https://img.shields.io/pypi/v/evaloheval?color=blue)](https://pypi.org/project/evaloheval/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python Support](https://img.shields.io/badge/python->=3.9-blue.svg)](https://www.python.org/)

`evaloheval` is a minimal, research-grade Python library for ML researchers who need honest, reproducible evaluation statistics for text and code generation tasks.

## Why evaloheval?

When evaluating generation models (like code LLMs on HumanEval or MBPP), researchers typically report a single point estimate for **pass@k**. 

Most stop there. They omit **confidence intervals** and fail to run **significance tests** when comparing models. Reporting results as a single number without variance or statistical validity weakens scientific papers and misleads readers. 

`evaloheval` solves this by providing a lightweight, dependency-light statistics layer that plugs right into your existing evaluation pipelines.

---

## Key Features

1. **Unbiased pass@k Estimation:** Computes the probability that at least one of $k$ sampled outputs is correct, using the unbiased estimator from the original Codex paper (Chen et al., 2021). Works per problem or aggregated across a dataset.
2. **Bootstrap Confidence Intervals:** Wraps any metric (not just pass@k) and returns empirical confidence intervals using the percentile bootstrap method.
3. **Statistical Significance Testing:** Compares two models on the same set of problems to tell you if the performance gap is real or statistical noise. Supports:
   * **Wilcoxon signed-rank test** (paired, default)
   * **Paired Student's t-test**
   * **Paired Bootstrap permutation test**

---

## Installation

Install the package via pip:

```bash
pip install evaloheval
```

Or install in editable mode for local development:

```bash
git clone https://github.com/abdullohndm/evaloheval.git
cd evaloheval
pip install -e ".[dev]"
```

---

## Quick Start

### 1. Single Model Evaluation (pass@k + CI)

Compute the mean pass@k score along with a 95% bootstrap confidence interval across tasks:

```python
from evaloheval import TaskResult, evaluate_passk

# Define outputs for 4 tasks (e.g. 5 generated samples per task)
tasks = [
    TaskResult("task_1", [True, False, False, False, False]),
    TaskResult("task_2", [True, True, False, False, False]),
    TaskResult("task_3", [False, False, False, False, False]),
    TaskResult("task_4", [True, True, True, True, True]),
]

# Run evaluation for k=2
result = evaluate_passk(tasks, k=2, random_state=42)

print(result.to_dict())
# Outputs:
# {
#   'estimate': 0.525, 
#   'ci_lower': 0.2, 
#   'ci_upper': 0.85, 
#   'ci': 0.95, 
#   'method': 'percentile_bootstrap', 
#   'n_resamples': 10000, 
#   'n_tasks': 4
# }
```

### 2. Compare Two Models (Significance Testing)

Compare two models on the same set of tasks to check if the fine-tuned model (Model B) is statistically superior to the baseline (Model A):

```python
from evaloheval import TaskResult, compare

model_a = [
    TaskResult("task_1", [True, False, False, False, False]),
    TaskResult("task_2", [True, True, False, False, False]),
    TaskResult("task_3", [False, False, False, False, False]),
    TaskResult("task_4", [True, True, True, True, True]),
]

model_b = [
    TaskResult("task_1", [True, True, False, False, False]),  # Improved
    TaskResult("task_2", [True, True, True, False, False]),   # Improved
    TaskResult("task_3", [True, False, False, False, False]),  # Improved
    TaskResult("task_4", [True, True, True, True, True]),      # Same
]

# Run Wilcoxon significance test at k=2
res = compare(model_a, model_b, k=2, method="wilcoxon")

print(res.to_dict())
# Outputs:
# {
#   'estimate_a': 0.525,
#   'estimate_b': 0.8,
#   'difference': -0.275,
#   'p_value': 0.0431,
#   'significant': True,
#   'alpha': 0.05,
#   'method': 'wilcoxon',
#   'n_tasks': 4
# }
```

---

## Statistical Methodology

### pass@k Estimator
Following Chen et al., 2021, the unbiased estimator for the probability of drawing at least one correct sample out of $k$ without replacement from a pool of $n$ total samples (where $c$ are correct) is:

$$\text{pass@k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}} = 1 - \prod_{i=0}^{k-1} \frac{n - c - i}{n - i}$$

`evaloheval` implements a numerically stable multiplicative formulation of this product to prevent overflow issues on large sample spaces.

### Bootstrap Confidence Intervals
To avoid assuming a normal distribution for pass@k values across problems, we use the **percentile bootstrap method**. The observed task scores are resampled with replacement $B$ times (default $10,000$). The confidence intervals are constructed from the corresponding quantiles (e.g. 2.5% and 97.5% for a 95% interval) of the resampled means distribution.

### Paired Significance Testing
When comparing Model A and Model B, tasks are automatically aligned by `task_id` and sorted. We compute the paired performance difference $\Delta = \text{pass@k}_A - \text{pass@k}_B$ per task, and run:
* **Wilcoxon signed-rank test:** A non-parametric test comparing the median of the differences to zero.
* **Paired t-test:** A parametric test comparing the mean difference to zero.
* **Paired Bootstrap permutation test:** Shakes up differences by subtracting the observed mean (enforcing the null hypothesis $H_0: \mu = 0$) and counting the proportion of bootstrap iterations where the resampled mean difference is at least as extreme as the observed difference.

---

## Dependencies

`evaloheval` relies on two core libraries:
* **NumPy** for fast, vectorized bootstrapping.
* **SciPy** for statistical tests.

There are no heavy machine learning frameworks or pipeline integrations. Anyone can install and run it in seconds.

---

## License

This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.
