Metadata-Version: 2.4
Name: everyeval
Version: 0.1.0
Summary: Open-source benchmark evaluation with traces: catalog, quote, run, compare.
License-Expression: Apache-2.0
Requires-Dist: alembic>=1.20.0
Requires-Dist: cryptography>=50.0.2
Requires-Dist: fastapi>=0.142.2
Requires-Dist: inspect-ai>=0.3.277
Requires-Dist: numpy>=2.5.3
Requires-Dist: opentelemetry-exporter-otlp-proto-common>=1.45.1
Requires-Dist: opentelemetry-proto>=1.45.1
Requires-Dist: opentelemetry-sdk>=1.45.1
Requires-Dist: pydantic>=2.13.5
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: rich>=14.0.0
Requires-Dist: scipy>=1.18.1
Requires-Dist: sqlalchemy>=2.1.3
Requires-Dist: typer>=0.27.3
Requires-Dist: uvicorn[standard]>=0.54.0
Requires-Dist: inspect-evals>=0.23.0 ; extra == 'benchmarks'
Requires-Dist: every-eval-ever[inspect]>=0.3.0 ; extra == 'eee'
Requires-Dist: harbor>=0.24 ; extra == 'harbor'
Requires-Dist: psycopg[binary]>=3.3.6 ; extra == 'postgres'
Requires-Python: >=3.12
Provides-Extra: benchmarks
Provides-Extra: eee
Provides-Extra: harbor
Provides-Extra: postgres
Description-Content-Type: text/markdown

# everyeval

[![CI](https://github.com/trycedar0x/benchmark-tracing/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/trycedar0x/benchmark-tracing/actions/workflows/ci.yml)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](pyproject.toml)

Run established benchmarks against any model, see a trace for every task, and compare runs with paired statistics. Open source, local-first, no account needed.

![Comparing two runs of the same benchmark: score difference with a confidence interval, McNemar's test, and the per-task regressions and improvements](docs/media/ui-2-compare.png)

everyeval reuses existing evaluation tools instead of rebuilding them: [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) runs the benchmarks and [inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals) supplies them. everyeval adds the parts around them:

- **A versioned catalog.** Each entry pins one benchmark variant (task, arguments, grader). Runs on different variants are never silently compared.
- **Quotes and budget caps.** Run a small sample, get a token and cost estimate with a 95% range, approve it, and the run stops when it reaches its cap.
- **Traces for every task.** Model calls, tool calls, sandbox commands and grading become a span tree linked to the task's outcome.
- **Paired comparison.** Per-task regressions and improvements, a bootstrap confidence interval on the score difference, and McNemar's test. Execution errors are reported separately and never counted as wrong answers.
- **Model identity checks.** Every model call records which model actually answered. A run served by more than one model revision is blocked from strict comparison.

## Quick start (offline, no API keys)

```bash
git clone https://github.com/trycedar0x/benchmark-tracing && cd benchmark-tracing
uv sync

uv run everyeval catalog
uv run everyeval run toy-arith -m mock/strong -m mock/weak
uv run everyeval compare <run-a> <run-b>
uv run everyeval trace <run-b> arith-001
```

![Terminal: comparing two runs, printing the span tree of a failed task, and diffing the two runs' trajectories to the step where they diverged](docs/media/cli-compare-trace-diff.gif)

`mock/*` models are deterministic mock models bundled for demos and tests. `toy-arith` and `toy-tools` are bundled benchmarks; `toy-tools` exercises tool calls.

New here? The [user guide](docs/guide.md) walks through reading comparisons, tracing your own agent and gating CI, and [`examples/`](examples) has runnable CLI, SDK, OTLP and HTTP API examples.

## Real models and benchmarks

```bash
uv sync --extra benchmarks            # installs inspect_evals
export OPENAI_API_KEY=...             # or any provider Inspect supports

uv run everyeval quote create gsm8k -m openai/gpt-4o-mini -m anthropic/claude-haiku-4-5 --limit 200
uv run everyeval quote approve <quote-id> --cap 5
uv run everyeval run --quote <quote-id>
```

Model names follow [Inspect's provider format](https://inspect.aisi.org.uk/models.html). everyeval ships no real provider prices because they change; add the models you use to `~/.everyeval/pricing.yaml` (see [`src/everyeval/pricing.yaml`](src/everyeval/pricing.yaml)). Models without a price get token counts but cannot run with a budget cap.

Commands that call paid models ask for confirmation, or take `--yes`.

## Agent benchmarks with Harbor

Agent benchmarks (Terminal-Bench, SWE-bench Verified, Aider Polyglot) run through [Harbor](https://github.com/laude-institute/harbor) in Docker sandboxes. Each trial becomes a sample with its reward, tokens, cost and an ATIF trajectory converted into model and tool spans.

```bash
uv sync --extra harbor
uv run everyeval run harbor-smoke -m oracle -m nop                 # bundled tasks, no API keys
uv run everyeval run terminal-bench -m terminus-2:openai/gpt-4o --limit 10 --yes
```

Models take the form `agent:model`, or a bare model for the catalog's default agent. `oracle` (the reference solution) and `nop` (does nothing) are free. Harbor resolves registry datasets to their latest version. Every trial's task checksum is recorded, and runs whose task contents differ are blocked from direct comparison. With `--budget`, a run stops when Harbor-reported cost reaches the cap. Agent benchmarks pull large images; SWE-bench needs substantial disk space.

## Web UI and server

```bash
npm --prefix web ci && npm --prefix web run build   # builds the UI into the Python package
uv run everyeval serve                              # http://127.0.0.1:8321, with 2 embedded workers
```

Or run everything with Postgres in Docker. Authentication is on in this setup, so set a secret key and create the first user:

```bash
export EVERYEVAL_SECRET_KEY="$(openssl rand -hex 32)"
docker compose up --build -d
docker compose run --rm api everyeval admin create-user you@example.com --workspace main
```

The API, a worker and Postgres start, and the UI is at http://127.0.0.1:8321. Over the API, paid models need an approved quote (`EVERYEVAL_REQUIRE_QUOTE=0` disables this).

The UI is built with [shadcn/ui](https://ui.shadcn.com) (Nova preset). Browser smoke tests: `npx --prefix web playwright test` against a running server.

## Tracing your own agent

Send traces from your own code with the SDK, or point any OpenTelemetry OTLP/HTTP exporter at `/v1/traces`. Spans from OpenInference and GenAI-semantic-convention instrumentations (OpenAI, Anthropic, LangChain and others) are mapped to model, tool and agent steps.

```python
from everyeval.sdk import EveryEval

ee = EveryEval("http://127.0.0.1:8321", content="metadata")   # or "redacted" / "full"
with ee.trace("answer-question", task_id="q-17"):
    with ee.span("call-model", kind="model", **{"gen_ai.request.model": "gpt-4o-mini"}) as span:
        ee.record(span, "output", "...")                         # kept only if the policy allows
print(ee.flush().receipts)                                       # per-trace receipt: expected vs received spans
```

An [OpenAI Agents SDK agent](examples/openai_agents_trace.py) traced through OpenInference, with its model calls and tool call:

![Span tree of an OpenAI Agents SDK run: agent, turns, model calls with token counts, and a lookup_order tool call](docs/media/ui-9-openai-agents-trace.png)

Delivery is crash-safe. The content policy is applied before anything touches disk. Every batch is written to a local spool (fsync and atomic rename) before it is sent, and is deleted only after the server acknowledges it. Batches left behind by a crash are sent by the next client for the same endpoint. Each batch has a stable id, so a retry after a lost acknowledgement is not stored twice. A full spool refuses new spans and reports them in `ee.health()` instead of silently dropping them. When a `trace` block exits, it seals the trace with its span count. The trace shows as `closed` once every span has arrived, or `incomplete` if some are missing. A closed trace means everything the producer declared was received. It does not prove the instrumentation was complete.

The server stores ingested content according to `EVERYEVAL_INGEST_CONTENT` (default `metadata`). A client can ask for a stricter policy than the server's, but not a looser one.

## Imports and datasets

Import production traces from other tools, then turn interesting ones into reviewed evaluation tasks.

```bash
export LANGFUSE_PUBLIC_KEY=... LANGFUSE_SECRET_KEY=...           # or LANGSMITH_API_KEY, BRAINTRUST_API_KEY
uv run everyeval import langfuse --max-traces 200 --rights own_data
uv run everyeval import langsmith --project support-bot --rights own_data
uv run everyeval import braintrust --project support-bot --rights own_data
uv run everyeval import otlp spans.jsonl --rights own_data      # OTLP/JSON files
uv run everyeval import inspect-log run.eval                   # an existing Inspect log becomes a run

uv run everyeval dataset create "support failures"
uv run everyeval dataset add <dataset> --run <run> --outcome incorrect --split dev
uv run everyeval dataset review <item> --reference "..." --split test --approve
uv run everyeval dataset export <dataset> -o tasks.jsonl       # approved items only, Inspect-compatible
```

Importing requires declaring your usage rights. Re-importing a trace replaces it instead of duplicating it. Dataset items start as drafts. The recorded output is kept as evidence and is never used as the reference answer automatically. An item can only be approved once a reviewer has set a reference answer, assigned a dev or held-out test split, and confirmed usage rights. The same flow is in the web UI under Imports and Datasets.

## Teams: authentication and workspaces

Authentication is off by default (single-user local mode, bound to localhost). For a shared server:

```bash
export EVERYEVAL_AUTH=1 EVERYEVAL_SECRET_KEY="$(openssl rand -hex 32)"
uv run everyeval admin create-user you@example.com --workspace main      # prompts for a password
uv run everyeval admin create-key --workspace main --name ci             # prints a bt_... key once
uv run everyeval admin set-secret OPENAI_API_KEY --workspace main        # encrypted provider key
uv run everyeval serve --host 0.0.0.0
```

- Every run, trace, quote, import and dataset belongs to one workspace. Members only see their own workspaces.
- Roles: owners manage secrets, members create runs and keys, and viewers are read-only.
- The browser uses an HttpOnly session cookie. Scripts, the SDK and OTLP exporters send `Authorization: Bearer bt_...`. Only key hashes are stored.
- Provider keys and import credentials are stored encrypted (Fernet, keyed from `EVERYEVAL_SECRET_KEY`). They are passed only to that workspace's run processes and imports, and are never returned by the API.
- Put TLS in front of the server before exposing it beyond your machine.

The database schema is managed with Alembic migrations, applied automatically on startup.

## Commands

| Command | What it does |
| --- | --- |
| `catalog [BENCHMARK]` | List benchmark variants, or show one |
| `quote create BENCH -m MODEL...` | Run a few samples per model and estimate full-run tokens and cost |
| `quote approve ID --cap USD` | Approve a quote with an optional budget cap |
| `run BENCH -m MODEL...` | Run with live progress; `--limit`, `--epochs`, `--budget`, `--content` |
| `run --quote ID` | Run an approved quote |
| `runs`, `show RUN` | List runs; show a run's summary, versions and samples |
| `trace RUN SAMPLE` | Print the span tree for one task |
| `compare A B` | Paired comparison; `--json` for every row |
| `diff A B SAMPLE` | Where the two runs' trajectories for one sample diverged |
| `cancel RUN` | Stop a running run gracefully |
| `import SOURCE`, `dataset ...` | Import traces; draft, review and export datasets |
| `admin ...` | Create users, workspaces, API keys and encrypted secrets |
| `serve`, `worker` | Start the API and web UI; run queued jobs in a separate process |
| `export RUN` | JSON export, or `--format eee` for [Every Eval Ever](https://github.com/evaleval/every_eval_ever) (needs `--extra eee`; real providers only) |

## How it works

Each run executes in its own child process. Inspect hooks stream every finished sample into the database as it completes, with its result and spans. Budget stops and cancellation send the child a SIGINT, which Inspect handles as a graceful cancel and writes a partial log. Samples interrupted that way are marked `cancelled`, not `error`. With a budget, at most four samples run at once, so a stop overshoots by at most a few in-flight calls.

State lives in `~/.everyeval` (override with `EVERYEVAL_HOME`): a SQLite database and the original Inspect logs, which open in [Inspect View](https://inspect.aisi.org.uk/log-viewer.html). Set `EVERYEVAL_DATABASE_URL` to use Postgres.

everyeval was called benchtrace before. Existing setups keep working: `BENCHTRACE_*` variables are read when the `EVERYEVAL_*` one is unset, `~/.benchtrace` is used until `~/.everyeval` exists, and `bt_` API keys are still accepted.

### Content policy

`--content full` (default for benchmark runs) stores prompts and outputs. `redacted` stores them after best-effort secret redaction. `metadata` stores only timings, token counts, scores and structure. Redaction catches common credential formats and sensitive field names; it cannot catch every secret in free text. Original Inspect logs are kept locally in full regardless of policy.

## Status

Early but complete for its first scope: CLI, API server and job queue, web UI, traces and trace diff, OTLP ingest and SDK, imports and dataset drafts, Harbor agent benchmarks, and workspaces with authentication. Not yet: remote execution backends (Modal, Daytona, Kubernetes), custom HTTPS scorers, and configuration tuning.

## Development

```bash
uv sync --all-extras
uv run pytest                                   # SQLite
EVERYEVAL_TEST_POSTGRES=postgresql+psycopg://postgres:pw@localhost/postgres uv run pytest   # Postgres
uv run ruff check src tests
npm --prefix web ci && npm --prefix web run build
```

## License

Apache-2.0. Benchmark datasets keep their own licenses, listed in the catalog.
