Metadata-Version: 2.4 Name: aehf Version: 0.2.1 Summary: agent evaluation harness framework Project-URL: Homepage, https://github.com/salasya2/aehf Project-URL: Repository, https://github.com/salasya2/aehf Author-email: Sai Teja Alasyam License-Expression: MIT License-File: LICENSE Requires-Python: >=3.11 Requires-Dist: anthropic Requires-Dist: pydantic>=2.7 Requires-Dist: python-dotenv Requires-Dist: pyyaml Requires-Dist: typer Provides-Extra: dev Requires-Dist: mypy; extra == 'dev' Requires-Dist: pytest; extra == 'dev' Requires-Dist: pytest-asyncio; extra == 'dev' Requires-Dist: ruff; extra == 'dev' Requires-Dist: types-pyyaml; extra == 'dev' Provides-Extra: openai Requires-Dist: openai; extra == 'openai' Description-Content-Type: text/markdown ![demo](https://raw.githubusercontent.com/salasya2/aehf/main/docs/demo.gif) # aehf — agent evaluation harness framework A framework-agnostic harness for evaluating tool-using LLM agents: run a suite of cases, judge the transcripts, and get **statistically honest** pass rates and regression checks — not a single flaky number. The point of aehf is not just to score agents, but to **validate the judge doing the scoring first**, then use that trusted judge to make claims you can defend. ## Headline finding On 90 hand-labeled agent transcripts: | Judge | Raw agreement | Cohen's kappa | |----------------|---------------|---------------| | AssertionJudge | 97% | **0.000** | | LLMJudge (v1) | 97% | **0.894** | Both judges agree with the human ~97% of the time, yet one is worthless and one is excellent. Raw agreement doesn't correct for chance given the base rates; **Cohen's kappa does.** The assertion judge passes essentially everything, so its 97% is meaningless (kappa 0); the calibrated LLM judge reaches "almost perfect" agreement. This gap is the whole reason the harness measures kappa before trusting any downstream number. (Details and caveats: `docs/calibration.md`.) ## Architecture — everything is a protocol The eval core is decoupled from any provider by three Python `Protocol`s. The runner, judges, and stats know only the interfaces: ```text EvalCase (YAML) | v runner | v Agent.run(case) -> Transcript <-- Agent protocol | AnthropicAdapter, OpenAIAdapter, FakeAgent, | or bring your own v ToolProvider.execute(name, args) <-- ToolProvider protocol | mock / record / replay v Judge.score(case, transcript) <-- Judge protocol | AssertionJudge, LLMJudge v Verdict | +----------------+----------------+ v v v stats calibration regression Wilson, McNemar, kappa vs human store, diff, CI gate n=k ``` The agent under test is a black box behind `Agent`; aehf evaluates whatever implements `run(case) -> Transcript`. The shipped `AnthropicAdapter` and `OpenAIAdapter` are reference implementations, not the framework. ## What's inside - **Runner** — async, budget/timeout-enforced, captures agent crashes as failed cases (never harness crashes), bounded concurrency. - **Adapters** — Anthropic and OpenAI, each enforcing the same step/token budgets and mapping provider-specific stop reasons onto one `Termination` enum. - **Tools** — mock fixtures, plus record/replay for deterministic offline reruns. - **Judges** — programmatic assertions and a versioned LLM judge with structured (forced-tool) verdicts; calibrated against human labels with Cohen's kappa. Bring your own judge prompt with `--judge-prompt-file`, then measure *its* kappa before trusting it. - **Stats** — n-sample execution, per-case Wilson confidence intervals, flakiness flags, and McNemar's exact paired test for model/prompt comparison. - **Regression** — results store keyed by (git SHA, model, judge version), `aehf diff`, a markdown scorecard, and a PR-gate GitHub Action. ## Install ```bash pip install aehf # core (Anthropic adapter included) pip install "aehf[openai]" # + the OpenAI adapter pip install -e ".[dev]" # editable, with dev tooling ``` Set `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` (a `.env` file is loaded automatically) for anything that runs a live model. The test suite runs **green without a key** — CI is free and offline; the two tests that do hit a provider are marked `live` and deselected with `-m "not live"`. ## Quickstart ```bash # run a suite with mock tools, assertion judge (no API cost for the judge) aehf run examples/calibration_suite.yaml anthropic mock --judgechoice assertion # n-sampled run: per-case pass rate + Wilson CI + flakiness aehf run examples/calibration_suite.yaml anthropic mock --n-samples 5 # same suite, OpenAI instead (--model is required: the default is Anthropic's) aehf run examples/calibration_suite.yaml openai mock --model gpt-4o-mini # calibrate a judge against human labels -> Cohen's kappa + disagreements aehf calibrate labels/labels_filled.jsonl llm --prompt-version v1 # bring your own judge prompt, then calibrate it before trusting its verdicts aehf calibrate labels/labels_filled.jsonl llm --judge-prompt-file my_judge.txt # compare two saved runs with McNemar's test aehf compare runA.json runB.json # regression diff between two stored runs (CI gate) aehf diff --store .aehf ``` Full command list: `run`, `calibrate`, `export-labels`, `label`, `compare`, `diff`. ### Custom judge prompts `--judge-prompt-file` (on both `run` and `calibrate`) swaps in your own rubric. The file must contain the `{task}`, `{rubric}`, and `{transcript}` placeholders — aehf checks up front and names the missing ones rather than failing mid-suite. The prompt's identity is its **content hash**, not its filename: a custom prompt is stored as `file--`, so editing it produces a new judge version. That keeps `aehf diff` honest — it can never compare two runs graded by different prompts and blame the difference on your agent. An uncalibrated judge is an unvalidated instrument. Run `calibrate` with the same `--judge-prompt-file` and check its kappa before you trust a number it produces. ## Two findings, in one narrative 1. **Judge calibration** — assertions kappa=0.00 -> LLM judge kappa=0.894 (n=90). The instrument is validated. 2. **Model comparison** — using that validated judge, Haiku and Sonnet were **statistically indistinguishable** on the suite (15/18 vs 14/18, McNemar p=1.0). The apparent edge is noise; a single-run comparison would have misled. (`docs/calibration.md`.) Both come with honest caveats stated in the docs (single annotator; a saturated suite that can't discriminate the two models). Reporting the null result honestly is the point — see `docs/decision.md` for every design choice and rejected alternative. ## Scope (v0.2) - Ships Anthropic and OpenAI reference adapters behind the same `Agent` protocol — the framework-agnostic claim, demonstrated rather than asserted. The judge is Anthropic-backed; a provider-agnostic judge is the next step. - The CI eval gate compares **committed** baseline runs (golden-file style), because replay pins tool results but not the model. See `docs/eval-gate.md`. - Tested: 128 offline tests, `mypy --strict` clean, ruff clean.