Metadata-Version: 2.5
Name: agentic-evalkit
Version: 0.4.1
Summary: Evaluate agentic systems with reproducible, evidence-first grading.
Project-URL: Changelog, https://github.com/tafreeman/agentic-evalkit/blob/main/CHANGELOG.md
Project-URL: Homepage, https://github.com/tafreeman/agentic-evalkit
Project-URL: Issues, https://github.com/tafreeman/agentic-evalkit/issues
Project-URL: Repository, https://github.com/tafreeman/agentic-evalkit
Project-URL: Security, https://github.com/tafreeman/agentic-evalkit/blob/main/SECURITY.md
Author-email: Andy Freeman <tandfreeman@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: agents,benchmarking,evaluation,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: httpx<1,>=0.28.1
Requires-Dist: huggingface-hub<2,>=0.20
Requires-Dist: jinja2<4,>=3
Requires-Dist: pydantic<3,>=2
Requires-Dist: pyyaml<7,>=6
Requires-Dist: rich<16,>=13
Requires-Dist: typer<1,>=0.12
Provides-Extra: claude
Requires-Dist: claude-agent-sdk<1,>=0.2; extra == 'claude'
Provides-Extra: langfuse
Requires-Dist: langfuse<5,>=3.3.1; extra == 'langfuse'
Provides-Extra: mlflow
Requires-Dist: mlflow-skinny<4,>=3; extra == 'mlflow'
Provides-Extra: swebench
Requires-Dist: docker<8,>=7.1; extra == 'swebench'
Requires-Dist: swebench<5,>=4.1; extra == 'swebench'
Description-Content-Type: text/markdown

# agentic-evalkit

`agentic-evalkit` grades AI agent output against checks you define, and writes up the evidence behind each score.

Use it from the command line or Python to run repeatable evaluations, apply
objective checks, compare results, and generate reports for review. It can
evaluate Python code, command-line programs, and HTTP services without tying
them to a specific AI framework.

Objective checks run before optional model-based judging. The package includes
tested support for calibrated judge evidence while leaving model-provider
selection to the caller. The built-in reference judge always returns the
same verdict for the same input, and it's advisory only; it cannot approve
a release.

**Why not promptfoo, Inspect, DeepEval, Braintrust, or LangSmith?**
Those tools handle the *workflow* side of evals — running lots of prompts,
wiring checks into CI, tracking experiments over time. Use one of them for
that. This package solves a narrower, stricter problem: trusting the
result you get, and these are the controls `agentic-evalkit` itself
enforces to do it, shipping as of the current release (0.4.1 on PyPI):

- An AI judge has to prove it agrees with real human-labeled answers before it's allowed to approve anything
- Two runs are only compared once we can prove they ran under matching conditions
- A bug in your own code is never confused with the AI simply getting the answer wrong
- A fuzzy AI opinion can never override a hard requirement like "the code must compile"
- On grounded-citation tasks, a planted marker echoed back in an answer is caught before it inflates a score

A documentation review of promptfoo, Inspect, DeepEval, Braintrust,
LangSmith, MLflow, and Langfuse's own primary docs (most recently
2026-08-04) found that none of them documents an equivalent control as a
core concept. Exactly what was checked, on what date, what was *not*
tested (no source audit was performed on any competing project), and where
each of those tools is the better choice instead: [docs/prior-art.md](docs/prior-art.md).

Start with the [quickstart guide](docs/guides/quickstart.md). Design
boundaries: [architecture specification](docs/specs/2026-07-02-agentic-evalkit-design.md).

## Quickstart

```bash
pip install agentic-evalkit
agentic-evalkit doctor
agentic-evalkit init --preset gsm8k --output eval.yaml
agentic-evalkit run eval.yaml --limit 5 --yes
```

This resolves the curated GSM8K preset from Hugging Face, runs five samples
through the packaged smoke target, grades them with a normalized exact-match
grader, and writes a canonical JSON report. No importer code, manual dataset
download, `datasets`, `pyarrow`, or Docker is required.

The run reports `passed=0 failed=5` — that's expected, not a bug. The
packaged demo target (`zero_target`) always answers `"0"`, so it always
fails GSM8K grading; the point of this walkthrough is to verify the
*pipeline* — live dataset resolution, target execution, objective grading,
and report generation — before you wire in a real system under test. See
[docs/guides/quickstart.md](docs/guides/quickstart.md) for the full
walkthrough, including the standalone `report` command that regenerates a
self-contained HTML report from that JSON. To see the report shape before
installing anything, a real evaluation run's checked-in JSON/HTML/Markdown
reports are at
[scripts/reports/2026-07-26-agent-workflow-eval/](scripts/reports/2026-07-26-agent-workflow-eval/README.md).

## Python API

The CLI is built on a small, curated Python API for integrations that need
more than the built-in presets: wrap your own system as a target, then
describe a run with a typed manifest.

```python
from agentic_evalkit import CallableTarget, DatasetRef, EvalRunManifest, EvalRunner

def my_system(sample_input: dict) -> dict:
    return {"answer": solve(sample_input["question"])}

target = CallableTarget(my_system, name="my-system")
manifest = EvalRunManifest(
    run_name="quickstart", adapter="gsm8k@1", grader="normalized-exact@1",
    target_name="my-system", dataset_ref=DatasetRef(provider="huggingface", dataset_id="openai/gsm8k"),
)
```

**This snippet is abbreviated for brevity** — it omits the catalog, adapter,
grader, and artifact-store wiring a real run needs. See
[the HTTP agent example](docs/guides/http-agent-example.md) for the complete,
runnable Python-API script.

`EvalRunner(...).run(manifest)` then drives dataset resolution, execution,
and grading end to end. `CallableTarget` satisfies `ExecutionTarget` —
agentic-evalkit's only system-under-test boundary — which is also exported
at the top level for anyone implementing a custom target. Everything
else — additional targets, graders, reporters, dataset providers,
benchmark adapters, and statistics — is one import away under its own
subpackage (`agentic_evalkit.graders`, `agentic_evalkit.reporters`, and so
on).

## Optional extras

`agentic-evalkit` declares four optional extras; the base install still
ships without Docker or any model-provider SDK — see
[ADR-0009](docs/adr/0009-optional-dependencies-and-plugins.md) for the
extras policy.

- **`swebench`** (`pip install agentic-evalkit[swebench]`, pulling in
  `swebench>=4.1,<5` and `docker>=7.1,<8`) backs
  `SweBenchDockerHarnessExecutor`, the container-based SWE-bench Verified
  harness executor landed in
  [ADR-0014](docs/adr/0014-swebench-docker-harness-executor.md): with the
  extra installed and a reachable Docker daemon, `swebench-harness@1` grades
  a real resolved/unresolved verdict instead of reporting `unavailable`.
- **`claude`** (`pip install agentic-evalkit[claude]`, pulling in
  `claude-agent-sdk`) backs `ClaudeAgentTarget`
  ([ADR-0025](docs/adr/0025-claude-subscription-execution-target.md)), which
  grades Claude itself through a Claude subscription sign-in rather than an
  API key.
- **`mlflow`** and **`langfuse`** (`pip install agentic-evalkit[mlflow]` or
  `agentic-evalkit[langfuse]`) back the calibration-gating bridges in
  [the MLflow & Langfuse guide](docs/guides/mlflow-langfuse-bridge.md)
  ([ADR-0022](docs/adr/0022-host-platform-integration-boundary.md)).

## Documentation

- [Quickstart](docs/guides/quickstart.md) — install to first report
- [CLI reference](docs/guides/cli-reference.md) — commands, options, offline behavior, and exit codes
- [Providers](docs/guides/providers.md) — local formats, Hugging Face auth, cache/offline
- [Graders](docs/guides/graders.md) — objective-first order, hard gates, calibrated judges
- [Targets](docs/guides/targets.md) — callable, subprocess, HTTP, MCP-stdio, and Claude Agent SDK execution targets
- [SWE-bench](docs/guides/swebench.md) — preview/prediction workflow and the harness boundary
- [HTTP agent example](docs/guides/http-agent-example.md) — evaluating a real HTTP agent endpoint
- [MLflow & Langfuse bridge](docs/guides/mlflow-langfuse-bridge.md) — gate judge authority in MLflow and Langfuse without leaving either platform
- [Example report](scripts/reports/2026-07-26-agent-workflow-eval/README.md) — a real 48-case run's write-up, with the canonical JSON/HTML/Markdown reports it came from

## Repository boundary

This project imports no host-repo internals — systems are reached only through the public `ExecutionTarget` protocol (callable, subprocess, HTTP, MCP-stdio, or Claude Agent SDK adapters); see [ADR-0001](docs/adr/0001-standalone-boundary.md) and [ADR-0006](docs/adr/0006-execution-target-boundary.md).
