Metadata-Version: 2.4
Name: llm-agent-bench
Version: 1.0.0
Summary: Benchmark autonomous AI agents on task completion, tool use, goal adherence, and safety. Works with any agent — just provide a callable.
Author: Linda Oraegbunam
License: MIT
Project-URL: Homepage, https://github.com/obielin/agent-bench
Keywords: llm,agents,benchmark,evaluation,ai-safety,autonomous-agents,tool-use,python
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: black>=24.0.0; extra == "dev"
Requires-Dist: ruff>=0.4.0; extra == "dev"
Dynamic: license-file

# agent-bench

**Benchmark autonomous AI agents on task completion, tool use, goal adherence, and safety. Works with any agent — just provide a callable.**

[![Tests](https://img.shields.io/badge/Tests-38%20passing-brightgreen?style=flat-square)](tests/)
[![Dependencies](https://img.shields.io/badge/Dependencies-zero-brightgreen?style=flat-square)](pyproject.toml)
[![Python](https://img.shields.io/badge/Python-3.10%2B-blue?style=flat-square)](pyproject.toml)
[![License](https://img.shields.io/badge/License-MIT-green?style=flat-square)](LICENSE)
[![LinkedIn](https://img.shields.io/badge/-Linda_Oraegbunam-blue?logo=linkedin&style=flat-square)](https://www.linkedin.com/in/linda-oraegbunam/)

---

## Why agent-bench?

Agents are hard to evaluate. Unlike single LLM calls, agents take multiple steps, call tools, and can drift from their purpose. Most evaluation frameworks require you to restructure your agent. `agent-bench` doesn't. Wrap your agent in a callable and pass it in.

Five evaluation dimensions:

| Dimension | Weight | What it measures |
|---|---|---|
| Task completion | 35% | Did it satisfy success criteria? |
| Tool use | 20% | Did it call the right tools? |
| Goal adherence | 20% | Did it stay on task? |
| Safety | 15% | Was the output safe? |
| Efficiency | 10% | Did it complete within step budget? |

## Install

```bash
pip install agent-bench
```

## Quick start

```python
from agent_bench import AgentBench, Task, AgentResponse

def my_agent(instruction: str) -> AgentResponse:
    result = run_my_agent(instruction)
    return AgentResponse(
        output=result.text,
        tools_called=result.tools_used,
        steps=result.step_count,
    )

bench = AgentBench(pass_threshold=0.7)

report = bench.run(
    agent=my_agent,
    tasks=[
        Task(
            id="research_task",
            instruction="Find the current UK base interest rate",
            expected_tools=["search"],
            success_criteria=["base rate", "Bank of England", "%"],
            max_steps=5,
        ),
    ],
)
print(report.summary())
print(f"Pass rate: {report.pass_rate:.0%}")
print(f"Weakest dimension: {report.weakest_dimension.value}")
```

## Evaluate a single response

```python
result = bench.evaluate_single(task, response)
print(result.overall_score)
print(result.score_by_dimension)
```

---

**Linda Oraegbunam** | [LinkedIn](https://www.linkedin.com/in/linda-oraegbunam/) | [GitHub](https://github.com/obielin)
