Metadata-Version: 2.4
Name: openagent-eval
Version: 0.4.8
Summary: Open-source CLI framework for evaluating RAG systems and AI Agents
Project-URL: Homepage, https://github.com/OpenAgentHQ/openagent-eval
Project-URL: Repository, https://github.com/OpenAgentHQ/openagent-eval
Project-URL: Changelog, https://github.com/OpenAgentHQ/openagent-eval/blob/main/CHANGELOG.md
Project-URL: Documentation, https://openagenthq.github.io/openagent-eval/
Author: Himanshu Kumar
License: Apache-2.0
License-File: LICENSE
Keywords: ai,cli,corpus-health,diagnosis,evaluation,llm,metrics,nli,rag,synthetic-data
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27.0
Requires-Dist: jinja2>=3.1.0
Requires-Dist: loguru>=0.7.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.12.0
Provides-Extra: all
Requires-Dist: anthropic>=0.40.0; extra == 'all'
Requires-Dist: datasets>=2.0.0; extra == 'all'
Requires-Dist: deepeval>=1.0.0; extra == 'all'
Requires-Dist: elasticsearch>=8.0.0; extra == 'all'
Requires-Dist: evaluate>=0.4.0; extra == 'all'
Requires-Dist: faiss-cpu>=1.7.0; extra == 'all'
Requires-Dist: google-genai>=0.1.0; extra == 'all'
Requires-Dist: groq>=0.4.0; extra == 'all'
Requires-Dist: mypy>=1.10.0; extra == 'all'
Requires-Dist: numpy>=1.24.0; extra == 'all'
Requires-Dist: openai>=1.0.0; extra == 'all'
Requires-Dist: pgvector>=0.2.0; extra == 'all'
Requires-Dist: pinecone-client>=2.2.0; extra == 'all'
Requires-Dist: psycopg[binary]>=3.1.0; extra == 'all'
Requires-Dist: pypdf>=4.0.0; extra == 'all'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'all'
Requires-Dist: pytest-cov>=5.0.0; extra == 'all'
Requires-Dist: pytest>=8.0.0; extra == 'all'
Requires-Dist: qdrant-client>=1.7.0; extra == 'all'
Requires-Dist: ragas>=0.1.0; extra == 'all'
Requires-Dist: rank-bm25>=0.2.0; extra == 'all'
Requires-Dist: ruff>=0.4.0; extra == 'all'
Requires-Dist: scikit-learn>=1.0.0; extra == 'all'
Requires-Dist: scikit-learn>=1.3.0; extra == 'all'
Requires-Dist: sentence-transformers>=2.0.0; extra == 'all'
Requires-Dist: tiktoken>=0.5.0; extra == 'all'
Requires-Dist: torch>=2.0.0; extra == 'all'
Requires-Dist: transformers>=4.30.0; extra == 'all'
Requires-Dist: weaviate-client>=4.5.0; extra == 'all'
Provides-Extra: bm25
Requires-Dist: rank-bm25>=0.2.0; extra == 'bm25'
Provides-Extra: corpus
Requires-Dist: numpy>=1.24.0; extra == 'corpus'
Requires-Dist: scikit-learn>=1.3.0; extra == 'corpus'
Provides-Extra: datasets
Requires-Dist: datasets>=2.0.0; extra == 'datasets'
Provides-Extra: dev
Requires-Dist: mypy>=1.10.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Provides-Extra: elasticsearch
Requires-Dist: elasticsearch>=8.0.0; extra == 'elasticsearch'
Provides-Extra: evaluation
Requires-Dist: deepeval>=1.0.0; extra == 'evaluation'
Requires-Dist: evaluate>=0.4.0; extra == 'evaluation'
Requires-Dist: ragas>=0.1.0; extra == 'evaluation'
Requires-Dist: scikit-learn>=1.0.0; extra == 'evaluation'
Requires-Dist: sentence-transformers>=2.0.0; extra == 'evaluation'
Provides-Extra: faiss
Requires-Dist: faiss-cpu>=1.7.0; extra == 'faiss'
Provides-Extra: nli
Requires-Dist: torch>=2.0.0; extra == 'nli'
Requires-Dist: transformers>=4.30.0; extra == 'nli'
Provides-Extra: pdf
Requires-Dist: pypdf>=4.0.0; extra == 'pdf'
Provides-Extra: pgvector
Requires-Dist: pgvector>=0.2.0; extra == 'pgvector'
Requires-Dist: psycopg[binary]>=3.1.0; extra == 'pgvector'
Provides-Extra: pinecone
Requires-Dist: pinecone-client>=2.2.0; extra == 'pinecone'
Provides-Extra: providers
Requires-Dist: anthropic>=0.40.0; extra == 'providers'
Requires-Dist: google-genai>=0.1.0; extra == 'providers'
Requires-Dist: groq>=0.4.0; extra == 'providers'
Requires-Dist: openai>=1.0.0; extra == 'providers'
Requires-Dist: tiktoken>=0.5.0; extra == 'providers'
Provides-Extra: qdrant
Requires-Dist: qdrant-client>=1.7.0; extra == 'qdrant'
Provides-Extra: weaviate
Requires-Dist: weaviate-client>=4.5.0; extra == 'weaviate'
Description-Content-Type: text/markdown

<div align="center">

# OpenAgent Eval

**The open-source evaluation framework for RAG systems and AI Agents.**

[![PyPI Version](https://img.shields.io/pypi/v/openagent-eval?color=blue&logo=pypi&logoColor=white)](https://pypi.org/project/openagent-eval/)
[![Python Versions](https://img.shields.io/pypi/pyversions/openagent-eval?logo=python&logoColor=white)](https://pypi.org/project/openagent-eval/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
[![Downloads](https://static.pepy.tech/personalized-badge/openagent-eval?period=total&units=INTERNATIONAL_SYSTEM&left_color=grey&right_color=green&left_text=downloads)](https://pepy.tech/projects/openagent-eval)
[![Code Style](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://docs.astral.sh/ruff/)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
[![GitHub Stars](https://img.shields.io/github/stars/OpenAgentHQ/openagent-eval?style=social)](https://github.com/OpenAgentHQ/openagent-eval/stargazers)
[![Forks](https://img.shields.io/github/forks/OpenAgentHQ/openagent-eval?style=social)](https://github.com/OpenAgentHQ/openagent-eval/network/members)
[![Contributors](https://img.shields.io/github/contributors/OpenAgentHQ/openagent-eval?style=social)](https://github.com/OpenAgentHQ/openagent-eval/graphs/contributors)
[![Issues](https://img.shields.io/github/issues/OpenAgentHQ/openagent-eval)](https://github.com/OpenAgentHQ/openagent-eval/issues)

Local-first. Framework-agnostic. Developer-friendly.

[Getting Started](#installation) · [Documentation](https://openagenthq.github.io/openagent-eval/) · [Contributing](#contributing)

</div>

---

## Why OpenAgent Eval?

Evaluating RAG systems shouldn't require a PhD or a cloud account. OpenAgent Eval brings **pytest-level simplicity** to AI evaluation — run from your terminal, get actionable insights, and ship with confidence.

- **Local-first** — No cloud services, dashboards, or authentication required
- **Framework-agnostic** — Works with LangChain, LlamaIndex, or any custom RAG pipeline
- **18+ metrics** — Retrieval, generation, faithfulness, relevancy, performance, and cost
- **Plugin-based** — Extend with custom metrics, providers, and report generators
- **Production-ready** — Corpus auditing, failure diagnosis, and synthetic test data generation

---

## Installation

```bash
pip install openagent-eval
```

For development:

```bash
git clone https://github.com/OpenAgentHQ/openagent-eval.git
cd openagent-eval
uv sync
```

---

## Quick Start

### 1. Initialize Configuration

```bash
oaeval init --interactive
```

### 2. Validate Configuration

```bash
oaeval validate config.yaml
```

### 3. Run Evaluation

```bash
oaeval run config.yaml
```

### 4. View Results

```bash
oaeval report latest
```

---

## Features

| Feature | Description |
|---------|-------------|
| **CLI + SDK** | Use via command line or import as a Python library |
| **Beautiful Reports** | Terminal, Markdown, HTML, and JSON output formats |
| **Failure Analysis** | Identify why evaluations fail, not just that they failed |
| **Corpus Health Auditor** | Detect contradictions, staleness, and duplicates before evaluation |
| **LLM-as-Judge Metrics** | NLI-based scoring for faithfulness and relevancy |
| **Component Diagnosis** | Blame attribution — retrieval vs generation vs chunking |
| **Synthetic Test Data** | Auto-generate test cases from your knowledge base |

---

## Evaluation Metrics

<details>
<summary><strong>Retrieval Metrics</strong></summary>

- Context Precision & Recall
- Precision@K & Recall@K
- Hit Rate
- Mean Reciprocal Rank (MRR)
- Normalized Discounted Cumulative Gain (NDCG)

</details>

<details>
<summary><strong>Generation Metrics</strong></summary>

- Faithfulness (NLI-based)
- Answer Relevancy (NLI-based)
- Hallucination Detection
- Semantic Similarity
- Exact Match & F1 Score
- BLEU & ROUGE
- BERTScore
- LLM-as-Judge (custom criteria)

</details>

<details>
<summary><strong>Performance & Cost</strong></summary>

- Latency tracking (embedding, retrieval, LLM stages)
- Token counting (prompt, completion, total)
- Cost estimation per provider

</details>

---

## Supported Providers

**LLM Providers:** OpenAI · Anthropic · Google Gemini · Groq · OpenRouter · Ollama

**Retriever Providers:** Chroma · Qdrant · Pinecone · Weaviate · FAISS · pgvector · Elasticsearch · BM25

---

## CLI Reference

| Command | Description |
|---------|-------------|
| `oaeval init` | Create configuration file (interactive wizard) |
| `oaeval run <config>` | Run evaluation pipeline |
| `oaeval report <id>` | View evaluation reports |
| `oaeval compare <a> <b>` | Compare two experiments |
| `oaeval list` | List previous evaluations |
| `oaeval validate <config>` | Validate configuration |
| `oaeval doctor` | Check environment and dependencies |
| `oaeval audit --corpus <path>` | Audit corpus health |
| `oaeval diagnose --report <id>` | Diagnose failures and attribute blame |
| `oaeval synth --corpus <path>` | Generate synthetic test cases |

---

## SDK Usage

```python
from openagent_eval.core import Engine
from openagent_eval.config import load_config

config = load_config("config.yaml")
engine = Engine(config)
report = await engine.run(dataset)

print(report.summary)
```

---

## Project Structure

```
openagent-eval/
├── openagent_eval/
│   ├── cli/              # CLI commands (Typer)
│   ├── config/           # Configuration system (Pydantic)
│   ├── core/             # Core orchestration engine
│   ├── metrics/          # 18+ evaluation metrics
│   ├── providers/        # LLM & Retriever adapters
│   ├── corpus/           # Corpus Health Auditor
│   ├── diagnosis/        # Component Diagnosis
│   ├── synthesis/        # Synthetic Test Data
│   ├── reports/          # Report generators
│   └── plugins/          # Plugin system
├── tests/                # Test suite
├── docs/                 # Documentation
└── examples/             # Tutorials and examples
```

---

## Contributing

We welcome contributions of all kinds. Whether you're fixing a bug, adding a feature, or improving documentation — we'd love your help.

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.

---

## Community

- [Documentation](https://openagenthq.github.io/openagent-eval/)
- [GitHub Issues](https://github.com/OpenAgentHQ/openagent-eval/issues) — Bug reports and feature requests
- [GitHub Discussions](https://github.com/OpenAgentHQ/openagent-eval/discussions) — Ideas and questions
- [Changelog](CHANGELOG.md) — Release history

---

## License

Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for details.

---

<div align="center">

**Built with care by the [OpenAgent](https://github.com/OpenAgentHQ) community.**

</div>
