Metadata-Version: 2.4
Name: trieval
Version: 0.0.1
Summary: RAG evaluation with exactly 6 metrics. 3 variables, 6 relationships, nothing more.
License: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: pydantic>=2.0
Requires-Dist: pydantic-ai>=0.1.0
Requires-Dist: langgraph>=0.2.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.24; extra == "dev"
Requires-Dist: pytest-cov>=6.0; extra == "dev"
Requires-Dist: ruff>=0.8; extra == "dev"
Requires-Dist: mypy>=1.13; extra == "dev"

# trieval

RAG evaluation with exactly 6 metrics. Nothing more.

Every RAG system has 3 variables: **Q** (Question), **C** (Context), **A** (Answer).

3 variables → 6 pairwise relationships → 6 metrics.

## The 6 Metrics

| # | Metric | Notation | Evaluates |
|---|--------|----------|-----------|
| 1 | Context Relevance | C\|Q | Is the retrieved context relevant to the question? |
| 2 | Faithfulness | A\|C | Does the answer stick to the context? |
| 3 | Answer Relevance | A\|Q | Does the answer solve the user's question? |
| 4 | Context Support | C\|A | Does the context fully support the answer? |
| 5 | Answerability | Q\|C | Can this question be answered with this context? |
| 6 | Self-Containment | Q\|A | Can someone understand the question from the answer? |

## Install

```bash
pip install trieval
```

## Quick Start

```python
from trieval import Evaluator

evaluator = Evaluator(model="openai:gpt-4o-mini")
result = await evaluator.evaluate(
    question="What is photosynthesis?",
    context="Photosynthesis is the process by which plants convert sunlight into energy.",
    answer="Photosynthesis is how plants make food from sunlight.",
)

print(result.overall_score)       # 0.0–1.0
print(result.retrieval_score)     # avg of C|Q + Q|C
print(result.generation_score)    # avg of A|C + A|Q
print(result.diagnose())          # ["All metrics healthy"] or failure categories
```

## Failure Diagnosis

When your RAG system fails, it's always one of these:

- **Retrieval issues** — C|Q and Q|C scores are low (wrong context retrieved)
- **Generation issues** — A|C and A|Q scores are low (bad answer generation)
- **End-to-end mismatch** — A|C is fine but C|A is low (faithful but unsupported)

## Architecture

Built with [pydantic-ai](https://ai.pydantic.dev/) (LLM-based metric agents) and [LangGraph](https://langchain-ai.github.io/langgraph/) (evaluation workflow orchestration).

```
RAGInput(Q, C, A)
    ↓
Evaluator.evaluate()
    ↓
LangGraph: evaluate_metrics → diagnose_failures
    ↓
EvaluationResult (scores + diagnosis)
```

Each metric is a pydantic-ai `Agent` with a tailored system prompt. All 6 run concurrently via `asyncio.gather` inside the LangGraph evaluation node.

## Development

```bash
uv sync --group dev
pytest                                    # run tests
pytest --cov=trieval --cov-branch         # with coverage
ruff check trieval/ tests/                # lint
ruff format trieval/ tests/               # format
mypy trieval/                             # type check
```

## Documentation

- [API Reference](docs/api.md) — Full API for `Evaluator`, `EvaluationResult`, `MetricResult`, `RAGInput`, and individual metric functions
- [Changelog](CHANGELOG.md) — Version history

## License

MIT
