Metadata-Version: 2.4
Name: agentproctor
Version: 1.0.1
Summary: The AgentProctor module is a testing and evaluation library designed for AI agents, enabling the testing of various behaviors and properties that cannot be effectively evaluated using standard regression or unit tests. This library provides a structured approach to evaluate AI responses against defined criteria, ensuring that agents behave as expected in diverse scenarios.
Author-email: Diego Agustin Mouriño Rohde Scheel <diego.mourino.rs@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/diego-mourino/agentproctor
Project-URL: Issues, https://github.com/diego-mourino/agentproctor/issues
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: docs/LICENSE.md
Requires-Dist: litellm>=1.83.0
Requires-Dist: pydantic>=2.12.5
Dynamic: license-file

# AgentProctor

![Version](https://img.shields.io/badge/version-1.0.1-blue)
![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)
![Python](https://img.shields.io/badge/python-3.10%2B-blue)
![Build](https://github.com/diego-mourino/AgentProctor/actions/workflows/ci.yml/badge.svg)

## Description
The AgentProctor module is a testing and evaluation library designed for AI agents, enabling the testing of various behaviors and properties that cannot be effectively evaluated using standard regression or unit tests. This library provides a structured approach to evaluate AI responses against defined criteria, ensuring that agents behave as expected in diverse scenarios.

## Why AgentProctor?
Traditional testing breaks down when applied to LLM-based systems.

AgentProctor provides a structured way to:
- Evaluate non-deterministic outputs
- Define reusable evaluation criteria
- Measure response quality with explainability
- Build confidence in AI systems before production

## Documentation

- **[README](docs/README.md)** - Project overview and features
- **[Quickstart](docs/QUICKSTART.md)** - Get started in 5 minutes
- **[Changelog](docs/CHANGELOG.md)** - Version history and updates
- **[License](docs/LICENSE.md)** - MIT License

## Features
- **LLM-native evaluation**: Designed specifically for testing AI-generated outputs
- **Schema Validation**: Utilizes Pydantic for robust validation of input data structures.
- **Custom Evaluators**: Supports the creation of custom evaluators to assess agent responses based on specific criteria.
- **Detailed Reporting**: Generates structured output that includes evaluation results, reasons for success or failure, and confidence levels.
- **Extensible Architecture**: Easily extendable to accommodate new evaluation criteria or agent behaviors.

## Tech Stack
- **Python**: Version 3.10 or higher.
- **LiteLLM**: For lightweight language model interactions.
- **Pydantic**: For data validation and settings management.
- **UV**: For dependencies management.
- **pytest**: For testing framework.
- **Ruff**: For linting and formatting.

## Installation
To install the AgentProctor library, you can use pip. Make sure you have Python 3.10 or higher installed.

```bash
pip install agentproctor
```

This command installs the library along with its development dependencies.

## Usage
To use the AgentProctor library, first import the necessary classes from the package. Below is a simple example of how to create an evaluation request and evaluate it using a custom evaluator.

### Fixed-type evaluator

```python
from agentproctor.schemas import EvaluationRequest
from agentproctor.base_evaluator import BaseEvaluator

# Create an evaluation request
request = EvaluationRequest(
    response="Your agent's response here",
    model="gpt-4o-mini",
    additional_context="Optional context to help the evaluator"
)

# Create a custom evaluator
class MyEvaluator(BaseEvaluator):
    @property
    def criteria(self):
        return "Your evaluation criteria here."

# Evaluate the request
evaluator = MyEvaluator()
result = evaluator.evaluate(request)

print(result)

>>> EvaluationResult(
...     passed=False,
...     reason="A rejection reason.",
...     confidence=0.75,
>>> )
```

### Dynamic type evaluator

```python
from agentproctor.schemas import EvaluationRequest
from agentproctor.base_evaluator import BaseEvaluator

# Create a custom evaluator
class EntityReferenceEvaluator(BaseEvaluator):
    @property
    def criteria(self):
        """
        Defines the evaluation criteria dynamically using parameters provided at initialization.

        The BaseEvaluator accepts arbitrary keyword arguments (`kwargs`) that can be used
        to customize evaluation logic at runtime. This enables reusable evaluators that
        adapt to different inputs without redefining new classes.

        For example, this evaluator checks whether a specific entity is referenced in
        the response by accessing `self.kwargs["entity"]`.
        """
        return f"The user is referencing {self.kwargs["entity"]}"

# Create an evaluation request
request = EvaluationRequest(
    response="I'm speaking about Lionel Messi.",
    model="gpt-4o-mini",
)

# Evaluate the request
evaluator = EntityReferenceEvaluator(entity="Lionel Messi")
result = evaluator.evaluate(request)

print(result)

>>> EvaluationResult(
...     passed=True,
...     reason="The user is clearly speaking about Lionel Messi",
...     confidence=1.0,
>>> )
```

## File Structure Overview
```
.
├── docs
│   ├── LICENSE.md
│   ├── QUICKSTART.md
│   └── README.md
├── notebooks
│   └── use_examples.ipynb
├── src
│   └── agentproctor
│       ├── __init__.py
│       ├── base_evaluator.py
│       ├── schemas.py
│       └── evaluators
│           ├── __init__.py
│           ├── entity_reference.py
│           └── language.py
├── tests
│   ├── conftest.py
│   ├── test_base_evaluator.py
│   ├── TEST_CATALOG.md
│   ├── test_entity_evaluator.py
│   ├── test_evaluation_request.py
│   └── test_language_evaluator.py
├── .gitignore.toml
├── .pre-commit-config.toml
├── pyproject.toml
└── uv.lock
```

- **docs**: Contains the library documentation.
- **notebooks**: Notebooks with worked examples about how to use this module.
- **src/agentproctor/**: Contains the main library code.
- **tests/**: Contains unit tests for the library components.
- **pyproject.toml**: Project metadata and dependencies.

## Design Philosophy
AgentProctor is built around a few core principles:
- **Evaluation > Assertion**: AI systems require qualitative validation, not just equality checks
- **Explainability matters**: Every evaluation should provide reasoning, not just a result.
- **Composable abstractions**: Evaluators should be reusable and easy to extend.
- **Developer experience first**: Minimal boilerplate, maximum clarity.

## Contributing Guidelines
We welcome contributions to improve the AgentProctor library! Here’s how you can help:

1. **Fork the repository**: Click the "Fork" button on the top right of the repository page.
2. **Create a new branch**: Use a descriptive name for your branch.
   ```bash
   git checkout -b feature/your-feature-name
   ```
3. **Make your changes**: Implement your feature or fix.
4. **Write tests**: Ensure that your changes are covered by tests.
5. **Commit your changes**: Write clear and concise commit messages.
   ```bash
   git commit -m "Add feature: your feature description"
   ```
6. **Push to your branch**:
   ```bash
   git push origin feature/your-feature-name
   ```
7. **Create a pull request**: Go to the original repository and create a pull request.

## ❤️ Support the Project
If you find AgentProctor useful, consider supporting its development.

Your support helps:
- Maintain and improve the library
- Add new evaluators and features
- Keep the project actively maintained

👉 Sponsor here: https://github.com/sponsors/diego-mourino

## 👋 About Me

Hi, I'm Diego — an AI Engineer focused on building reliable systems around LLMs and AI agents.

This project is part of my effort to:
- Explore better ways to evaluate AI systems
- Share practical tools with the community
- Demonstrate production-grade engineering practices

## 📬 Contact

💼 LinkedIn: https://www.linkedin.com/in/diego-mourino/?locale=en-US

📧 Email: diego.mourino.rs@gmail.com

If you're interested in collaboration, consulting, or have feedback, feel free to reach out.
