Metadata-Version: 2.4
Name: pytest-llm-assert
Version: 0.2.4
Summary: Simple LLM-powered assertions for any pytest test
Project-URL: Homepage, https://github.com/sbroenne/pytest-llm-assert
Project-URL: Documentation, https://sbroenne.github.io/pytest-llm-assert
Project-URL: Repository, https://github.com/sbroenne/pytest-llm-assert
Author: Stefan Broenner
License-Expression: MIT
License-File: LICENSE
Keywords: ai,assertions,llm,pytest,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Requires-Dist: azure-identity>=1.25
Requires-Dist: pydantic-ai>=1.59.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pytest>=9.0
Provides-Extra: dev
Requires-Dist: pre-commit>=4.5; extra == 'dev'
Requires-Dist: pyright>=1.1.408; extra == 'dev'
Requires-Dist: pytest-cov>=6.0; extra == 'dev'
Requires-Dist: pytest>=9.0; extra == 'dev'
Requires-Dist: python-dotenv>=1.2; extra == 'dev'
Requires-Dist: ruff>=0.14; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.7; extra == 'docs'
Requires-Dist: mkdocs>=1.6; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.24; extra == 'docs'
Description-Content-Type: text/markdown

# pytest-llm-assert

[![PyPI version](https://img.shields.io/pypi/v/pytest-llm-assert)](https://pypi.org/project/pytest-llm-assert/)
[![Python versions](https://img.shields.io/pypi/pyversions/pytest-llm-assert)](https://pypi.org/project/pytest-llm-assert/)
[![CI](https://github.com/sbroenne/pytest-llm-assert/actions/workflows/ci.yml/badge.svg)](https://github.com/sbroenne/pytest-llm-assert/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Natural language assertions for pytest.**

Testing a text-to-SQL agent? Validating LLM-generated code? Checking if error messages are helpful? Now you can:

```python
def test_sql_agent_output(llm):
    sql = my_agent.generate("Get names of users over 21")
    
    assert llm(sql, "Is this a valid SQL query that selects user names filtered by age > 21?")
```

The LLM evaluates your criterion and returns pass/fail — no regex, no parsing, no exact string matching.

## Features

- **Semantic assertions** — Assert meaning, not exact strings
- **Multiple LLM providers** — OpenAI, Azure, Anthropic, Gemini, Groq via [Pydantic AI](https://ai.pydantic.dev/)
- **pytest native** — Works as a standard pytest plugin/fixture
- **Response introspection** — Access tokens and reasoning via `llm.response`
- **Type-safe** — Built with Pydantic for structured outputs

## Installation

```bash
pip install pytest-llm-assert
```

## Quick Start

```python
# conftest.py
import pytest
from pytest_llm_assert import LLMAssert

@pytest.fixture
def llm():
    return LLMAssert(model="openai:gpt-4o-mini")
```

```python
# test_my_agent.py
def test_generated_sql_is_correct(llm):
    sql = "SELECT name FROM users WHERE age > 21 ORDER BY name"
    assert llm(sql, "Is this a valid SELECT query that returns names of users over 21?")

def test_error_message_is_helpful(llm):
    error = "ValidationError: 'port' must be an integer, got 'abc'"
    assert llm(error, "Does this explain what went wrong and how to fix it?")

def test_summary_captures_key_points(llm):
    summary = generate_summary(document)
    assert llm(summary, "Does this mention the contract duration and parties involved?")
```

## Setup

Works out of the box with cloud identity — no API keys to manage:

```bash
# Azure (Entra ID)
export AZURE_API_BASE=https://your-resource.openai.azure.com
az login

# Google Cloud (Vertex AI)
gcloud auth application-default login

# AWS (Bedrock)
aws configure  # Uses IAM credentials
```

Supports multiple providers via [Pydantic AI](https://ai.pydantic.dev/) — including API key auth for OpenAI, Anthropic, and more.

## Documentation

- **[Documentation](https://sbroenne.github.io/pytest-llm-assert)** — Full documentation with examples
- **[Configuration](https://sbroenne.github.io/pytest-llm-assert/configuration/)** — All providers, CLI options, environment variables
- **[API Reference](https://sbroenne.github.io/pytest-llm-assert/api-reference/)** — Full API documentation
- **[Comparing Judge Models](https://sbroenne.github.io/pytest-llm-assert/comparing-models/)** — Evaluate which LLM works best for your assertions
- **[Examples](examples/)** — Working pytest examples
- **[Integration Tests](docs/INTEGRATION_TESTS.md)** — Running tests with real LLM APIs

## Related

- **[pytest-aitest](https://github.com/sbroenne/pytest-aitest)** — Full framework for testing MCP servers, CLIs, and AI agents
- **[Contributing](CONTRIBUTING.md)** — Development setup and guidelines

## Requirements

- Python 3.11+
- pytest 9.0+
- An LLM (OpenAI, Azure, Anthropic, etc.) or local Ollama

## Security

- **Sensitive data**: Test content is sent to LLM providers — consider data policies

## License

MIT
