Metadata-Version: 2.4
Name: ai-yardstick
Version: 0.0.1
Summary: A CLI tool for running and managing LLM evaluations
Author: Kevin Schaul
License: MIT
Project-URL: Homepage, https://github.com/kevinschaul/ai-yardstick
Project-URL: Changelog, https://github.com/kevinschaul/ai-yardstick/releases
Project-URL: Issues, https://github.com/kevinschaul/ai-yardstick/issues
Project-URL: CI, https://github.com/kevinschaul/ai-yardstick/actions
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click
Requires-Dist: llm>=0.24
Requires-Dist: pandas>=2.2
Requires-Dist: pydantic_evals>=0.1.3
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Dynamic: license-file

# ai-yardstick

[![PyPI](https://img.shields.io/pypi/v/ai-yardstick.svg)](https://pypi.org/project/ai-yardstick/)
[![Changelog](https://img.shields.io/github/v/release/kevinschaul/ai-yardstick?include_prereleases&label=changelog)](https://github.com/kevinschaul/ai-yardstick/releases)
[![Tests](https://github.com/kevinschaul/ai-yardstick/actions/workflows/test.yml/badge.svg)](https://github.com/kevinschaul/ai-yardstick/actions/workflows/test.yml)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/kevinschaul/ai-yardstick/blob/master/LICENSE)

A CLI tool for running and managing LLM evaluations

See [my own evals](https://github.com/kevinschaul/llm-evals/)

## Installation

Install this tool using `pip`:
```bash
pip install ai-yardstick
```
## Usage

Create a new eval with:
```bash
ai-yardstick create EVAL_NAME
```

Then edit prompts.csv, models.csv and tests.csv with your test cases.

Run an eval with:
```bash
ai-yardstick run path/to/config.yaml
```

For help, run:
```bash
ai-yardstick --help
```
You can also use:
```bash
python -m llm_evals_cli --help
```
## Development

To contribute to this tool, first checkout the code. Then create a new virtual environment:
```bash
cd ai-yardstick
python -m venv venv
source venv/bin/activate
```
Now install the dependencies and test dependencies:
```bash
pip install -e '.[test]'
```
To run the tests:
```bash
python -m pytest
```
