Metadata-Version: 2.5
Name: cat-experiments
Version: 0.0.13
Summary: Standalone evaluation engine for LLM applications
Project-URL: Homepage, https://github.com/thisisartium/cat-experiments
Project-URL: Repository, https://github.com/thisisartium/cat-experiments
Project-URL: Issues, https://github.com/thisisartium/cat-experiments/issues
Author-email: CAT Cafe Team <team@cat-cafe.dev>
License: MIT
License-File: LICENSE
Keywords: ai,evaluation,llm,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: typing-extensions>=4.5.0
Provides-Extra: cat-cafe
Requires-Dist: cat-cafe-client>=0.0.4; extra == 'cat-cafe'
Requires-Dist: opentelemetry-api>=1.36.0; extra == 'cat-cafe'
Requires-Dist: opentelemetry-sdk>=1.36.0; extra == 'cat-cafe'
Provides-Extra: phoenix
Requires-Dist: arize-phoenix-client>=1.22.0; extra == 'phoenix'
Provides-Extra: tracing
Requires-Dist: opentelemetry-api>=1.36.0; extra == 'tracing'
Requires-Dist: opentelemetry-sdk>=1.36.0; extra == 'tracing'
Description-Content-Type: text/markdown

# cat-experiments

Agnostic experiment runner for LLM applications that you can take to any server stack.

![Cat Experiments](cat_experiments_small.png)

Most experiment frameworks are glued to a specific hosted platform, forcing you to swap libraries when you switch servers. `cat-experiments` keeps the core experiment loop (data model, runner, evaluators) identical whether you are running locally or wiring into Phoenix, CAT Cafe, or another backend. That gives teams a common starting point for new projects while still letting them plug into whichever server platform fits the deployment.

## Features

- **CLI-First Design**: Define experiments as Python files and run them with `cat-experiments run`
- **Flexible Data Models**: Support any dataset structure with dictionary-based input/output
- **Deterministic Preview Runs**: Limit execution to an exact number of examples with `--dry-run`
- **Explicit Repetitions**: Run each example multiple times with `--repetitions`
- **Comprehensive Evaluators**: Built-in evaluators for tool call correctness and more
- **Modern Python**: Targets Python 3.10+ with modern typing features
- **Async Support**: Full async/await support for evaluation pipelines
- **Tool Call Evaluation**: Advanced matching algorithms for tool call correctness

## Install

```bash
# from PyPI
pip install cat-experiments              # core package
pip install "cat-experiments[cat-cafe]"  # add extras for CAT Cafe
pip install "cat-experiments[phoenix]"   # add extras for Phoenix
```

### CLI via Homebrew

The standalone Go CLI is also distributed via a Homebrew tap:

```bash
brew tap thisisartium/tap
brew install cat-experiments
```

Released for macOS (arm64, x86_64) and Linux (arm64, x86_64). Released independently of the Python and Node SDKs — see `cli-v*` tags on GitHub Releases.

### .NET SDK via NuGet

```bash
dotnet add package CatExperiments
```

See [`dotnet/CatExperiments/README.md`](dotnet/CatExperiments/README.md) for the canonical project shape.

## Quick Start

Create an experiment file:

```python
# my_experiment.py
from cat.experiments.protocol import TaskInput, TaskOutput, EvalInput, EvalOutput
from cat.experiments.sdk import task, evaluator

@task
async def my_task(input: TaskInput) -> TaskOutput:
    """The system under test."""
    question = input.input.get("question", "")
    # In a real experiment, you'd call your LLM here
    return TaskOutput(output={"answer": question.upper()})

@evaluator
def exact_match(input: EvalInput) -> EvalOutput:
    """Check if actual matches expected."""
    expected = input.expected_output.get("answer", "") if input.expected_output else ""
    actual = input.actual_output.get("answer", "") if input.actual_output else ""
    score = 1.0 if expected == actual else 0.0
    return EvalOutput(
        score=score,
        label="match" if score == 1.0 else "mismatch",
    )
```

Run it:

```bash
# With a local JSON/JSONL dataset
cat-experiments run my_experiment.py --dataset data.jsonl

# With dry-run mode (run only N examples, no persistence)
cat-experiments run my_experiment.py --dataset data.jsonl --dry-run 5

# With multiple repetitions
cat-experiments run my_experiment.py --dataset data.jsonl --repetitions 3

# With parallel workers
cat-experiments run my_experiment.py --dataset data.jsonl --max-workers 10

# Stream results to Phoenix
cat-experiments run my_experiment.py --dataset my-dataset --backend phoenix

# Stream results to CAT Cafe
cat-experiments run my_experiment.py --dataset my-dataset --backend cat-cafe
```

## Dataset Format

Datasets are JSON or JSONL files with examples containing `input`, `output`, and optional `metadata`:

```json
{"input": {"question": "How do I reset my password?"}, "output": {"answer": "Visit settings."}, "metadata": {"category": "support"}}
{"input": {"question": "Can I upgrade mid-cycle?"}, "output": {"answer": "Yes, prorated."}, "metadata": {"category": "billing"}}
```

## Writing Experiments

### Task Function

The `@task` decorator marks your system under test. It receives a `TaskInput` with the example data and returns the output:

```python
from cat.experiments.protocol import TaskInput, TaskOutput
from cat.experiments.sdk import task

@task
async def my_llm_task(input: TaskInput) -> TaskOutput:
    """Call your LLM or agent here."""
    question = input.input.get("question", "")
    params = input.params  # Access experiment params (e.g., model name)

    # Your LLM call here
    response = await call_my_llm(question, model=params.get("model"))

    return TaskOutput(
        output={"answer": response},
        metadata={"tokens": 150},  # Optional metadata
    )

# Module-level config (optional)
params = {"model": "gpt-4o-mini"}
name = "My Experiment"
```

### Evaluator Functions

The `@evaluator` decorator marks evaluation functions. They receive an `EvalInput` with both expected and actual outputs:

```python
from cat.experiments.protocol import EvalInput, EvalOutput
from cat.experiments.sdk import evaluator

@evaluator
def accuracy(input: EvalInput) -> EvalOutput:
    """Check if the answer is correct."""
    expected = input.expected_output.get("answer", "") if input.expected_output else ""
    actual = input.actual_output.get("answer", "") if input.actual_output else ""

    score = 1.0 if expected.lower() == actual.lower() else 0.0
    return EvalOutput(score=score, label="correct" if score else "incorrect")

@evaluator
async def llm_judge(input: EvalInput) -> EvalOutput:
    """Use an LLM to judge quality (async evaluators supported)."""
    # Your LLM judge call here
    judgment = await call_judge_llm(input.actual_output)
    return EvalOutput(score=judgment.score, metadata={"reasoning": judgment.reason})
```

## Storage Backends

Select a backend with `--backend` (or `backend:` in a config file). Flags apply
to the selected backend, and are equivalent to setting `backends.<name>.<key>`
in a config file.

### Local Storage (default)

Results are written to a directory:

```bash
cat-experiments run my_experiment.py --dataset data.jsonl --output-dir ./results
```

Each experiment gets its own directory under `--output-dir`, and that directory
name is the ID `--resume` takes.

### Phoenix Integration

Stream results to Phoenix. `--dataset` is a dataset name or ID on the server:

```bash
# --url defaults to $PHOENIX_BASE_URL, then http://localhost:6006
cat-experiments run my_experiment.py --dataset my-dataset \
    --backend phoenix \
    --url http://localhost:6006
```

### CAT Cafe Integration

Stream results to CAT Cafe (v0.7.0 or later):

```bash
# --url defaults to $CAT_BASE_URL, then http://localhost:8000
cat-experiments run my_experiment.py --dataset my-dataset \
    --backend cat-cafe \
    --url http://localhost:8000
```

### Backend Plugins

A backend can also run as a separate executable, so one can be written and
shipped independently of the CLI. Name it under `command` in a config file; the
CLI launches it and hands it every other key in its section:

```yaml
backend: my-backend
backends:
  my-backend:
    command: ["/usr/local/bin/my-backend-plugin"]
    checksum: "sha256:<digest>"   # optional; the CLI refuses a binary that does not match
    url: https://example.invalid  # passed to the plugin
```

See [Writing a Backend Plugin](docs/writing-a-backend-plugin.md) to build one,
and [Backend Protocol](docs/backend-protocol.md) for the design.

### Resuming

`--resume <experiment-id>` continues an interrupted experiment, running only
what did not finish. For the local backend the ID is the experiment's directory
name; for Phoenix and CAT Cafe it is the ID the server assigned, printed in the
run summary as `Backend experiment ID`. An ID the backend does not know is an error rather than a fresh
start.

## CLI Options

```bash
cat-experiments run <experiment> [OPTIONS]

Dataset Options:
  -d, --dataset REF           Local file (JSON/JSONL), or a dataset name or ID on a remote backend
      --dataset-version V     Dataset version (Phoenix)

Storage Options:
  -b, --backend NAME          local, phoenix, cat-cafe, or a configured plugin (default: local)
  -o, --output-dir DIR        Output directory (local backend)
      --url URL               Backend URL (phoenix, cat-cafe)
  -c, --config PATH           Config file (YAML)

Experiment Options:
      --param KEY=VALUE       Override params (can be repeated)
      --max-workers N         Parallel workers (default: 5)
      --repetitions N         Repetitions per example (default: 1)
      --dry-run N             Run N examples without persisting
      --resume EXPERIMENT_ID  Resume a previous experiment

Output Options:
      --output FORMAT         Output format: text, json (default: text)
```

Run `cat-experiments run --help` for the full list, including tracing options.

## Built-in Tool Call Evaluators

The package includes tool call correctness evaluators for agent evaluation:

```python
from cat.experiments.sdk import match_tool_calls, ToolCallMatch, ToolCallMatchingResult

# Match expected vs actual tool calls
result: ToolCallMatchingResult = match_tool_calls(
    expected_calls=[{"name": "search", "arguments": {"query": "weather"}}],
    actual_calls=[{"name": "search", "arguments": {"query": "weather today"}}],
)

print(result.precision, result.recall, result.f1_score)
```

## Architecture

```
┌─────────────────────────────────────────────────────────┐
│                    CLI / Orchestrator                   │
│  - Dataset loading                                      │
│  - Parallel execution (windowed dispatch)               │
│  - Storage backends (Local, Phoenix, Cat Cafe)          │
│  - Progress reporting                                   │
└───────────────────────┬─────────────────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────────┐
│                      Executor                           │
│  - Loads experiment file                                │
│  - Runs @task and @evaluator functions                  │
│  - Returns results                                      │
└─────────────────────────────────────────────────────────┘
```

The package uses an executor protocol that separates orchestration from execution, enabling future support for:
- Multi-language experiments (TypeScript, etc.)
- Distributed execution
- Custom execution backends
