Metadata-Version: 2.4
Name: brooder
Version: 0.4.0
Summary: Snapshot testing for AI agents — catch behavior regressions before they ship.
Project-URL: Homepage, https://brooder.dev
Project-URL: Repository, https://github.com/agentbrooder/brooder
Project-URL: Issues, https://github.com/agentbrooder/brooder/issues
Author: Brooder
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: agents,ai,ci,evals,llm,regression,snapshot,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: pydantic>=2.5
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.12
Provides-Extra: claude-agent
Requires-Dist: claude-agent-sdk>=0.1; extra == 'claude-agent'
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pre-commit>=3.7; extra == 'dev'
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.25; extra == 'docs'
Provides-Extra: judge-anthropic
Requires-Dist: anthropic>=0.30; extra == 'judge-anthropic'
Provides-Extra: judge-litellm
Requires-Dist: litellm>=1.40; extra == 'judge-litellm'
Provides-Extra: judge-openai
Requires-Dist: openai>=1.0; extra == 'judge-openai'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Provides-Extra: openai-agents
Requires-Dist: openai-agents>=0.1; extra == 'openai-agents'
Provides-Extra: otel
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.20; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'otel'
Provides-Extra: pytest
Requires-Dist: pytest>=8.0; extra == 'pytest'
Description-Content-Type: text/markdown

<p align="center">
  <img src="assets/banner.svg" alt="Brooder — snapshot testing for AI agents" width="760">
</p>

<p align="center">
  <a href="https://github.com/agentbrooder/brooder/actions/workflows/ci.yml"><img src="https://github.com/agentbrooder/brooder/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://pypi.org/project/brooder/"><img src="https://img.shields.io/pypi/v/brooder?color=3b82f6" alt="PyPI"></a>
  <a href="https://pypi.org/project/brooder/"><img src="https://img.shields.io/pypi/pyversions/brooder" alt="Python versions"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-blue" alt="License: Apache-2.0"></a>
  <a href="https://github.com/astral-sh/ruff"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff"></a>
</p>

**Snapshot testing for AI agents. Catch behavior regressions before they ship.**

Your AI agent is one model upgrade away from silently breaking. You bump the model, tweak a
prompt, or change a tool — and the agent starts behaving differently. You find out from a customer.

Brooder is the safety net. Wrap your agent once, and Brooder records its real runs as **golden
baselines**. Every time you change the model, a prompt, or a tool, it re-runs and shows you a
**behavioral diff** — what changed, what broke — and fails your CI if it regressed.

No eval datasets to hand-write. One command. It's `jest --updateSnapshot`, but for agents.

```bash
pip install brooder
```

<p align="center">
  <img src="assets/demo.svg" alt="brooder migrate catching a dropped tool call and a flipped answer" width="760">
</p>

> Status: early alpha, built in public. Apache-2.0.

---

## 60-second demo (no API keys needed)

The included example agent simulates a model upgrade with an env var, so you can see Brooder catch
a real regression completely offline.

```bash
git clone https://github.com/agentbrooder/brooder && cd brooder
pip install -e .

# The signature move: what breaks if I migrate from one model to another?
brooder migrate --from gpt-4o --to gpt-5-new examples/regressing_agent.py
```

Output (abridged):

```
──────────────────────── Model Migration Report ────────────────────────
 1 of 3 cases change behavior when migrating gpt-4o → gpt-5-new.

 support-agent · e1ded4070eee · REGRESSED · stability 40
   path diverged at step 0: was TOOL create_ticket(order=12345), now dropped
   - trajectory[0]  {'name': 'create_ticket', 'args': {'order': '12345'}}
   ~ output
       before: I've started your refund.
       after:  Refunds are not supported.
```

The "new model" silently stopped creating the refund ticket **and** flipped its answer. That would
have shipped to production unnoticed. Brooder caught it — and exited non-zero, so CI would block it.

---

## The workflow

```bash
brooder record examples/regressing_agent.py     # capture golden baselines from real runs
brooder run    examples/regressing_agent.py     # re-run after a change, diff vs baseline
brooder diff                                    # see exactly what changed
brooder approve --only expected                 # bulk-accept the cosmetic drift...
brooder approve <case>                           # ...then accept the reviewed ones case-by-case
```

`brooder run` exits non-zero when behavior regressed — drop it into CI and it gates your PRs.

**No snapshot fatigue.** When a model bump surfaces a wall of diffs, Brooder classifies each as
**suspicious** (the output, tool path, or a guardrail actually changed) or **expected** (cosmetic
reasoning-turn / count drift), headlines the summary `N suspicious · M expected`, and sorts the
scary ones first — so `brooder approve --only expected` clears the noise in one command and you spend
review on the few that matter. (`brooder approve` with no args still accepts everything.)

---

## Instrument your agent

Add one decorator. That's the whole SDK. Log tool calls explicitly with `brooder.tool_call`, or
wrap your LLM client with `brooder.instrument(...)` and Brooder captures the model's tool-call
decisions for you.

```python
import brooder
import openai

client = brooder.instrument(openai.OpenAI())   # auto-captures tool calls while recording

@brooder.record("support-agent")
def agent(question: str) -> str:
    docs = client.chat.completions.create(model="gpt-4o", messages=[...])
    return answer_from(docs)

# call it over your real inputs; brooder records/replays automatically
```

Baselines are plain JSON committed to your repo, so diffs show up in code review like any other
change.

**It tests the whole trajectory, not single LLM calls.** `@brooder.record` wraps your *entire*
agent — every step of its plan → act → observe loop. The baseline is the full trajectory: every
tool call across every turn, in order, plus the final output. So Brooder catches a `verify` step
that silently disappears *inside the loop* — the kind of agent-level regression an LLM-output eval
never sees.

### Works with your stack

| Layer | Supported |
| --- | --- |
| **LLM providers** | OpenAI, Azure OpenAI, Anthropic, AWS Bedrock, Google (Gemini / Vertex) — auto-detected |
| **Agent frameworks** | LangChain, LangGraph, CrewAI, AutoGen (via OpenTelemetry), OpenAI Agents SDK, Claude Agent SDK |
| **Async** | `AsyncOpenAI`, `AsyncAzureOpenAI`, `AsyncAnthropic`, Google `generate_content_async` — no extra setup |
| **Custom endpoints** | Any base URL / proxy / OpenAI-compatible gateway — Brooder never touches credentials |

Setup for each is in **[Integrations](#integrations)** below.

---

## Why not just use observability / eval tools?

| Tool type | Examples | What it does | The gap Brooder fills |
| --- | --- | --- | --- |
| Observability | Langfuse, Laminar, Phoenix | Trace/monitor **after** it runs | Doesn't gate **before** you ship |
| Eval frameworks | DeepEval, Braintrust, Ragas | Score against **hand-written** datasets | Requires eval authoring nobody maintains |
| **Brooder** | — | **Record real runs → behavioral diff on every change → CI gate** | **Zero eval-writing, catches model-migration regressions** |

Your baselines are JSON files in **your** repo. No SaaS, no cloud account — nobody can acquire your
test suite out from under you.

---

## Gate your PRs (GitHub Action)

Drop Brooder into CI and it re-runs your agent on every pull request, comments the behavioral diff,
and fails the check when behavior regresses. Copy [examples/github-action.yml](examples/github-action.yml)
to `.github/workflows/brooder.yml`:

```yaml
permissions:
  contents: read
  pull-requests: write        # so it can comment the diff

jobs:
  agent-snapshot:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: agentbrooder/brooder@v1
        with:
          script: tests/agent_snapshot.py
```

The comment is upserted (updated in place, not spammed) and looks like the `--format markdown`
output.

> **Security:** `brooder ci` runs your checked-out agent script, so don't wire live provider
> secrets into a `pull_request`-triggered job — see the
> [CI trust model](SECURITY.md#running-brooder-in-ci-safely-trust-model).

---

## Snapshot-test inside pytest

Prefer to stay in the test runner you already have? `pip install brooder[pytest]` and use the
`brooder` fixture — no separate CLI harness:

```python
def test_support_agent(brooder):
    answer = support_agent("refund my order")   # tool calls / LLM turns auto-captured
    brooder.snapshot(answer, inputs="refund my order")
```

- `pytest --brooder-update` records the golden baseline (commit it, like any snapshot).
- `pytest` checks each run against that baseline: a behavioral regression **fails the test**, and a
  missing baseline fails with a hint instead of passing silently.

It honors your `brooder.yaml` (judge, normalization, redaction) and reuses the same capture and diff
engine as the CLI. Configure a case with `@pytest.mark.brooder(agent="support", inputs=...)`.

---

## What it checks

- **Structural diff** — the sequence of tool calls, their arguments, and the final output.
- **Semantic diff** — a pluggable judge (`judge: exact | llm`) so equivalent wording isn't a regression.
- **Flakiness** — `brooder run --runs 3` runs each case N times and flags non-determinism (`FLAKY`).
- **Review triage** — each regression is tagged **suspicious** (material) or **expected** (cosmetic)
  so a model bump's wall of diffs sorts by attention, not just count (see *The workflow* above).

Each case gets a verdict — `PASS` / `REGRESSED` / `NEW` / `FLAKY` — a review class, and a stability
score.

---

## Gate on cost & latency drift

Behavior isn't the only thing that regresses — a model swap can keep the *same* behavior while
quietly doubling your token bill. Brooder captures each run's **latency and token usage** (and cost,
if you configure prices) and can fail CI when they spike. Usage is tracked separately from behavior,
so a noisy latency blip never reads as a behavioral regression.

```yaml
# brooder.yaml
budget:
  max_total_tokens: 3000      # absolute ceiling per case
  max_tokens_increase: 0.2    # …or fail if tokens drift >20% vs the baseline
  prices:                     # optional: enable USD cost caps
    gpt-4o: { input_per_mtok: 2.5, output_per_mtok: 10.0 }
```

```console
$ brooder ci --budget agent.py
💸 Budget — 1 limit(s) exceeded
 • assistant/19761739: total_tokens 2600 is +160% vs baseline 1000 (limit 1200)
```

---

## Integrations

Everything above works with one decorator. These sections show the exact setup for each provider,
framework, and output format — expand what you need.

<details>
<summary><b>All providers, custom endpoints & async</b></summary>

Wrap your LLM client and Brooder records the model's tool-call decisions automatically. The provider
is auto-detected from the client; override it with a name, an alias, or a `Provider`:

```python
import brooder
from brooder import Provider

brooder.instrument(openai.OpenAI())                          # OpenAI
brooder.instrument(openai.AzureOpenAI(...))                  # Azure OpenAI (or provider="azure")
brooder.instrument(anthropic.Anthropic())                   # Anthropic (or provider=Provider.ANTHROPIC)
brooder.instrument(boto3.client("bedrock-runtime"))         # AWS Bedrock (or provider="aws")
brooder.instrument(genai.GenerativeModel("gemini-1.5-pro")) # Google Gemini / Vertex (or provider="gemini")
```

The canonical set is `brooder.Provider`: **OpenAI**, **Azure OpenAI**, **Anthropic**, **AWS
Bedrock**, and **Google (Gemini / Vertex)**.

**Custom endpoints & proxies.** Brooder never manages credentials or URLs — the provider's own SDK
does. Point a client at any base URL (an internal gateway, an OpenAI-compatible server, an
Azure-APIM-proxied Anthropic endpoint, …) and hand it to `instrument` unchanged:

```python
client = anthropic.Anthropic(base_url="https://your-proxy/…", api_key="…")
brooder.instrument(client)   # captured exactly the same
```

Model *names* are intentionally not diffed, so switching models isn't itself a change — only the
model's *behavior* (which tools it calls, with what arguments) is.

**Async works too.** `@brooder.record` and `instrument(...)` handle `async def` agents and async
clients — `AsyncOpenAI`, `AsyncAzureOpenAI`, `AsyncAnthropic`, and Google's `generate_content_async`
— with no extra setup (the recording context follows your `await`s and into child tasks):

```python
client = brooder.instrument(openai.AsyncOpenAI())

@brooder.record("support-agent")
async def agent(question: str) -> str:
    await client.chat.completions.create(model="gpt-4o", messages=[...])
    ...
```

(Async AWS Bedrock via aioboto3 isn't covered yet — the sync boto3 client is.)

</details>

<details>
<summary><b>OpenTelemetry (LangGraph, CrewAI, AutoGen, …)</b></summary>

If your framework emits OpenTelemetry GenAI spans, add one span processor and Brooder ingests the
whole trajectory — no manual `tool_call`:

```python
from opentelemetry import trace
from brooder.integrations.otel import BrooderSpanProcessor

trace.get_tracer_provider().add_span_processor(BrooderSpanProcessor(agent="support-agent"))
```

It maps inference spans → turns, `execute_tool` spans → tool calls, and the agent-root span's
input/output → the case identity and final answer. It also drops straight into the OTel pipelines
you already run (Datadog / Arize / Honeycomb).

</details>

<details>
<summary><b>Claude Agent SDK</b></summary>

Register Brooder's hooks and it records the tool trajectory and the final answer automatically:

```python
import brooder
from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions

options = ClaudeAgentOptions(hooks=brooder.claude_agent_hooks(agent="support-agent"))
async with ClaudeSDKClient(options=options) as client:
    await client.query(prompt)
    async for msg in client.receive_response():
        ...  # nothing Brooder-specific needed
```

`UserPromptSubmit` opens a run (the prompt is the case identity), tool-use hooks become tool steps,
and `Stop` finalizes it.

</details>

<details>
<summary><b>OpenAI Agents SDK</b></summary>

Its tracing is on by default — install Brooder's trace processor once and every run is captured (no
OpenAI API key required for capture):

```python
import brooder.integrations.openai_agents as bd_agents

bd_agents.install(agent="support-agent")   # then run your agents as usual
```

It maps generation/response spans → turns, function spans → tool calls, and handoffs and triggered
guardrails into the trajectory too — so both tool selection *and* control-flow regressions get
diffed.

</details>

<details>
<summary><b>LangChain / LangGraph</b></summary>

Attach one callback handler — no OpenTelemetry setup required:

```python
import brooder.integrations.langchain as bd_lc

handler = bd_lc.callback_handler(agent="support-agent")
graph.invoke({"messages": [...]}, config={"callbacks": [handler]})
```

The root chain start opens a run (its input is the case identity), model calls become turns, and
tool calls become tool steps — one handler covers both LangChain and LangGraph.

</details>

<details>
<summary><b>Machine-readable output & dashboards (<code>--json</code> / OTLP)</b></summary>

`run`, `ci`, and `diff` take `--format table|json|markdown` (`--json` is a shortcut). Exit codes are
unchanged, so you can gate *and* parse:

```bash
brooder run agent.py --json | jq '.summary'
# { "total": 3, "passed": 2, "regressed": 1, "flaky": 0, "regressions": 1,
#   "suspicious": 1, "expected": 0, "mean_stability": 80 }
```

Each case also carries a `severity` (`suspicious` / `expected` / `none`) so a dashboard can rank the
regressions that need a human by attention, not just count them.

For dashboards, point Brooder at any OTLP endpoint and each run emits a snapshot of gauges
(`brooder.cases.*`, `brooder.stability.mean`) — **one exporter** that reaches Datadog, Grafana,
Honeycomb, and CloudWatch:

```bash
pip install 'brooder[otel]'
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/metrics   # or metrics.otlp_endpoint in brooder.yaml
brooder ci agent.py
```

</details>

---

## Roadmap

See **[ROADMAP.md](ROADMAP.md)** for what's shipped and what's planned.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). Issues and PRs welcome — this is being built in public.

## License

[Apache-2.0](LICENSE).
