Metadata-Version: 2.4
Name: agent-run-diff
Version: 0.1.0
Summary: Compare baseline vs current agent runs and surface regressions as structured reasons: success loss, new errors, failed tool calls, output drift, step/latency/cost bloat.
Project-URL: Homepage, https://github.com/MukundaKatta/agent-run-diff
Project-URL: Issues, https://github.com/MukundaKatta/agent-run-diff/issues
Project-URL: Source, https://github.com/MukundaKatta/agent-run-diff
Author-email: Mukunda Katta <mukundkst@gmail.com>
License: MIT
License-File: LICENSE
Keywords: agent,anthropic,cli,evaluation,llm,observability,openai,regression,testing,trace
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# agent-run-diff

[![CI](https://github.com/MukundaKatta/agent-run-diff/actions/workflows/ci.yml/badge.svg)](https://github.com/MukundaKatta/agent-run-diff/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/agent-run-diff.svg)](https://pypi.org/project/agent-run-diff/)
[![Python](https://img.shields.io/pypi/pyversions/agent-run-diff.svg)](https://pypi.org/project/agent-run-diff/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)

Compare baseline vs current agent runs and surface regressions as **structured reasons**, not vibes. Answers "why does the agent feel worse?" with seven named signals:

1. **Success loss** — cases that passed baseline, fail current
2. **New error signatures** — error kinds present in current but absent in baseline
3. **Tool failure rises** — tools that failed more often in current than baseline
4. **Output drift** — for cases both passed, how much the final text changed (token-F1)
5. **Step bloat** — step count ratio above threshold
6. **Latency bloat** — wall-clock ratio above threshold
7. **Cost bloat** — USD ratio above threshold

Zero runtime dependencies. Works on any agent framework — you hand it JSONL traces, it tells you what changed.

## Install

```bash
pip install agent-run-diff
```

## Quick start

```bash
agent-run-diff baseline.jsonl current.jsonl
```

Exits `0` on clean, `1` on any regression, `2` on bad input. The markdown report prints to stdout:

```markdown
# Agent Regression Report

Pass rate: baseline **92%** → current **78%** (-14pp)
Total latency: 48,200 ms → 81,400 ms
Total cost:    $1.2400 → $2.1800

## Summary
- Success losses:    **3**
- New error kinds:   **1**
- Tool failure rises: **2**
- Step bloat cases:  **4**
- Latency bloat:     **6**
- Cost bloat:        **5**

## Success losses (was passing, now not)
- `login-flow-happy-path` — status: success → failed
...

## Tool failures rising
| Tool | Baseline | Current | Sample errors |
| --- | ---: | ---: | --- |
| browser.click | 0 | 4 | `Element not found: #submit` |
...
```

## Trace format

A run is one JSON object. The shape is lenient — accepts snake_case or camelCase, and several common aliases:

```json
{
  "run_id": "run-2026-04-24-abc",
  "case_id": "login-flow-happy-path",
  "status": "success",
  "final_output": "Logged in as alice.",
  "steps": [
    {
      "type": "tool_call",
      "tool_name": "browser.click",
      "tool_args": {"selector": "#submit"},
      "error": null,
      "latency_ms": 420,
      "cost_usd": 0.001
    }
  ],
  "total_latency_ms": 2300,
  "total_cost_usd": 0.08
}
```

Aliases recognized:

- `case_id` | `caseId` | `test_id` | `testId` (falls back to `run_id` if none present)
- `status` | `outcome` | `result.status` — values like `pass`/`passed`/`ok`/`success` all normalize to `"success"`
- `final_output` | `finalOutput` | `output`
- `steps` | `trace` | `events`
- per-step: `tool_name`/`toolName`/`name`, `error`/`error.message`, `latency_ms`/`latencyMs`/`duration_ms`
- totals: `total_cost_usd`/`totalCostUsd`, `total_latency_ms`/`totalLatencyMs` (auto-summed from steps if absent)

Runs are matched between baseline and current by `case_id`.

## Configuring thresholds

Defaults are transparent (1.3x bloat, 0.7 output-drift F1) and overridable:

```bash
agent-run-diff base.jsonl curr.jsonl \
  --step-ratio 1.5 \
  --latency-ratio 2.0 \
  --cost-ratio 1.2 \
  --output-drift-f1 0.6 \
  --min-latency-ms 200 \
  --min-cost-usd 0.01
```

The `--min-latency-ms` and `--min-cost-usd` floors prevent noise on trivially-short or trivially-cheap runs.

## CI usage

```yaml
- name: Compare against baseline
  run: |
    agent-run-diff \
      baselines/2026-04-20.jsonl \
      runs/$GITHUB_SHA.jsonl \
      --format json > regression.json
```

The workflow step fails when any regression is detected (exit 1). Attach `regression.json` as an artifact to make the seven signals browsable in the PR review.

## Honest scope

- **Not semantic similarity.** Output drift is token-F1, not embeddings. Flagging is "text changed a lot," not "meaning changed." Labeled as such in every code path.
- **Not a framework.** The tool never touches your agent, never calls an LLM, never runs your tests. It only reads traces you already have.
- **No pricing built-in.** If your traces include `cost_usd` we use it; otherwise cost bloat is skipped. Pricing is your framework's job, not this tool's.

## Library API

```python
from pathlib import Path
from agent_run_diff import parse_runs_file, analyze, render_markdown, Thresholds

baseline = parse_runs_file("baseline.jsonl")
current = parse_runs_file("current.jsonl")
report = analyze(baseline, current, thresholds=Thresholds(latency_ratio=1.5))

print(render_markdown(report))
if report.has_regressions:
    raise SystemExit(1)
```

## License

MIT.
