Metadata-Version: 2.4
Name: evalgrid
Version: 0.1.2
Summary: Agent evaluation framework — run, score, and track AI agent correctness at scale.
Project-URL: Homepage, https://github.com/naman006-rai/evalgrid
Project-URL: Documentation, https://github.com/naman006-rai/evalgrid#readme
Project-URL: Repository, https://github.com/naman006-rai/evalgrid
Project-URL: Issues, https://github.com/naman006-rai/evalgrid/issues
Project-URL: Bug Tracker, https://github.com/naman006-rai/evalgrid/issues
Author-email: Naman Rai <naman.rai006@gmail.com>
License: MIT
License-File: LICENSE
Keywords: agents,ai,evaluation,llm,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: aiosqlite>=0.20.0
Requires-Dist: click>=8.1.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: jsonschema>=4.0.0
Requires-Dist: litellm<2.0.0,>=1.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0.0
Requires-Dist: sqlalchemy>=2.0.0
Requires-Dist: uvicorn[standard]>=0.32.0
Provides-Extra: dev
Requires-Dist: mypy>=1.13.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.7.0; extra == 'dev'
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.30.0; extra == 'postgres'
Requires-Dist: psycopg2-binary>=2.9.0; extra == 'postgres'
Description-Content-Type: text/markdown

<div align="center">

<svg width="52" height="52" viewBox="0 0 36 36" fill="none" xmlns="http://www.w3.org/2000/svg">
  <rect x="2" y="2" width="14" height="14" rx="3" fill="#5b5ef4"/>
  <rect x="20" y="2" width="14" height="14" rx="3" fill="#5b5ef4" opacity="0.4"/>
  <rect x="2" y="20" width="14" height="14" rx="3" fill="#5b5ef4" opacity="0.4"/>
  <rect x="20" y="20" width="14" height="14" rx="3" fill="#5b5ef4" opacity="0.15"/>
  <path d="M8 9 L11 12 L16 6" stroke="white" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"/>
</svg>

# evalgrid

**Agent evaluation framework — run, score, and track AI agent correctness at scale.**

[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

AI agents fail in ways unit tests can't catch: the customer support agent stops calling the refund tool after a prompt change. The research agent loops on the same Google search 23 times. The reconciliation agent silently picks the wrong account format on edge cases. evalgrid catches these regressions before they ship.

</div>

## Status

EvalGrid is in early development (v0.1). The framework runs reliably for local-first agent evaluation, but you should expect:

- Some rough edges in the dashboard UI
- Schema changes in scenario YAML between v0.1 and v0.2
- GitHub status check / CI gate coming in v0.2

If you're building agents and want eval-driven workflows now, this works. If you need production-grade CI integration, watch for v0.2.

## Screenshots

![Dashboard overview](docs/screenshots/dashboard-overview.png)
*Dashboard showing run history with pass rates and per-scenario breakdowns*

![Run detail](docs/screenshots/run-detail.png)
*Per-step verdicts with judge reasoning for a failed scenario*

---

## Install

```bash
pip install evalgrid
```

## Quick start

**1. Initialize**
```bash
evalgrid init
```
Scaffolds `.evalgrid/` with a config file and an example scenario.

**2. Write a scenario**

`.evalgrid/scenarios/qualify_lead.yml`:
```yaml
id: sdr-qualify-lead
name: SDR Lead Qualification
description: Verify BANT qualification on an inbound enterprise lead
agent_type: sdr

prompt: |
  You are an expert SDR. Qualify leads using the BANT framework.
  Identify budget, authority, need, and timeline. Recommend a next step.

test_input: |
  Prospect: Sarah Chen, VP Engineering at TechCorp (500 employees)
  Budget: $50k allocated this quarter for developer tooling
  She filled out a demo request form 20 minutes ago.

expected_actions:
  - action: identify_bant_criteria
    required: true
    description: Identify which BANT criteria are present
  - action: ask_timeline_question
    required: true
    description: Ask about the buying timeline
  - action: recommend_next_step
    required: false
    description: Propose a clear next step (demo, call, etc.)

tags: [sdr, bant]
```

**3. Run**
```bash
export ANTHROPIC_API_KEY=sk-ant-...
evalgrid run
```

**4. See results**
```
Running 1 scenario(s)...

 ✓  SDR Lead Qualification   sdr   PASS   0.94   3/3   $0.0042

1/1 scenarios passed (100%)
Total cost: $0.0042
Results saved to .evalgrid/results/
```

Per-run cost (sum of agent + judge LiteLLM calls) is tracked automatically when LiteLLM has pricing data for your model. Token counts are reported for any model; cost shows `—` for local/unpriced models.

**5. Open the dashboard**
```bash
evalgrid server
```
Then visit `http://localhost:8000` for the full run history, pass rate trends, and per-step breakdowns.

> **Note:** The dashboard binds to `127.0.0.1` by default and is designed for local use only. Do not expose it to a public network.

---

## Commands

| Command | Description |
|---------|-------------|
| `evalgrid init` | Scaffold `.evalgrid/` in your project |
| `evalgrid run` | Run all scenarios and print pass/fail |
| `evalgrid run --tag sdr` | Filter by tag |
| `evalgrid run --mock` | Dry run with no API calls (great for CI) |
| `evalgrid run --output results.json` | Save results to a file |
| `evalgrid run --suite <id>` | Run a named benchmark suite (4-rate scoring) |
| `evalgrid run --report results.html` | Generate an HTML/Markdown report (extension picks format) |
| `evalgrid scenario add` | Interactively create a new scenario |
| `evalgrid scenario list` | List all scenarios |
| `evalgrid scenario validate <path>` | Validate a scenario YAML file before committing |
| `evalgrid server` | Start the React dashboard |
| `evalgrid flakiness` | Report scenario stability across recent runs (exits 1 if any are flaky) |

## Scenario format

```yaml
id: unique-scenario-id
name: Human Readable Name
description: What this scenario tests
agent_type: sdr | coding | support | research | finance | legal | generic

prompt: |
  System prompt for your agent...

test_input: |
  The user message / input to evaluate...

expected_actions:
  - action: action_name
    required: true
    description: What this action entails

success_criteria: >
  Plain-language description of what a passing response looks like.

tags: [tag1, tag2]
timeout_seconds: 60
weight: 1.0                  # used by suite weighted scoring

checks:                      # optional: deterministic structural assertions
  - type: valid_json         # output must parse as JSON
  - type: required_fields    # JSON output must have these top-level keys
    fields: [decision, reasoning]
  - type: enum               # a specific JSON field must be in an allowed set
    field: decision
    allowed: [APPROVE, REJECT, ESCALATE]
    hard_fail: true          # short-circuits before judge LLM cost
```

### Deterministic checks

Checks run before the judge LLM. A `hard_fail: true` check (default) short-circuits the run and skips the judge entirely, saving cost. A `hard_fail: false` check records a soft failure but still calls the judge for semantic scoring.

| `type:` | What it asserts | Extra keys |
|---|---|---|
| `valid_json` | Output parses as JSON | — |
| `no_markdown` | No fenced code blocks, headings, or bold markers | — |
| `required_fields` | Top-level JSON has these keys | `fields: [str]` |
| `enum` | A JSON field value is in an allowed set | `field: str`, `allowed: [str]` |
| `forbidden_regex` | A regex does NOT match the output (case-insensitive) | `pattern: str` |
| `contains` | A literal substring is present (case-insensitive) | `text: str` |
| `max_length` | Output length ≤ N chars | `value: int` |
| `json_schema` | Output validates against a JSON Schema (Draft 7) | `schema: {...}`, `strict: bool` |

### Benchmark suites

Group scenarios into named suites with a pass threshold and weighted scoring. Suite results are tracked separately as four rates: semantic pass, hard fail, soft fail, error — never collapsed into a single number.

`.evalgrid/suites/python_bug_fixer.yml`:
```yaml
id: python-bug-fixer
name: Python bug-fixer benchmark
version: 1                   # bump when the benchmark definition changes
pass_threshold: 0.8          # suite passes if semantic_pass_rate >= 80%
scenarios:                   # by scenario id, OR omit and use `tags:` to match
  - coding-fix-off-by-one
  - coding-fix-mutable-default
  - coding-fix-iterator-exhausted
tags: [coding]               # alternative to scenarios:
```

Run with `evalgrid run --suite python-bug-fixer`. The dashboard's Benchmarks view tracks suite history.

### Flakiness

A scenario whose pass/fail outcome flips across identical runs is flaky — exactly the signal you want before trusting a CI gate.

```bash
evalgrid flakiness               # all scenarios, last 10 runs
evalgrid flakiness --flaky-only  # only show unstable scenarios
evalgrid flakiness --window 30   # widen the rolling window
```

Exit code is 1 if any scenarios are flaky, 0 otherwise — so this is itself CI-gateable.

## Architecture: agents and judges

EvalGrid is a runner-and-scorer framework. The agent (the LLM being tested) and the judge (the LLM doing the scoring) are configured independently and can use different providers via LiteLLM.

The example scenarios that ship with v0.1 use Claude for both — that's a demo choice, not an architectural constraint. Common real-world setups include:

- GPT-4o agent, Claude Sonnet judge (cross-model validation)
- Claude Haiku agent, Claude Sonnet judge (cost-efficient testing)
- Llama 3.1 agent, GPT-4o judge (open-source agent, frontier judge)

Configure in `.evalgrid/config.yml`:

```yaml
model: gpt-4o          # the agent being tested
judge:
  provider: anthropic
  model: claude-sonnet-4-6   # the model scoring the agent
```

## Examples

Example scenarios are in the [`examples/`](./examples/) directory. Copy them to get started:

```bash
cp examples/*.yml .evalgrid/scenarios/
```

## GitHub Actions

Add to `.github/workflows/ci.yml`:

```yaml
- name: Run evalgrid
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: evalgrid run --output results.json
```

## Python API

```python
from evalgrid.core import EvalConfig, ScenarioLoader, ScenarioRunner

config = EvalConfig(model="claude-sonnet-4-6")
loader = ScenarioLoader()
scenarios = loader.load_dir(".evalgrid/scenarios")

runner = ScenarioRunner(config)
results = runner.run_all(scenarios, on_result=lambda r: print(r.scenario_name, r.status))

for result in results:
    print(f"{result.scenario_name}: {result.pass_rate:.0%}")
```

## Security

evalgrid is a **local-only developer tool**. Keep these constraints in mind:

- The API server binds to `127.0.0.1` by default. Use `--bind` to change this, but be aware that exposing the server to a network grants unauthenticated access to your scenarios and eval runs.
- Scenario files are sandboxed to `~/.evalgrid/scenarios/`. Paths supplied via the API are validated and rejected if they escape this directory.
- Provider API keys (e.g. `ANTHROPIC_API_KEY`) are read from environment variables by default. The dashboard's "Run new eval" form may also accept a key in-browser and forward it on a single `POST /api/runs` call for convenience; that key is held only for the duration of the run and is never written to disk, logged, or persisted. The env-var path is preferred for any non-interactive use (CLI, CI).
- All error messages are passed through a regex that redacts strings matching common provider key prefixes (`sk-ant-`, `sk-`, `pypi-`, `AIza…`) before they reach logs or HTTP responses.
- Do not run `evalgrid server` on a shared or public-facing machine.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for full setup instructions and PR guidelines.

## License

MIT. See [LICENSE](LICENSE). Copyright (c) 2026 Naman Rai.
