Metadata-Version: 2.5
Name: rag-eval-harness
Version: 0.2.0
Summary: RAGAS with a memory and a diff view. Evaluate RAG pipelines, persist runs, and fail CI on regressions.
Author-email: John Carl Santos <johncarlsantos30@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: ci,evaluation,lexical,llm,rag,ragas,regression
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: fastapi>=0.115
Requires-Dist: httpx>=0.28
Requires-Dist: pydantic-settings>=2.6
Requires-Dist: pydantic>=2.10
Requires-Dist: python-dotenv>=1.0
Requires-Dist: python-multipart>=0.0.18
Requires-Dist: sqlalchemy>=2.0.36
Requires-Dist: typer>=0.15
Requires-Dist: uvicorn[standard]>=0.32
Provides-Extra: agent
Requires-Dist: langgraph>=1.2.2; extra == 'agent'
Requires-Dist: pydantic-ai-slim[anthropic,openai]>=2.51.0; extra == 'agent'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
Provides-Extra: ragas
Requires-Dist: openai>=1.40; extra == 'ragas'
Requires-Dist: ragas>=0.2; extra == 'ragas'
Provides-Extra: trace
Requires-Dist: logfire>=5.1.1; extra == 'trace'
Description-Content-Type: text/markdown

# rag-eval-harness

[![CI](https://github.com/JohnCarl-30/rag-eval-harness/actions/workflows/ci.yml/badge.svg)](https://github.com/JohnCarl-30/rag-eval-harness/actions/workflows/ci.yml)

RAGAS with a memory and a diff view.

**What this proved:** a weaker retriever on a 40-question Nimbus help-center set cut mean **context recall from 0.880 to 0.740**. `rag-eval regress --threshold 0.05` exited **1**. Full write-up: [Nimbus case study](docs/nimbus-case-study.md). 90-second recording: `./scripts/demo.sh`.

Chunking the same corpus doubled lexical precision and failed the recall gate. Production stayed on whole articles. [Chunking case study](docs/nimbus-chunking.md). `./scripts/demo-chunking.sh`.

Upload a golden RAG dataset, run a pipeline (precomputed traces or HTTP), score with **RAGAS**, **lexical** overlap, or a keyless **stub** evaluator, persist runs, and fail CI when mean metrics drop versus a tagged baseline.

This is not a RAG framework. There is no ingestion or retrieval stack. Evaluation orchestration is the product.

## 10-minute path

Requires Python 3.11+ ([uv](https://docs.astral.sh/uv/) recommended) and, for the UI, Node 20+.

```bash
git clone https://github.com/JohnCarl-30/rag-eval-harness.git && cd rag-eval-harness
uv sync
./scripts/demo.sh
```

You should see `context_recall` FAIL (0.880 → 0.740) and the script exit 0. That red `rag-eval regress` is the product. No API key. Then score dummy traces if you want a green run:

```bash
uv run rag-eval eval examples/dummy-rag/traces.jsonl --evaluator stub -o /tmp/rag-eval-head.json
```

Optional. Call a live HTTP SUT (the canned FAQ server):

```bash
python3 examples/dummy-rag/server.py --port 8080 &
uv run rag-eval eval examples/dummy-rag/golden.csv \
  --sut-url http://127.0.0.1:8080/query \
  --evaluator stub
```

UI (dev):

```bash
uv run rag-eval serve          # API on 127.0.0.1:8000, open on loopback (no API key)
cd web && npm install && npm run dev   # Vite on :5173, proxies /api
```

Open http://127.0.0.1:5173, upload `examples/dummy-rag/traces.jsonl` or `golden.csv`, create a run, tag a baseline, open Diff.

Docker:

```bash
cp .env.example .env          # set RAG_EVAL_API_KEY (required: the API binds 0.0.0.0)
docker compose up --build
# UI+API: http://127.0.0.1:8000   dummy RAG: http://127.0.0.1:8080/query
```

## Install

```bash
pip install git+https://github.com/JohnCarl-30/rag-eval-harness.git
pip install 'rag-eval-harness[ragas]'   # after a PyPI release, or: pip install '.[ragas]' from a clone
```

```bash
rag-eval eval traces.jsonl --evaluator stub
rag-eval eval golden.csv --sut-url https://your-rag.example/query --sut-token "$TOKEN"
rag-eval baseline <run-id>
rag-eval regress --baseline <run-id-or.json> --head <run-id-or.json> --threshold 0.05
rag-eval diff --baseline <run-id-or.json> --head <run-id-or.json>
rag-eval investigate --baseline <run-id-or.json> --head <run-id-or.json>   # needs [agent]
rag-eval investigate ... --mode supervisor --trace trace.json
rag-eval serve
```

`regress` exits **1** if any mean drops by more than `--threshold` (default `0.05`), or if head has more errored rows than baseline (means skip errored rows, so a timing-out SUT would otherwise pass). `diff` prints the same means plus the worst per-row drops (rows paired by question) and always exits 0 on a successful compare.

`investigate` runs the `regress` gate, then a LangGraph fan-out of Pydantic AI agents explains a failure: one diagnosis per failing stage (retrieval, generation), then a summary. `--mode supervisor` swaps the fixed graph for one agent that picks its own tools. `--trace` writes every agent run as JSON; `--otel` sends spans via Logfire. Same exit codes as `regress`. Install with `pip install 'rag-eval-harness[agent]'`. See [docs/agent.md](docs/agent.md).

## Contracts

| Piece | Behavior |
| --- | --- |
| Columns | `question` required. Optional `ground_truth`. Traces also need `answer` and `retrieved_contexts` (JSON list or `\|` / newline delimited). Aliases: `user_input`, `reference`, `response`, `contexts`. |
| HTTP SUT | `POST {"question"}` → `{"answer","retrieved_contexts"}`. Optional bearer. Per-row timeout; a failed row is an error, the run continues. |
| Metrics | Always faithfulness + answer relevancy. Context precision + context recall when `ground_truth` is present. |
| Evaluators | `stub` (stable hashes, CI smoke), `lexical` (token overlap vs context, use this for real gates), `ragas` (`OPENAI_API_KEY`, optional `OPENAI_BASE_URL`). |
| Store | SQLite default (`DATABASE_URL`). Postgres via `postgresql+psycopg://…`. |
| Jobs | In-process. **100-row cap.** No Redis. |
| Auth | Open on loopback. `RAG_EVAL_API_KEY` required when binding a non-loopback address. No cross-origin access unless `RAG_EVAL_CORS_ORIGINS` lists origins (comma-separated). |
| Regression | Tag a run as baseline. Gate on **mean** delta. Per-row diffs are UI-only. |

Docs: [adapters](docs/adapters.md) · [metrics](docs/metrics.md) · [CI](docs/ci.md) · [investigator](docs/agent.md) · [storage](docs/storage.md) · [Nimbus case study](docs/nimbus-case-study.md) · [Chunking case study](docs/nimbus-chunking.md) · [Cost and p95](docs/nimbus-cost.md) · [Lexical vs judge](docs/nimbus-judge.md)

## GitHub Action

`stub` is keyless CI smoke. Hashes look healthy on junk contexts. Use `lexical` for a real gate.

```yaml
- uses: JohnCarl-30/rag-eval-harness@v0.2.0
  with:
    traces: tests/golden/traces.jsonl
    evaluator: lexical
    baseline: tests/golden/baseline.json
    threshold: "0.05"
```

Public consumer: [Relaydesk](https://github.com/JohnCarl-30/relaydesk) fails PRs when Nimbus recall drops. See [docs/ci.md](docs/ci.md).

## Development

```bash
uv sync --extra agent   # without the extra, tests/test_agent.py is skipped
uv run pytest
uv run ruff check src tests
cd web && npm install && npm run build
docker compose up --build   # then ./scripts/smoke.sh
```

## License

MIT. See [CHANGELOG](CHANGELOG.md) and [CONTRIBUTING](CONTRIBUTING.md).
