CI FOR LLM PROMPTS

Your model regressed.
Your CI should have caught it.

evaldiff scores your LLM output against a dataset of expected answers and gates your pipeline with a green/red checkmark — the npm test of prompts.

Try the example repo → Read the docs

GitHub Actions — main

2026-10-07T18:21:14Z  run 43: done passed=1/3
2026-10-07T18:21:14Z  | passed | 1 |
##[error]GATE FAIL: pass_rate=33.33% (threshold 80%)
2026-10-07T18:21:14Z  Process completed with exit code 5.

Three files. One gate.

No agents, no dashboards you have to babysit. A dataset, a run, a threshold.

Write cases

Plain JSON. What the model should do, and what counts as correct.

[
  { "input": "What is the capital of France?",
    "expected": "Paris" },
  { "input": "Reply with exactly one word: PONG",
    "expected": "PONG" }
]

Score any model

OpenAI, self-hosted, Ollama, vLLM — anything with an OpenAI-compatible endpoint. Runs are per-case scored, with latency and tokens.

$ pip install evaldiff
$ evaldiff run --dataset evals/cases.json \
    --endpoint https://your-model/v1 \
    --model qwen3.8:27b --threshold 0.8

Gate the merge

One line in your workflow. Exit code 0 or 1, drop it anywhere CI runs.

- name: Run the evaldiff gate
  uses: evaldiff/action@v2
  with:
    api_key: ${{ secrets.EVALDIFF_API_KEY }}
    dataset_json: evals/cases.json
    threshold: 0.8

Green ships. Red blocks.

Both below are real runs against a real model. That's the whole product surface: a pass rate, a threshold, an exit code.

main @ fix: capital of France is ParisPASS

| pass rate | 100% |
GATE PASS: pass_rate=100% (threshold 80%)

main @ break: wrong expectedFAIL

| pass rate | 33.33% |
GATE FAIL: pass_rate=33.33% (threshold 80%)

Install

Python CLI today; GitHub Action ready. Diff any two runs to see exactly which cases regressed.

$ pip install evaldiff
$ npm i -g @evaldiff/evaldiff
$ evaldiff diff <run-a> <run-b>  # regressions, side by side