Metadata-Version: 2.4
Name: regress-llm
Version: 0.1.0
Summary: Dead-simple regression testing for LLM prompts, built for CI
Author: Vinay Bhapkar
License-Expression: MIT
Project-URL: Homepage, https://github.com/vinayvb231-pixel/regress
Project-URL: Repository, https://github.com/vinayvb231-pixel/regress
Project-URL: Issues, https://github.com/vinayvb231-pixel/regress/issues
Keywords: llm,evals,testing,ci,prompts
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.30; extra == "anthropic"
Provides-Extra: all
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: anthropic>=0.30; extra == "all"
Dynamic: license-file

# regress

Dead-simple regression testing for LLM prompts, built for CI. One YAML file in your repo; prompt breakages fail the build like broken unit tests.

```bash
pip install regress-llm
regress init        # writes a starter regress.yaml
regress run         # runs your prompt tests
```

## Why

You changed a prompt. Or swapped models. Did anything break? Right now you find out from customers. `regress` catches it on the PR instead.

Unlike heavyweight eval platforms, there's nothing to adopt: no dashboard to learn, no SDK to integrate. Just a YAML file and a GitHub Action.

## 5-minute quickstart

1. Install: `pip install 'regress-llm[openai]'` (or `[anthropic]`, or `[all]`)
2. Set your key: `export OPENAI_API_KEY=...`
3. `regress init` and edit the generated `regress.yaml`
4. `regress run`

No key handy? Use `provider: mock` with `mock_response` on a test to try the harness with zero setup (see the repo's own `regress.yaml`).

## The YAML schema

```yaml
version: 1

defaults:
  provider: openai        # openai | anthropic | mock
  model: gpt-4o-mini
  temperature: 0
  judge_model: gpt-4o     # model used for llm_judge (defaults to `model`)
  max_tokens: 512

tests:
  - name: refund_answer_uses_policy
    prompt: |
      You are a support bot. Answer using ONLY this policy:
      {policy}
      Customer: {question}
    inputs:
      policy: "Refunds within 30 days."
      question: "Can I get a refund?"
    assert:
      - contains: "30 days"      # substring must appear
      - not_contains: "cannot"   # substring must NOT appear
      - regex: "\\$\\d+"         # regex must match
      - equals: "exact text"     # exact match after stripping
      - max_words: 50            # output word count cap
      - llm_judge:               # ask a model whether the output meets a criterion
          criteria: "The answer approves the refund and cites the policy"
```

Any test can override `provider`, `model`, or `temperature` individually. `{placeholders}` in `prompt` are filled from `inputs` (escape literal braces as `{{` / `}}`).

## CLI

```
regress run [--config regress.yaml] [--format terminal|markdown|json]
            [--output report.md] [--fail-fast]
regress init [--force]
```

Exit codes: `0` all passed, `1` failures, `2` config error — so CI fails the check on regressions automatically.

## GitHub Action

This repo dogfoods itself: [`.github/workflows/regress.yml`](.github/workflows/regress.yml) runs the suite on every PR and comments a Markdown report. To use it in your repo:

```yaml
- run: pip install regress-llm
- run: regress run --format markdown --output regress-report.md
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```

## Roadmap

- [ ] Baseline diffs: compare against the last green run, not just assertions
- [ ] Flaky detection: re-run failures N times before calling them real
- [ ] RAG-aware assertions: faithfulness / citation checks against retrieved context
- [ ] Hosted: history, trends, and alerts across runs (the paid tier)

## Contributing

PRs welcome. Run the mock suite before pushing: `python -m regress run`.
