Metadata-Version: 2.4
Name: goldenrun
Version: 0.1.0
Summary: The evaluation and regression-detection engine behind EvalCI — LLM-as-judge scoring, lexical metrics, regression detection against an explicit baseline, and trend/anomaly analysis.
Author-email: Christian Chukwuka <chuks@chuksforge.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/chuksforge/goldenrun
Project-URL: Repository, https://github.com/chuksforge/goldenrun
Project-URL: Issues, https://github.com/chuksforge/goldenrun/issues
Project-URL: Documentation, https://github.com/chuksforge/goldenrun/blob/main/docs/architecture.md
Project-URL: Changelog, https://github.com/chuksforge/goldenrun/releases
Keywords: llm,evaluation,eval,regression-testing,ci-cd,llm-as-judge,prompt-testing,model-evaluation,mlops,llmops
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE:Zone.Identifier
Requires-Dist: structlog>=24.1.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: tenacity>=8.2.0
Provides-Extra: metrics
Requires-Dist: nltk==3.9.1; extra == "metrics"
Requires-Dist: rouge-score==0.1.2; extra == "metrics"
Requires-Dist: numpy==1.26.4; extra == "metrics"
Requires-Dist: scipy==1.14.1; extra == "metrics"
Requires-Dist: pandas==2.2.3; extra == "metrics"
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.24.0; extra == "dev"
Dynamic: license-file

# GoldenRun

**The evaluation and regression engine behind EvalCI, a hosted CI/CD platform for LLM quality.**

GoldenRun is the open sourced algorithmic core of the platform. It
includes LLM-as-judge scoring, lexical evaluation, explicit-baseline
regression detection, trend analysis, anomaly detection, and the
production test suite that validates them.

It intentionally excludes the hosted platform's multi-tenant API,
dashboard, authentication, scheduling, and notification infrastructure.
See `docs/architecture.md` for the complete architecture and repository boundary.

## See it work in 30 seconds

No Docker. No database. No API key.

``` bash
pip install goldenrun
python examples/regression_demo.py
```

Expected output:

``` text
-- Case A: real regression injected --
  [CRITICAL] overall: 0.910 -> 0.820 (delta +0.090)
  [CRITICAL] factual_accuracy: 0.890 -> 0.800 (delta +0.090)
  summarize() -> regression_detected=True, severity=critical

-- Case B: clean run, same baseline --
  alerts: []

  trend_direction: degrading
  slope:           -0.0120
  r_squared:       0.7412
  latest anomaly:  True (z_score=-2.790)
```

The demo executes the same `RegressionDetector` and `TrendAnalyzer` used
by the library. `examples/eval_demo.py` demonstrates BLEU, ROUGE, F1,
and exact-match scoring while illustrating where lexical metrics are and
are not appropriate.

## Quick API example

``` python
from goldenrun.regression import RegressionDetector

detector = RegressionDetector(...)
summary = detector.compare(baseline_run, current_run)

if summary.regression_detected:
    print(summary.severity)
```

## Try it on your own data

``` bash
python examples/compare_runs.py \
  --baseline path/to/baseline.json \
  --current path/to/current.json \
  --pipeline-type factuality
```

Input is a JSON file containing an `aggregate_scores` object. Sample
datasets are provided in `examples/sample_data/`.

The command exits with:

-   `0` --- no regression detected
-   `1` --- regression detected

making it suitable for CI pipelines.

## Architecture

``` text
Evaluation Runs
       │
       ▼
Metrics & LLM Judge
       │
       ▼
RegressionDetector
       │
       ├── Severity Classification
       ├── Per-metric Analysis
       ▼
TrendAnalyzer + AnomalyDetector
       │
       ▼
Decision / CI Gate
```

## What's included

  -----------------------------------------------------------------------
  Module                              Purpose
  ----------------------------------- -----------------------------------
  `goldenrun/evals/`                  Evaluation pipelines

  `goldenrun/metrics/`                LLM-as-judge, BLEU, ROUGE, F1,
                                      Exact Match, composite scoring

  `goldenrun/models/`                 Model adapter interface with
                                      OpenRouter implementation

  `goldenrun/regression/`             Explicit-baseline regression
                                      detection and drill-down

  `goldenrun/analytics/`              Trend fitting and multivariate
                                      anomaly detection
  -----------------------------------------------------------------------

## Repository scope

Included:

-   Regression detection
-   Trend analysis
-   Anomaly detection
-   Evaluation metrics
-   Production test suite

Not included:

-   Multi-tenancy
-   Authentication
-   Dashboard
-   Hosted scheduling
-   Notification delivery
-   GitHub integration

These components belong to the hosted platform rather than the
evaluation engine itself.

## Installation

``` bash
pip install goldenrun
pip install "goldenrun[metrics]"
```

Both installation paths are supported and exercised by the examples.

## Development

``` bash
git clone https://github.com/chuksforge/goldenrun
cd goldenrun
pip install -e ".[metrics,dev]"
pytest tests/ -v
```

## Further reading

-   `docs/architecture.md` explains the complete production architecture
    and includes a real scheduling incident and its resolution.
-   Engineering write-ups discuss lessons learned while building the
    hosted platform.

## License

Apache 2.0. Built and maintained by Christian Chukwuka, ChuksForge AI
Solutions Ltd.
