Metadata-Version: 2.4
Name: evalkit-ml
Version: 0.1.1
Summary: ML model evaluation toolkit
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: scikit-learn
Requires-Dist: pyyaml>=6.0
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"

# EvalKit

[![CI](https://img.shields.io/badge/CI-GitHub%20Actions-blue)](.github/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](pyproject.toml)
[![Version](https://img.shields.io/badge/version-0.1.0-informational)](pyproject.toml)
[![License](https://img.shields.io/badge/license-not%20specified-lightgrey)](pyproject.toml)

**A lightweight, production-focused evaluation and monitoring toolkit for scikit-learn models.**

EvalKit helps you evaluate models, track performance, detect data drift, compare runs, enforce custom evaluation rules, and integrate model checks directly into CI/CD.

```text
Train → Predict → Evaluate → Monitor → Compare → Validate → CI/CD
```

## Why EvalKit?

Model evaluation often ends with a few metrics printed in a notebook.

EvalKit turns that into a repeatable engineering workflow:

* 📊 **Model evaluation** — classification and regression metrics
* 🔍 **Diagnostics** — confusion matrices, class distributions, and dataset profiling
* 📈 **Performance tracking** — training and prediction latency
* 🧪 **Run history** — save and compare evaluation results
* 🔄 **Data drift detection** — numeric and categorical feature drift
* 🏷️ **Model versioning** — deterministic model configuration versions
* 🚨 **Regression detection** — detect performance degradation against previous runs
* 🧩 **Custom metrics & rules** — extend evaluation for your own requirements
* 🔌 **Monitoring hooks** — connect evaluation results to external workflows
* 🤖 **CI/CD integration** — automatically fail builds when evaluation rules fail
* 🖥️ **CLI** — run evaluations and comparisons outside Python
* 📦 **JSON reports** — save evaluation results for automation and analysis

---

## Installation

### From source

```powershell
python -m pip install -e .
```

Once installed, the `evalkit` CLI is available:

```powershell
evalkit --help
```

---

## Quick Start

```python
from evalkit import track

tracker = track(model)

tracker.fit(X_train, y_train)

tracker.predict(X_test)

report = tracker.evaluate(y_test)

report.show()
```

Save the evaluation:

```python
report.save("current.json")
```

A report contains information such as:

```text
Model
Version
Metrics
Diagnostics
Dataset Profile
Performance
Drift
Timestamp
```

---

## Model Evaluation

EvalKit automatically detects whether the model is a classifier or regressor.

### Classification

Supported metrics include:

* Accuracy
* Precision
* Recall
* F1

### Regression

Supported metrics include:

* MAE
* MSE
* RMSE
* R²

Example:

```python
tracker = track(model)

tracker.fit(X_train, y_train)
tracker.predict(X_test)

report = tracker.evaluate(y_test)
report.show()
```

---

## Data & Feature Monitoring

EvalKit profiles the datasets used during evaluation and can detect changes between training and evaluation data.

### Numeric features

Uses **Population Stability Index (PSI)** to identify distribution changes.

### Categorical features

Compares categorical distributions and reports drift using PSI.

Example report:

```text
Data Drift:

Numeric:
  age:    PSI=0.0842  Status=stable
  income: PSI=0.7341  Status=drifted

Categorical:
  city:   PSI=0.4217  Status=drifted
```

---

## Model Versioning

EvalKit automatically generates a deterministic version from the model configuration.

For example:

```text
RandomForestClassifier
→ v-caaca18c
```

Changing the model configuration produces a different version.

This makes it possible to identify which model configuration produced an evaluation result.

---

## Run History & Comparison

History is explicit rather than globally shared.

```python
tracker = track(
    model,
    history_file="history.json"
)
```

This allows separate models or experiments to maintain independent histories.

Compare two reports:

```powershell
evalkit compare `
  --current current.json `
  --previous previous.json
```

Example:

```text
ACCURACY
  0.9412 -> 0.9125
  Change: -0.0287
  Status: DEGRADED
```

---

## Regression Checks

EvalKit can detect whether a model's performance has degraded compared with a previous evaluation.

```powershell
evalkit check `
  --report current.json `
  --previous previous.json `
  --threshold accuracy=0.02 `
  --threshold f1=0.02
```

Metric direction is handled automatically.

For example:

```text
Accuracy ↓  → degradation
RMSE ↑      → degradation
R² ↑        → improvement
```

---

## CI/CD

EvalKit can act as a quality gate in CI pipelines.

```powershell
evalkit check `
  --report current.json `
  --previous previous.json
```

Exit codes:

```text
0 → Passed
1 → Regression / rule failure
```

This makes EvalKit suitable for automated CI/CD workflows.

### GitHub Actions

A typical workflow can simply run:

```yaml
- name: Run EvalKit checks
  run: evalkit check --report current.json --previous previous.json
```

A failed evaluation causes the CI job to fail automatically.

---

## Configuration

Create an `evalkit.yaml` file:

```yaml
thresholds:
  accuracy: 0.02
  precision: 0.02
  recall: 0.02
  f1: 0.02

rules:
  accuracy:
    min: 0.8
```

This keeps evaluation policy separate from application code.

---

## Custom Metrics

Register reusable metrics:

```python
from evalkit import register_metric

register_metric(
    "custom_score",
    my_metric_function
)
```

Or provide metrics directly:

```python
tracker = track(
    model,
    metrics={
        "custom_score": my_metric_function
    }
)
```

This allows EvalKit to support domain-specific evaluation without modifying the core library.

---

## Custom Evaluation Rules

Define rules for model quality requirements.

For example:

```yaml
rules:
  accuracy:
    min: 0.8
```

A rule failure can cause the CI check to return exit code `1`.

This lets you enforce requirements such as:

```text
Accuracy must be ≥ 0.80
F1 must be ≥ 0.75
RMSE must be ≤ 50
```

---

## Monitoring Hooks

Evaluation results can be passed to monitoring hooks.

Hooks are isolated from model evaluation, meaning a monitoring hook failure does **not** silently break the model evaluation itself.

This makes integrations safer for production workflows.

---

## CLI

Available commands:

```powershell
evalkit check --report current.json --previous previous.json

evalkit compare --current current.json --previous previous.json

evalkit history --file evalkit_history.json
```

For automation, output formats are available:

```powershell
evalkit check ... --output json
```

or:

```powershell
evalkit check ... --output github
```

---

## Architecture

```text
                    ┌─────────────────┐
                    │   scikit-learn  │
                    │      Model      │
                    └────────┬────────┘
                             │
                          track()
                             │
                    ┌────────▼────────┐
                    │  ModelTracker   │
                    └────────┬────────┘
                             │
          ┌──────────────────┼──────────────────┐
          │                  │                  │
          ▼                  ▼                  ▼
      Evaluation         Monitoring          Versioning
          │                  │                  │
          ▼                  ▼                  ▼
       Metrics             Drift             Model ID
      Diagnostics        Features
      Profiling          Predictions
          │                  │
          └──────────┬───────┘
                     ▼
              EvaluationReport
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
       History    Compare      Rules
          │          │          │
          └──────────┼──────────┘
                     ▼
                 CI / CLI
```

---

## Design Goals

EvalKit is designed around a few principles:

**Simple**

Wrap an existing scikit-learn model without changing your ML workflow.

**Explicit**

History and monitoring configuration should be controlled by the user rather than hidden global state.

**Extensible**

Custom metrics, rules, and monitoring hooks can be added without modifying the core evaluator.

**CI-friendly**

Evaluation results should be machine-readable and capable of failing a build when quality requirements are not met.

**Production-focused**

Evaluation should include more than a single accuracy number: performance, data quality, drift, model version, and reproducibility all matter.

---

## Development

Clone the repository and install it in editable mode:

```powershell
python -m pip install -e .
```

Run the test suite:

```powershell
pytest
```

Run the examples:

```powershell
python examples/basic_classification.py

python examples/regression.py

python examples/drift.py

python examples/versioning.py
```

---

## Project Status

EvalKit is currently under active development.

The core evaluation, monitoring, comparison, versioning, CLI, and CI/CD workflows are implemented and covered by automated validation.

Current validation includes:

```text
5 tests passed
Classification example     ✓
Regression example         ✓
CLI check/compare/history  ✓
CI exit codes 0/1          ✓
Custom metrics             ✓
Custom rules               ✓
Monitoring hooks           ✓
Edge cases                 ✓
```

---

## License

Add your project license here.

---

**EvalKit — evaluate models like production systems, not just notebooks.**
