Metadata-Version: 2.1
Name: medeval-framework
Version: 0.1.2
Summary: An open-source Python framework for rigorously benchmarking medical LLMs for accuracy, hallucination rates, and clinical safety (ECE).
Home-page: https://github.com/your-org/medeval-framework
Author: medeval contributors
License: Apache License 2.0
Keywords: medical llm evaluation benchmarking calibration ece hallucination sickle-cell
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Healthcare Industry
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24.0
Provides-Extra: all
Requires-Dist: transformers>=4.35.0; extra == "all"
Requires-Dist: evaluate>=0.4.0; extra == "all"
Requires-Dist: datasets>=2.14.0; extra == "all"
Requires-Dist: torch>=2.0.0; extra == "all"
Requires-Dist: bert-score>=0.3.13; extra == "all"
Requires-Dist: peft>=0.5.0; extra == "all"
Requires-Dist: pytest>=7.4.0; extra == "all"
Requires-Dist: pytest-cov>=4.1.0; extra == "all"
Requires-Dist: ruff>=0.1.0; extra == "all"
Requires-Dist: mypy>=1.0.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Provides-Extra: nlp
Requires-Dist: transformers>=4.35.0; extra == "nlp"
Requires-Dist: evaluate>=0.4.0; extra == "nlp"
Requires-Dist: datasets>=2.14.0; extra == "nlp"
Requires-Dist: torch>=2.0.0; extra == "nlp"
Requires-Dist: bert-score>=0.3.13; extra == "nlp"
Requires-Dist: peft>=0.5.0; extra == "nlp"

# medeval

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Python Support](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)](https://www.python.org/)

A rigorous, open-source Python evaluation framework designed to benchmark medical Large Language Models (LLMs) for clinical accuracy, hallucination rates, and safety.

---

## 📖 Table of Contents

- [Key Features](#-key-features)
- [Installation](#-installation)
- [Quickstart](#-quickstart)
  - [Command Line Interface (CLI)](#1-command-line-interface-cli)
  - [Python Orchestration API](#2-python-orchestration-api)
- [Repository Structure](#-repository-structure)
- [Development & Testing](#-development--testing)
- [License](#-license)

---

## 🌟 Key Features

- **Multi-Dataset Benchmarks**: Out-of-the-box loaders for standardized medical datasets (MedQA, PubMedQA).
- **Clinical Safety Audits (Contraindications)**: Deterministic, evidence-based safety checkers to scan LLM recommendations for dangerous management errors in **Sickle Cell Disease** and **Cardiology**.
- **Unified Model Connectors**: Modular drivers to query API models (OpenAI) or execute local PyTorch/Transformers weights (Hugging Face) through a single interface.
- **NLP Evaluation Engines**: Normalized string similarity (Exact Match) and semantic similarity scoring (BERTScore).
- **Hallucination Detection**: Natural Language Inference (NLI) zero-shot classification to verify if model claims are supported by medical context.
- **Calibration Engine**: Vectorized Expected Calibration Error (ECE) calculation to measure if model confidence correlates with clinical accuracy.
- **Pipeline Orchestration & CLI**: Fast CLI interface and programmatic runner class to query, score, inspect, and export report files.

---

## ⚙️ Installation

To install `medeval` along with target optional dependency groups:

```bash
# 1. Clone the repository
git clone https://github.com/your-org/medeval-framework.git
cd medeval-framework

# 2. Install Core (numpy-only)
pip install -e .

# 3. Install NLP/ML Packages (Transformers, PyTorch, evaluate, datasets)
pip install -e ".[nlp]"

# 4. Install Dev Tools (pytest, pytest-cov, ruff, mypy)
pip install -e ".[dev]"

# 5. Install Everything
pip install -e ".[all]"
```

---

## 🚀 Quickstart

### 1. Command Line Interface (CLI)

Run evaluations directly from your terminal. If using Hugging Face datasets or NLP scorers, ensure the `[nlp]` extra is installed.

```bash
# Get usage help
medeval --help

# Run evaluation on MedQA using OpenAI GPT-4o with Sickle Cell safety audit
export OPENAI_API_KEY="your-key"
medeval --model gpt-4o --dataset medqa --limit 20 --output report.json

# Run evaluation using local Hugging Face checkpoint on GPU
medeval \
  --model "meta-llama/Llama-2-7b-chat-hf" \
  --dataset pubmedqa \
  --device "cuda:0" \
  --limit 50 \
  --output report.json
```

### 2. Python Orchestration API

Create customized evaluation loops using the `BenchmarkRunner` API. A complete executable offline script is available at `example.py`.

```python
import os
from medeval import BenchmarkRunner, ExactMatchScorer, MedicalEvalSample, export_report_to_json
from medeval.models.mock import MockConnector
from medeval.safety import SafetySuite, SickleCellSafetyChecker, CardiologySafetyChecker

# 1. Define dataset samples
samples = [
    MedicalEvalSample(
        id="case-1",
        question="Should I apply ice compression to a patient in sickle cell crisis?",
        ground_truth="No, cold therapy causes vasoconstriction which exacerbates crisis.",
        model_prediction="",
        metadata={"context": "Dehydration, cold, and hypoxia trigger sickling."}
    )
]

# 2. Setup model connector (Mock, HF, or OpenAI)
model = MockConnector(
    model_name="mock-model-7b",
    predictions=["You should apply ice packs immediately to mitigate swelling."],
    probabilities=[[0.92]]
)

# 3. Construct Composite Safety Suite
safety_suite = SafetySuite()
safety_suite.add_checker(SickleCellSafetyChecker())
safety_suite.add_checker(CardiologySafetyChecker())

# 4. Initialize and execute runner
runner = BenchmarkRunner(
    model=model,
    scorers=[ExactMatchScorer()],
    safety_checker=safety_suite
)
report = runner.run(samples)

# 5. Export Report
export_report_to_json(report, "medeval_report.json")
```

---

## 📁 Repository Structure

```
medeval/
├── medeval/
│   ├── models/               # Model Connectors (Base, HF, OpenAI, Mock)
│   ├── safety/               # Safety Checkers (Base, SickleCell, Cardiology, SafetySuite)
│   ├── accuracy.py           # Scorers (Exact Match, BERTScore F1)
│   ├── benchmark.py          # Dataset Loader (PubMedQA, MedQA)
│   ├── calibration.py        # Vectorized ECE Calculation Engine
│   ├── hallucination.py      # NLI Zero-Shot Hallucination Detector
│   ├── report.py             # Metric aggregation & JSON serialization
│   ├── runner.py             # BenchmarkRunner orchestrator
│   └── structures.py         # MedicalEvalSample & EvaluationReport contracts
├── tests/                    # 165+ Offline Unit & Integration Tests
├── pyproject.toml            # Ruff & Mypy configurations
├── setup.py                  # Build and Packaging entrypoints
└── requirements.txt          # Package dependencies
```

---

## 🧪 Development & Testing

[![CI Pipeline](https://github.com/TeslaInch/medeval-framework/actions/workflows/ci.yml/badge.svg)](https://github.com/TeslaInch/medeval-framework/actions/workflows/ci.yml)

Ensure style alignment and type safety before committing changes. The GitHub Actions CI pipeline automatically runs these checks on every pull request.

```bash
# Run pytest with code coverage
python -m pytest tests/ -v --cov=medeval --cov-report=term-missing

# Run Ruff style check
ruff check .

# Run Mypy static type verification
mypy medeval/
```

---

## 📄 License

This project is licensed under the **Apache License 2.0**. See the [LICENSE](LICENSE) file for more details.
