Metadata-Version: 2.5
Name: finetunecheck
Version: 2.0.0
Summary: Automated base vs fine-tuned LLM comparison with forgetting detection, capability retention scoring, and visual diff reports.
Project-URL: Homepage, https://github.com/shuhulx/finetunecheck
Project-URL: Repository, https://github.com/shuhulx/finetunecheck
Project-URL: Issues, https://github.com/shuhulx/finetunecheck/issues
Project-URL: Changelog, https://github.com/shuhulx/finetunecheck/blob/main/CHANGELOG.md
Author-email: Shuhul Razdan <shuhul.aiml@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: catastrophic-forgetting,deep-learning,evaluation,fine-tuning,forgetting,llm,lora,model-comparison,peft,transformers
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: diskcache>=5.6
Requires-Dist: jinja2>=3.1
Requires-Dist: plotly>=5.18
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: rouge-score>=0.1
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: anthropic>=0.25; extra == 'all'
Requires-Dist: llama-cpp-python>=0.2; extra == 'all'
Requires-Dist: mcp<2.0,>=1.26; extra == 'all'
Requires-Dist: numpy>=1.24; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Requires-Dist: peft>=0.10; extra == 'all'
Requires-Dist: scipy>=1.11; extra == 'all'
Requires-Dist: torch<3.0,>=2.0; extra == 'all'
Requires-Dist: transformers<5.0,>=4.40; extra == 'all'
Requires-Dist: vllm>=0.4; extra == 'all'
Provides-Extra: api-judge
Requires-Dist: anthropic>=0.25; extra == 'api-judge'
Requires-Dist: openai>=1.0; extra == 'api-judge'
Provides-Extra: deep
Requires-Dist: numpy>=1.24; extra == 'deep'
Requires-Dist: peft>=0.10; extra == 'deep'
Requires-Dist: scipy>=1.11; extra == 'deep'
Requires-Dist: torch<3.0,>=2.0; extra == 'deep'
Requires-Dist: transformers<5.0,>=4.40; extra == 'deep'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: playwright>=1.40; extra == 'dev'
Requires-Dist: pyright>=1.1.390; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: gguf
Requires-Dist: llama-cpp-python>=0.2; extra == 'gguf'
Provides-Extra: inference
Requires-Dist: peft>=0.10; extra == 'inference'
Requires-Dist: torch<3.0,>=2.0; extra == 'inference'
Requires-Dist: transformers<5.0,>=4.40; extra == 'inference'
Provides-Extra: mcp
Requires-Dist: mcp<2.0,>=1.26; extra == 'mcp'
Provides-Extra: vllm
Requires-Dist: vllm>=0.4; extra == 'vllm'
Description-Content-Type: text/markdown

# FineTuneCheck

Evidence-aware comparison of base and fine-tuned language models.

[![CI](https://github.com/shuhulx/finetunecheck/actions/workflows/ci.yml/badge.svg)](https://github.com/shuhulx/finetunecheck/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/finetunecheck)](https://pypi.org/project/finetunecheck/)
[![Development Status: Beta](https://img.shields.io/badge/status-beta-blue.svg)](https://pypi.org/project/finetunecheck/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://python.org)
[![License](https://img.shields.io/badge/license-Apache--2.0-green.svg)](LICENSE)

FineTuneCheck measures target-task change, general-capability retention, safety smoke behavior, and sample-level regressions. It preserves the underlying evidence and returns `INSUFFICIENT_EVIDENCE` whenever required measurements are missing, errored, incompatible, or too small for a confident verdict.

Results support investigation. They are not independently sufficient for deployment approval.

## Highlights

- Paired base-versus-fine-tuned evaluation with raw outputs and judge evidence
- Explicit local, OpenAI, Anthropic, or caller-supplied judge providers
- Fail-closed code evaluation through an external `Executor` boundary
- BWT, CRR, SFI, SAR, bounded-score target deltas, paired intervals, and ROI provenance
- Multi-target profiles and compatible multi-run Pareto comparison
- Transformers, vLLM, llama.cpp/GGUF, and local or remote PEFT adapters
- Self-contained HTML plus JSON, CSV, and Markdown reports
- Experimental CKA, rank@k spectral, sliding-window perplexity, calibration, and activation diagnostics
- Nine asynchronous MCP tools

## Install

The core package keeps configuration, metrics, caching, and reporting lightweight:

```bash
pip install finetunecheck
```

Install a model backend for evaluation:

```bash
pip install "finetunecheck[inference]"   # Transformers + PEFT
pip install "finetunecheck[vllm]"        # vLLM
pip install "finetunecheck[gguf]"        # llama.cpp / GGUF
```

Local PEFT adapter directories are detected automatically. Use `peft://ORG/ADAPTER` or
`peft://ORG/ADAPTER@REVISION` for an adapter hosted on Hugging Face.

Other extras:

```bash
pip install "finetunecheck[deep]"        # experimental deep analysis
pip install "finetunecheck[api-judge]"   # OpenAI and Anthropic judge clients
pip install "finetunecheck[mcp]"         # MCP SDK 1.x
```

## Quick check

Quick mode is an offline-runnable evaluation path: it needs local model weights but no API judge. It selects 10 cases from each of math, classification, instruction following, and safety.

```bash
ftcheck quick BASE_MODEL FINETUNED_MODEL --report quick-report.html
```

The bundled cases are small smoke probes, so the verdict will normally be `INSUFFICIENT_EVIDENCE`. That is intentional.

## Full evaluation

LLM-judged probes require a dedicated judge. FineTuneCheck never silently reuses either evaluated model.

```bash
ftcheck run BASE_MODEL FINETUNED_MODEL \
  --profile classification \
  --judge local:JUDGE_MODEL \
  --report report.html
```

API judges are explicit:

```bash
export OPENAI_API_KEY=...
ftcheck run BASE_MODEL FINETUNED_MODEL \
  --profile chat \
  --judge openai:gpt-4o-mini
```

If a required judge is missing, evaluation fails before the base or fine-tuned model is loaded. Unparseable judge output becomes `ERROR`, not a neutral score.

Code probes do not execute generated Python on the host. Without a caller-supplied isolation runtime implementing `Executor`, their status is `NOT_RUN` and the overall verdict is evidence-limited.

## Python API

```python
from finetunecheck.config import EvalConfig, JudgeConfig
from finetunecheck.eval.runner import EvalRunner
from finetunecheck.profiles.loader import ProfileLoader

config = EvalConfig(
    base_model="BASE_MODEL",
    finetuned_model="FINETUNED_MODEL",
    judge=JudgeConfig(provider="local", model="JUDGE_MODEL"),
    device="auto",
)
config = ProfileLoader.apply_to_config("classification", config)

results = EvalRunner(config).run()

print(results.verdict.value)
print(results.target_improvements)
print(results.roi_score, results.roi_coverage)
if results.forgetting:
    print(results.forgetting.backward_transfer)
```

For deterministic smoke evaluation, use `QuickConfig`:

```python
from finetunecheck.config import QuickConfig
from finetunecheck.eval.runner import EvalRunner

results = EvalRunner(
    QuickConfig(base_model="BASE_MODEL", finetuned_model="FINETUNED_MODEL")
).run()
```

`device="auto"` is preserved through Python, CLI, and MCP. The selected inference backend is recorded in result provenance.

## Evidence and verdicts

Every category carries one of these statuses:

- `MEASURED`
- `NOT_RUN`
- `ERROR`
- `INCOMPATIBLE`
- `INSUFFICIENT_SAMPLE`

Overall verdicts are `EXCELLENT`, `GOOD`, `GOOD_WITH_CONCERNS`, `POOR`, `HARMFUL`, or `INSUFFICIENT_EVIDENCE`.

Missing evidence contributes no perfect retention or safety points. Confident verdicts require complete paired measurements, target evidence, full ROI coverage, adequate sample counts, and probe provenance that supports the claim. Even a confident verdict is decision support, not deployment authorization.

## Metrics

| Metric | 2.0.0 meaning |
|---|---|
| Target delta | Fine-tuned minus base bounded score, aggregated as a macro mean across every target |
| BWT | Mean fine-tuned minus base score on non-target categories; higher is better |
| CRR | Fine-tuned/base ratio on non-target categories; undefined near a zero baseline |
| SFI | Dispersion of downside-only retention losses |
| SAR | Safety smoke-score ratio; undefined when safety evidence or its baseline is missing |
| ROI | Versioned weighted composite with component values, weights, and evidence coverage |

The `target_task` field remains as a compatibility alias for the first entry in `target_tasks`.

## Bundled probes

All bundled probes are versioned Apache-2.0 smoke fixtures, not independently validated benchmark datasets.

| Probe | Seed cases | Judge |
|---|---:|---|
| reasoning | 15 | dedicated LLM |
| code | 15 | external isolated executor |
| math | 15 | numeric equivalence |
| safety | 15 | refusal/over-refusal heuristic smoke check |
| chat_quality | 10 | dedicated LLM |
| creative_writing | 8 | dedicated LLM |
| summarization | 10 | ROUGE-L lexical overlap only |
| extraction | 10 | token F1 |
| classification | 12 | exact label |
| instruction_following | 12 | validated constraints |
| multilingual | 10 | dedicated LLM |
| world_knowledge | 15 | exact answer/alias |

Safety reports separate harmful-request refusal from benign over-refusal and detect refusal followed by apparent compliance. The heuristic is not called alignment certification and cannot satisfy the stronger safety requirement in `safety_critical`.

## Profiles

```bash
ftcheck list-profiles
ftcheck list-probes
```

Profiles: `general`, `code`, `chat`, `classification`, `rag`, `medical`, `legal`, and `safety_critical`.

Every target in a profile is evaluated and excluded consistently from retention metrics. `safety_critical` enforces measured SAR >= 0.99 and also requires stronger safety evidence than the bundled heuristic.

## Custom probes

```python
from finetunecheck.probes.custom import CustomProbe
from finetunecheck.probes.registry import ProbeRegistry

probe = CustomProbe.from_csv(
    name="domain_eval",
    csv_path="domain_eval.csv",
    category="domain",
    judge_type="exact_match",
)
ProbeRegistry.register(probe)
```

`CustomProbe.from_jsonl(...)` follows the same pattern. Use sourced, licensed, contamination-reviewed data with enough paired samples when making claims beyond smoke diagnosis.

## Reports and comparison

```bash
ftcheck run BASE_MODEL FINETUNED_MODEL \
  --profile classification \
  --judge local:JUDGE_MODEL \
  --report results.html

ftcheck compare BASE_MODEL RUN_1 RUN_2 RUN_3 \
  --profile classification \
  --judge local:JUDGE_MODEL \
  --report comparison.html
```

HTML reports embed Plotly by default and include statuses, configured ROI weights, selected sample IDs, raw outputs, judge/test evidence, and provenance. Comparison rejects runs with mismatched bases, probe digests, judges, targets, or schema versions and renders all compatible runs.

## MCP

Install `finetunecheck[mcp]`, then configure:

```json
{
  "mcpServers": {
    "finetunecheck": {
      "command": "ftcheck",
      "args": ["serve", "--stdio"]
    }
  }
}
```

The server exposes `evaluate_finetune`, `quick_check`, `detect_forgetting`, `compare_runs`, `get_verdict`, `suggest_fixes`, `generate_report`, `list_profiles`, and `run_probe`. Model work runs through a bounded asynchronous worker gate. Tool failures are protocol errors.

## Development

```bash
pip install -e ".[dev,mcp]"
ruff check .
ruff format --check .
pyright
pytest
python -m build
```

CI tests Python 3.10, 3.11, and 3.12, enforces coverage, builds the package, installs fresh wheels, verifies MCP registration, and loads a report in Chromium.

See [VALIDATION.md](VALIDATION.md), [MIGRATION.md](MIGRATION.md), and [LIMITATIONS.md](LIMITATIONS.md) before interpreting results.

## License

Apache-2.0
