Metadata-Version: 2.4
Name: modelgauntlet
Version: 0.1.0a1
Summary: Run every model through the same gauntlet
Project-URL: Homepage, https://github.com/Suraj-Bangade/modelgauntlet
Project-URL: Repository, https://github.com/Suraj-Bangade/modelgauntlet
Project-URL: Issues, https://github.com/Suraj-Bangade/modelgauntlet/issues
License: MIT License
        
        Copyright (c) 2026 ModelGauntlet contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: agents,benchmark,evaluation,inference,llm
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: duckdb>=1.1
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.12
Provides-Extra: deepeval
Requires-Dist: deepeval; extra == 'deepeval'
Provides-Extra: dev
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: evalscope
Requires-Dist: evalscope; extra == 'evalscope'
Provides-Extra: guidellm
Requires-Dist: guidellm==0.4.0; extra == 'guidellm'
Provides-Extra: harbor
Requires-Dist: harbor==0.20.0; (python_version >= '3.12') and extra == 'harbor'
Provides-Extra: inspect
Requires-Dist: inspect-ai; extra == 'inspect'
Requires-Dist: inspect-evals==0.16.0; extra == 'inspect'
Requires-Dist: langdetect; extra == 'inspect'
Requires-Dist: mpmath; extra == 'inspect'
Provides-Extra: litellm
Requires-Dist: litellm; extra == 'litellm'
Description-Content-Type: text/markdown

# ModelGauntlet

**Run every model through the same gauntlet.**

ModelGauntlet is an open-source benchmark runner for comparing commercial,
self-hosted, and routed language models through one reproducible workflow. It
connects model providers to local diagnostics, maintained external evaluations,
agent environments, and serving benchmarks, then writes normalized results and
auditable reports.

Install the latest release from [PyPI](https://pypi.org/project/modelgauntlet/):

```bash
pip install modelgauntlet
```

It is designed for apples-to-apples comparison without pretending that every
benchmark measures the same thing:

- **Capability** — reasoning, coding, instruction following, long context,
  structured output, and tool use.
- **Agent system** — an agent harness plus model, evaluated by outcome-verified
  tasks.
- **Deployment** — reliability, latency, throughput, and serving behavior.

ModelGauntlet is an orchestration and comparison layer. It does not reimplement
or vendor the datasets and scorers maintained by upstream benchmark projects.

## Contents

- [Why ModelGauntlet](#why-modelgauntlet)
- [Installation](#installation)
- [Quick start](#quick-start)
- [Model configuration](#model-configuration)
- [Benchmark suites](#benchmark-suites)
- [Scoring and comparison](#scoring-and-comparison)
- [Judging](#judging)
- [Agent skill](#agent-skill)
- [Outputs](#outputs)
- [Sample result](#sample-result)
- [Optional integrations](#optional-integrations)
- [Private evaluations](#private-evaluations)
- [Adding benchmarks](#adding-benchmarks)
- [Development](#development)
- [Security](#security)
- [Third-party notices](#third-party-notices)
- [License](#license)

## Why ModelGauntlet

Many evaluation tools focus on one model API, one benchmark family, or one
metric. ModelGauntlet provides a common run manifest and result schema across
those boundaries:

- Configure hosted APIs, OpenAI-compatible servers, Ollama, Open WebUI, or
  custom HTTP endpoints as model records.
- Run the same suite against one model, many models, or a historical collection
  of runs.
- Keep objective grading, model judging, agent verification, and serving
  telemetry distinct.
- Record benchmark releases, harness versions, parameters, prompt hashes,
  deployment fingerprints, and runtime fingerprints.
- Preserve raw artifacts alongside JSONL results and materialized Parquet and
  DuckDB data.
- Refuse or clearly label incomplete, incomparable, or environment-drifted
  comparisons.

The design follows practical conventions used by projects such as
[Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai),
[Inspect Evals](https://github.com/UKGovernmentBEIS/inspect_evals),
[lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness),
and [SWE-bench](https://github.com/SWE-bench/SWE-bench): a small core,
optional backends, explicit task configuration, reproducible outputs, and
upstream-owned benchmark implementations.

## Installation

ModelGauntlet requires Python 3.11 or newer. For normal use, install the
published package:

```bash
pip install modelgauntlet
```

For checkout-based development, [uv](https://docs.astral.sh/uv/) is recommended:

```bash
uv sync
```

The base installation includes the core runner and provider adapters. Optional
integrations are installed separately:

```bash
uv sync --extra inspect    # Inspect AI and Inspect Evals
uv sync --extra harbor     # Harbor; Python 3.12+ and Docker required
uv sync --extra guidellm   # serving/load benchmarks
uv sync --extra litellm    # LiteLLM provider support
uv sync --extra dev        # tests, coverage, and Ruff
```

The pinned IFEval implementation has an upstream Git dependency that cannot be
represented in PyPI package metadata. Install it separately only when the
Inspect environment requires it:

```bash
uv pip install \
  "instruction-following-eval @ git+https://github.com/josejg/instruction_following_eval@0c495b2"
```

After installation, both `modelgauntlet` and the shorter `mg` command are
available.

## Quick start

Create a local configuration from the example:

```bash
cp configs/models.example.yaml configs/models.yaml
```

Set the required environment variables, then validate and probe the models:

```bash
export OPENAI_API_KEY=...
export OPENAI_MODEL=...

uv run modelgauntlet models validate
uv run modelgauntlet models probe
```

Run the diagnostic suite:

```bash
uv run modelgauntlet run \
  --models local-openai-compatible \
  --suite core
```

Generate reports using the returned run ID:

```bash
uv run modelgauntlet report <run-id> \
  --formats html,markdown,json,csv
uv run modelgauntlet analyze <run-id> --profile capability
```

The `core`, `comprehensive`, and `stress` suites are diagnostic regression
suites. They are useful for checking prompts, providers, retries, timeouts, and
local behavior; they should not be presented as broad public model-quality
claims.

For maintained objective benchmarks:

```bash
uv sync --extra inspect
uv run modelgauntlet run \
  --models candidate \
  --suite objective-smoke
```

Use `objective` for the full configured objective catalog. Full external runs
can be expensive and may require additional upstream dependencies or explicit
privileged-container consent.

## Model configuration

Models are declared in YAML. API credentials are referenced by environment
variable name and should never be written directly into configuration:

```yaml
models:
  - id: local-vllm
    protocol: openai_compatible
    base_url: http://localhost:8000/v1
    model: served-model
    capabilities:
      chat: true
      tools: true
      streaming: true
      usage: true
    deployment:
      engine: vllm
      hardware: unknown

  - id: hosted-model
    protocol: openai
    model: ${OPENAI_MODEL}
    api_key_env: OPENAI_API_KEY
```

Supported protocols include:

| Protocol | Typical use |
| --- | --- |
| `openai` | OpenAI hosted API |
| `anthropic` | Anthropic Messages API |
| `openai_compatible` | vLLM, SGLang, TGI, local gateways, and proxies |
| `ollama` | Ollama server |
| `openwebui` | Open WebUI gateway |
| `custom_http` | Templated HTTP request/response mapping |
| `litellm` | Optional LiteLLM routing |

Give separately routed endpoints separate model IDs. This keeps proxy and
deployment behavior visible in reports instead of hiding it behind one name.
See [configs/models.example.yaml](configs/models.example.yaml) and
[gauntlet.yaml](gauntlet.yaml).

## Benchmark suites

Suites live in [suites/](suites/) and specify the engine, benchmark releases,
comparison mode, repetitions, concurrency, retries, timeouts, and score
weights.

| Suite | Purpose | Engine |
| --- | --- | --- |
| `smoke` | Small local diagnostic run | `core` |
| `core` | Local diagnostic regression suite | `core` |
| `comprehensive` | Larger local diagnostic coverage | `core` |
| `stress` | Local concurrency and timeout testing | `core` |
| `objective-smoke` | Small external capability run | Inspect |
| `objective` | Full configured capability catalog | Inspect |
| `livebench` | Pinned LiveBench environment | Inspect |
| `agent-system` | Outcome-verified terminal agents | Harbor |
| `serving` | Load and serving behavior | GuideLLM |

The external catalog in [benchmarks/catalog.yaml](benchmarks/catalog.yaml)
records the upstream task, release, harness version, profile, metric contract,
source, and license pointer. Validate suites before running them:

```bash
uv run modelgauntlet suites validate objective-smoke
uv run modelgauntlet suites validate objective
```

### Recommended evaluation workflow

Do not run the largest possible catalog for every model change. Use staged
evaluation:

1. **Smoke**: run a small local suite to verify connectivity, configuration,
   grading, and basic regressions.
2. **Targeted**: select a small external subset while developing. Use the
   benchmark that matches the capability you are changing—for example, BFCL
   for tool use, IFEval for instruction following, or InfiniteBench for
   long-context behavior.
3. **Release evaluation**: run the broader external catalog when you need a
   durable model comparison or release report. These runs can be slow and
   expensive, especially long-context and agent/container benchmarks.
4. **Serving evaluation**: run `serving` separately when measuring throughput,
   latency, or GPU utilization. Do not mix serving load with capability claims.

External selections support a configurable item limit for practical diagnostic
runs. A limited run is useful for validation and iteration, but its scores have
low statistical resolution and should not be presented as definitive model
quality claims. Full suites are best reserved for scheduled evaluations,
benchmark cards, or research comparisons. See
[docs/benchmark-authoring.md](docs/benchmark-authoring.md) for selection and
release-pinning guidance.

## Scoring and comparison

Every run produces a normalized item-level record. Reports expose three
scorecards rather than one opaque leaderboard score:

- **Capability** uses graded benchmark items and objective or subjective
  quality.
- **Agent-system** uses stable agent-plus-model subjects and verifier rewards.
- **Deployment** uses deployment records for serving reliability, latency, and
  throughput.

Legacy dimensions such as quality, reliability, efficiency, serving performance,
and agent effectiveness remain available for compatibility. Missing telemetry is
`null`, not zero. Missing weighted dimensions do not silently improve a
headline score.

Run comparisons require compatible benchmark releases, harness versions,
parameters, prompt hashes, score weights, and strict/native modes. Historical
comparisons recompute peer-relative efficiency and serving scores across the
selected runs:

```bash
uv run modelgauntlet compare <older-run-id> <newer-run-id>
```

Runtime and deployment fingerprints prevent latency comparisons across changed
client or serving environments. To compare quality while suppressing serving
scores:

```bash
uv run modelgauntlet compare <run-a> <run-b> \
  --allow-environment-drift
```

See [docs/scoring.md](docs/scoring.md) for the scoring model and
[references/report-interpretation.md](references/report-interpretation.md) for
how to read reports.

## Judging

Prefer deterministic assertions whenever possible. Local packs support exact,
containment, regex, numeric, JSON, and tool-call assertions. Rubrics and
pairwise judgments are available for work that cannot be graded reliably by
deterministic checks.

Use a configured model as an LLM judge:

```bash
uv run modelgauntlet judge <run-id> --judge local-judge
```

The active Cursor, Codex, or Claude Code harness can explicitly judge blinded
tasks:

```bash
uv run modelgauntlet judge <run-id> --judge agent:self
# inspect only agent-judge-tasks.jsonl, then write one JSONL answer per task
uv run modelgauntlet judge-import <run-id> \
  --responses judgments.jsonl \
  --harness cursor
```

`agent:self` is never selected automatically. Its identity is recorded as an
agent harness and self-judge conflicts require explicit acknowledgement. See
[references/judging-policy.md](references/judging-policy.md).

## Agent skill

[`SKILL.md`](SKILL.md) provides the workflow for an AI coding agent operating
ModelGauntlet. It covers model validation, suite selection, preflight cost and
runtime warnings, external benchmark safeguards, profile-specific analysis,
historical comparisons, human-readable reports, and explicit `agent:self`
judging.

The skill includes representative evaluation prompts in
[evals/evals.json](evals/evals.json) so its workflow can be checked against
provider comparison, full capability evaluation, datacenter GPU serving, and
self-judging requests.

## Outputs

Each run is stored under `results/<run-id>/`:

```text
run.json              # manifest, fingerprints, configuration, and status
results.jsonl         # canonical item-level result records
results.parquet       # columnar materialization
run.duckdb             # queryable result and manifest metadata
raw-artifacts/        # provider responses and upstream artifacts
reports/              # generated HTML, Markdown, JSON, and CSV reports
```

External harness runs are immutable audit units. They cannot resume in place;
`rerun --all` creates a new run. Partial external runs return a nonzero exit
status unless `--allow-partial` is explicit, and remain non-publishable:

```bash
uv run modelgauntlet rerun <external-run-id> --all
uv run modelgauntlet run --models candidate --suite objective-smoke \
  --allow-partial
```

## Sample result

Results are designed to be readable by people first and queryable by tools
second. The report starts with an executive summary, then separates capability,
agent-system, and deployment scorecards. It also explains missing coverage,
confidence intervals, cost, latency, and reproducibility fingerprints.

Example human-readable report:

```text
# ModelGauntlet report: sample-run

## Executive summary

Capability quality: 82.4 / 100
Capability coverage: 100.0%
Reliability: 98.7%
Median latency: 420 ms
P95 latency: 680 ms
Output throughput: 126.5 tokens/sec
Estimated cost: $3.42
Quality 95% CI: 79.8–84.6

Capability: 82.4 (publishable)
Agent system: n/a (not measured)
Deployment: 88.6 (publishable)
```

The complete synthetic examples are in
[examples/sample-results/](examples/sample-results/):

- [`executive-summary.md`](examples/sample-results/executive-summary.md) is
  the human-readable version.
- [`comparison.json`](examples/sample-results/comparison.json) is the
  machine-readable scorecard.

These examples are illustrative and are not results from a real model.

## Optional integrations

The integrations are boundaries around upstream tools, not core dependencies:

- [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) and
  [Inspect Evals](https://github.com/UKGovernmentBEIS/inspect_evals) provide
  maintained capability benchmarks.
- [Harbor](https://github.com/laude-institute/harbor) provides containerized,
  outcome-verified agent tasks.
- [GuideLLM](https://github.com/vllm-project/guidellm) provides serving
  profiles and load measurements.
- NVIDIA DCGM provides optional GPU telemetry for local serving runs.

`modelgauntlet doctor` reports which optional executables are installed:

```bash
uv run modelgauntlet doctor
```

LiveBench intentionally uses an isolated upstream environment because its
dependencies can conflict with the shared Inspect installation. Set
`MODELGAUNTLET_LIVEBENCH_INSPECT` to the pinned environment's `inspect`
executable as described in [suites/livebench.yaml](suites/livebench.yaml).

## Private evaluations

Private rotating evaluations stay outside this repository. Point
`MODELGAUNTLET_PRIVATE_CATALOGS` at one or more catalog YAML files:

```bash
export MODELGAUNTLET_PRIVATE_CATALOGS=/secure/path/private-catalog.yaml
```

Mark entries `visibility: private` and `public_item_ids: false`. ModelGauntlet
hashes item identifiers, avoids packaging private task content, and emits
metadata-only benchmark cards.

## Adding benchmarks

For local diagnostic packs, add a YAML file under
[benchmarks/prompts/](benchmarks/prompts/) with:

- stable test IDs;
- category and difficulty;
- explicit point values;
- deterministic assertions or a rubric;
- license metadata in [benchmarks/LICENSES.yaml](benchmarks/LICENSES.yaml).

For an external evaluation, add a catalog entry with an exact release, adapter,
upstream task, harness version, primary metric, declared range, source URL, and
aggregation policy. ModelGauntlet uses a versioned adapter envelope containing
the adapter identity, protocol version, expected item count, observed item
count, artifacts, and normalized records.

See [docs/benchmark-authoring.md](docs/benchmark-authoring.md).

## Development

```bash
uv sync --extra dev
uv run ruff check .
uv run pytest --cov=modelgauntlet --cov-report=term-missing \
  --cov-fail-under=80
uv build
```

Contributions should preserve result-schema compatibility, include tests for
new behavior, and never commit credentials, private prompts, model outputs, or
generated result directories. See [CONTRIBUTING.md](CONTRIBUTING.md).

## Security

Model outputs and provider responses are untrusted. The core runner does not
execute generated code. Use the isolated Inspect or Harbor environments for
execution-scored coding and agent tasks. Never put credentials in model YAML,
request templates, benchmark prompts, or committed artifacts.

Report vulnerabilities privately through the repository's security advisory
feature. See [SECURITY.md](SECURITY.md).

## Third-party notices

See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for the projects,
benchmarks, dependencies, and GPU tools ModelGauntlet integrates with or
references, including attribution and redistribution boundaries.

## License

ModelGauntlet is released under the [MIT License](LICENSE). See
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for upstream benchmark,
dependency, engine, and GPU-tool attribution and redistribution boundaries.
