open source · MIT · v0.1.0

Benchmark any LLM
against your actual prompts

Stop guessing. Run gpt-4o, claude-3-5-sonnet, gemini-2.0-flash, and more against your real use cases. Get p50/p95 latency, cost per 1k requests, and optional quality scores — in one command.

$ pip install llm-benchmarker

GitHub

llm-bench run --prompt "Classify sentiment: I love this!" --models gpt-4o,claude-3-5-sonnet,gemini-2.0-flash

Running benchmark: 3 models × 1 prompt × 3 runs

llm-bench results — 3 runs × 1 prompt(s)

  Model                      Provider    p50 (ms)  p95 (ms)  Avg tokens ↑↓   $/1k req   Errors
  ─────────────────────────────────────────────────────────────────────────────────────────────
  gemini-2.0-flash           gemini        289       412      14↑  16↓        $0.002      —
  gpt-4o                     openai        498       631      14↑  19↓        $0.058      —
  claude-3-5-sonnet          anthropic     903      1187      14↑  28↓        $0.119      —

Total time: 8.3s | Requests: 9

What it measures

Everything you need to pick the right model

⚡

Latency (p50 / p95)

Real wall-clock time, not marketing numbers. See the 95th percentile tail latency that your users actually experience.

💰

Cost per 1k requests

Calculated from actual token usage and current pricing. Know your bill before you commit to a model.

🔢

Token counts

Average input and output tokens. Understand why some models cost more — and optimize your prompts.

🧑‍⚖️

Quality scoring

Optional LLM-as-judge mode. Scores coherence and relevance 1-10 using any judge model you choose.

📋

YAML config

Define prompts, models, and settings in a reusable YAML file. Run llm-bench init to get started.

🔧

JSON output

Machine-readable output for CI/CD pipelines. Detect regressions, compare model versions, automate decisions.

Providers

5 providers, 30+ models

Set the relevant API key env var and any model from that provider just works. No config changes needed.

OpenAI — gpt-4o, gpt-4o-mini, o3-mini

Anthropic — claude-opus-4, claude-sonnet-4, claude-3-5

Gemini — gemini-2.5-pro, gemini-2.0-flash

Mistral — mistral-large, codestral, mixtral

Groq — llama-3.3-70b, gemma2-9b (ultra-fast)

API Keys (env vars)

OPENAI_API_KEY ANTHROPIC_API_KEY GEMINI_API_KEY MISTRAL_API_KEY GROQ_API_KEY

Usage

Three ways to run

# Quick single-prompt benchmark
$ llm-bench run \
    --prompt "Write a regex to match email addresses" \
    --models gpt-4o,claude-3-5-sonnet,gemini-2.0-flash

# Multiple prompts, with judge scoring
$ llm-bench run \
    --prompt "Summarize in 2 sentences: ..." \
    --prompt "Classify: positive, negative, neutral: ..." \
    --models gpt-4o-mini,claude-3-5-haiku \
    --judge gpt-4o-mini \
    --runs 5

# JSON output for CI/CD
$ llm-bench run --config benchmark.yaml --json > results.json

# List all supported models
$ llm-bench list-models

# benchmark.yaml
models:
  - gpt-4o
  - claude-3-5-sonnet
  - gemini-2.0-flash
  - llama-3.3-70b-versatile  # Groq

prompts:
  - text: "Classify sentiment: 'Shipping was late but product is great'"
    name: sentiment_mixed

  - text: "Write a Python function to flatten a nested list"
    name: code_gen

judge_model: gpt-4o-mini
n_runs: 5
temperature: 0.0
output: results.json

import asyncio
from llm_bench.benchmark import BenchmarkConfig, run_benchmark
from llm_bench.reporter import print_results_table

config = BenchmarkConfig(
    models=["gpt-4o", "claude-3-5-sonnet", "gemini-2.0-flash"],
    prompts=["What are the key differences between TCP and UDP?"],
    n_runs=3,
    judge_model="gpt-4o-mini",
)

result = asyncio.run(run_benchmark(config))
print_results_table(result)

# Access metrics programmatically
for model, metrics in result.metrics.items():
    print(f"{model}: p50={metrics.latency_p50_ms:.0f}ms")

CI/CD integration

Machine-readable output

Use --json flag to get structured output. Catch latency regressions, cost overruns, or quality drops in your pipeline.

{ "timestamp": "2026-03-27T03:00:00Z", "duration_seconds": 8.3, "results": { "gpt-4o": { "latency": { "p50_ms": 498.2, "p95_ms": 631.4 }, "tokens": { "avg_input": 14.0, "avg_output": 19.3 }, "cost_per_1k_requests_usd": 0.0581, "quality_score": 8.7, "n_success": 3, "n_errors": 0 } } }

Get started

Install in seconds

Core only

pip install llm-benchmarker

With OpenAI + Anthropic

pip install "llm-benchmarker[openai,anthropic]"

All providers

pip install "llm-benchmarker[all]"

Generate starter config

llm-bench init

Requirements

Python ≥ 3.10 · MIT license · source on GitHub