Metadata-Version: 2.4
Name: localbenchmark
Version: 1.1.0
Summary: localbenchmark: local LLM evaluation — coding benchmarks, long-context stress tests, and ranking plots via LM Studio.
Author-email: Erdogan Taskesen <erdogant@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://erdogant.github.io/localbenchmark
Project-URL: Download, https://github.com/erdogant/localbenchmark/archive/{version}.tar.gz
Keywords: Python,localbenchmark,LLM,benchmark,HumanEval,LM Studio,Machine Learning,AI
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: Unix
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: MacOS
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: matplotlib
Requires-Dist: requests
Requires-Dist: tqdm
Requires-Dist: openai
Requires-Dist: scikit-learn
Requires-Dist: pymupdf
Requires-Dist: json_repair
Requires-Dist: datasets
Dynamic: license-file

[![Python](https://img.shields.io/pypi/pyversions/localbenchmark)](https://img.shields.io/pypi/pyversions/localbenchmark)
[![Pypi](https://img.shields.io/pypi/v/localbenchmark)](https://pypi.org/project/localbenchmark/)
[![Docs](https://img.shields.io/badge/Sphinx-Docs-Green)](https://erdogant.github.io/localbenchmark/)
[![LOC](https://sloc.xyz/github/erdogant/localbenchmark/?category=code)](https://github.com/erdogant/localbenchmark/)
[![Downloads](https://static.pepy.tech/personalized-badge/localbenchmark?period=month&units=international_system&left_color=grey&right_color=brightgreen&left_text=PyPI%20downloads/month)](https://pepy.tech/project/localbenchmark)
[![Downloads](https://static.pepy.tech/personalized-badge/localbenchmark?period=total&units=international_system&left_color=grey&right_color=brightgreen&left_text=Downloads)](https://pepy.tech/project/localbenchmark)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/erdogant/localbenchmark/blob/master/LICENSE)
[![Forks](https://img.shields.io/github/forks/erdogant/localbenchmark.svg)](https://github.com/erdogant/localbenchmark/network)
[![Issues](https://img.shields.io/github/issues/erdogant/localbenchmark.svg)](https://github.com/erdogant/localbenchmark/issues)
[![Project Status](http://www.repostatus.org/badges/latest/active.svg)](http://www.repostatus.org/#active)
[![Medium](https://img.shields.io/badge/Medium-Blog-black)](https://erdogant.github.io/localbenchmark/pages/html/Documentation.html#medium-blog)
[![Donate](https://img.shields.io/badge/Support%20this%20project-grey.svg?logo=github%20sponsors)](https://erdogant.github.io/localbenchmark/pages/html/Documentation.html#)

<div>
<a href="https://erdogant.github.io/localbenchmark/">
  <img src="https://raw.githubusercontent.com/erdogant/localbenchmark/master/docs/source/figs/logo.png"
       width="100"
       align="left" />
</a>

**localbenchmark** is a Python package for **local LLM evaluation**: coding benchmarks, long-context stress tests, speed probes, and ranking plots — driven through [LM Studio](https://lmstudio.ai/) or any other OpenAI-compatible server (llama.cpp, vLLM, Ollama, ...).
It answers a practical question: *can this model still write correct code when the context window is large, and is it fast enough that you would actually use it?* ⭐️Star it if you like it⭐️
</div>

<br>

---

### Key Features

| Feature | Description |
|--------|-------------|
| **Coding benchmark** | HumanEval-style tasks via LM Studio or any OpenAI-compatible server; temperature 0 for comparable runs |
| **Long context** | Test fixed windows (e.g. 50k–180k); optional context *fill* so the KV cache is exercised |
| **Speed probe** | Prefill / decode tok/s at controlled tokens-in-cache levels |
| **Resume-safe CSVs** | Durable results; skip finished configs; retry LLM errors |
| **Slow-model abort** | Stop a config after a streak of low decode speed; ranking accounts for incomplete runs |
| **Rank-sum plots** | Pass rate, decode speed, TTFT, GPU peak, context — weighted ranking (model × context and per model) |
| **Any backend** | LM Studio's native API or any OpenAI-compatible server, detected automatically |
| **Model discovery** | `--list` shows every available model; running without `--models` tests all of them |

---


### Installation

##### Install from PyPI
```bash
pip install localbenchmark
```

##### Install from GitHub
```bash
pip install git+https://github.com/erdogant/localbenchmark
```

##### Install from source
```bash
pip install git+https://github.com/erdogant/localbenchmark
```

---

### Quick start — Dry run

Make a dry run. This is ideal for a fast check of model GPU requirements and their theoretical (expected) coding speed.

```bash
# Run for all models with context window 120K and quantization of Q4_K_M

localbenchmark --dry-run --contexts 120000 --quants Q4_K_M
localbenchmark --dry-run --contexts 120000 --quants Q4_K_M --models unsloth/qwen3-coder-30b-a3b-instruct openai/gpt-oss-20b

```

Output:

```bash
rank  model                         quant   context  total GB  headroom GB  offload %  bandwidth  decode ceiling tok/s
----------------------------------------------------------------------------------------------------------------------
1     gpt-oss-20b                   Q4_K_M  120k     13.1      4.0          0%         GPU        74
2     qwen3-coder-30b-a3b-instruct  Q4_K_M  120k     18.5      -1.4         8%         GPU+RAM    21

```

---

### Quick start — coding benchmark

Start your server (for LM Studio: enable the local server), then:

```bash
# Full default suite: every available model × every context window
localbenchmark

# Full default with base URL
localbenchmark --base-url http://localhost:1234

# Subset of models/contexts
localbenchmark --models unsloth/qwen3-coder-30b-a3b-instruct openai/gpt-oss-20b --contexts 50000 80000 100000 --base-url http://localhost:1234

```

---

### Backends and model discovery

localbenchmark detects which API the server speaks. LM Studio's native API is used when present; otherwise any server that exposes the OpenAI-compatible `/v1/models` and `/v1/chat/completions` endpoints works (llama.cpp, vLLM, Ollama, ...).

```bash
# List every available model and exit
localbenchmark --list
localbenchmark --list --base-url http://localhost:8080

# Other servers
localbenchmark --base-url http://localhost:8080  --models my-model
localbenchmark --base-url http://localhost:11434 --backend openai --models qwen3-coder:30b
localbenchmark --base-url https://my-server/v1   --api-key $MY_KEY
```

Without `--models`, every model the server (or, if it is not running, the local LM Studio model folders) reports is tested. Embedding models are skipped.

| | LM Studio | Other OpenAI-compatible servers |
|---|---|---|
| Model loading / unloading | by localbenchmark | not possible: the server decides |
| Context window | every `--contexts` value is loaded and tested | fixed by the server; each model is tested once at that size |
| Context detection | from the load response | `max_model_len` (vLLM) or `/props` (llama.cpp); if unknown, the run is labelled with the largest `--contexts` value, so start the server with that size |
| `--judge` model | loaded for you | must be served by the same server |

Run the other benchmarks:

```bash
# Different types of benchmarks

localbenchmark --benchmark humaneval --models "openai/gpt-oss-20b"
localbenchmark --benchmark mbpp      --models "openai/gpt-oss-20b"
localbenchmark --benchmark swebench  --models "openai/gpt-oss-20b" --contexts 80000 150000

```

<a href="https://erdogant.github.io/localbenchmark/">
  <img src="https://raw.githubusercontent.com/erdogant/localbenchmark/master/docs/source/figs/09_rank_sum.png" width="75%" alt="Localbenchmark model ranking comparison">
</a>

Results are written to CSV. Configurations already present in the **summary** CSV are skipped on the next run. Configurations that abort for low tok/s stay in the summary so they are not reloaded endlessly.

**Useful flags**

| Flag | Meaning |
|------|---------|
| `--models` | One or more model keys (default: all available models) |
| `--list` | List all available models and exit |
| `--base-url` | Server URL (default `http://localhost:1234`; a trailing `/v1` is accepted) |
| `--backend` | `auto` (default), `lmstudio` or `openai`: force which API is used |
| `--api-key` | API key for servers that need one (default: `$OPENAI_API_KEY`) |
| `--contexts` | Context sizes in tokens (default includes 50k, 80k, 100k, 120k, 150k, 180k). Only LM Studio can change the window; other servers are tested once at their own size |
| `--limit` | Run only the first N HumanEval problems |
| `--results` | Results CSV path (default depends on `--benchmark`) |
| `--benchmark` | `humaneval`, `mbpp`, `swebench` or `suite` |
| `--suite` | Suite file (`.jsonl`) for `--benchmark suite` |
| `--docs` | Document folder for suite mode (default `company_docs`) |
| `--docs-limit` | Use only the first N documents, in name order |
| `--docs-ids` | Use exactly these document file names |

---




### Custom (Company) QA

Suite mode asks questions about a set of company documents. Every selected document goes into the prompt, and we score each answer. The suite includes **unanswerable** questions: the correct answer is to say the documents do not contain the information. An invented answer scores zero, which is the most important check for company use.

#### Commands

Run the suite with the server running:

```bash
Example of lines:
{"id":"ctry-01","domain":"countries","type":"short","prompt":"What is the capital of Australia?","accept":["Canberra"],"distractors":["Sydney","Melbourne"]}
{"id":"ctry-03","domain":"countries","type":"numeric","prompt":"Roughly how large is Iceland, in square kilometres?","gold":103000,"rel_tol":0.05}
{"id":"co-04","domain":"companies","type":"short","prompt":"Which company acquired LinkedIn in 2016?","accept":["Microsoft"],"distractors":["Google","Oracle"]}
{"id":"co-01","domain":"companies","type":"numeric","prompt":"In which year was Microsoft founded?","gold":1975}

localbenchmark --benchmark suite --suite general_knowledge.jsonl --models "openai/gpt-oss-20b" --contexts 120000
```

#### Run with longer/ more detailled answers

- `company_docs/`: the document folder. Any `.txt` or `.md` file can be used.
- `suites/<name>.jsonl`: one question per line. Each question names its source documents in `context`.

The demo suite (`northwind_demo`) is fictional: 12 documents for a company called Northwind Analytics and 40 questions, 30 answerable and 10 unanswerable.

```bash
Example of jsonl:

{"id": "nw-01", "domain": "hr", "type": "short", "prompt": "How many days of annual leave do full-time employees get per calendar year?", "accept": ["25"], "context": ["hr_leave_policy.md"]}

```

#### Commands

Run the suite with the server running:

```bash
localbenchmark --benchmark suite --suite northwind_demo.jsonl --docs path/to/company_docs \
    --models "openai/gpt-oss-20b" --contexts 120000
```

Use fewer documents:

```bash

# first 40 files, in name order
localbenchmark --benchmark suite --suite northwind_demo.jsonl --docs path/to/company_docs \
    --models "openai/gpt-oss-20b" --docs-limit 40

localbenchmark --benchmark suite --suite northwind_demo.jsonl --docs path/to/company_docs \
    --models "openai/gpt-oss-20b" --docs-ids hr_leave_policy.md expense_policy.md
```

#### Output

- `suite_results_<suite>.csv`: one row per question, with the score (0 to 1), pass/fail, domain, item type, whether the cited documents were included, and an error label (`WRONG_ANSWER`, `INVENTED_ANSWER`, `MISSING_DOC`, `TRUNCATED`, `EMPTY_RESPONSE`, `LLM_ERROR`).
- `suite_results_<suite>_by_domain.csv`: pass rate and mean score per domain. The console also shows answerable and unanswerable results separately, and the number of invented answers.


### Plots and ranking

```bash
localbenchmarkplot --rank-agg sum      # default
localbenchmarkplot --rank-agg mean
localbenchmarkplot --rank-agg median

# Only large windows (context_actual >= 100000)
localbenchmarkplot --min-context 100000
```

**Ranking metrics** (lower combined score is better):

1. Pass rate (early aborts count unfinished work against the model)
2. Decode speed (percentage of best — large speed gaps stay large)
3. Time to first token
4. Peak GPU memory
5. Context size (percentage of best)

Pass rate and decode speed are **weighted**. Figures include pass rate vs context, tok/s, TTFT, GPU memory, quality–speed trade-off, and rank-sum panels (model × context and per model).

---

### What is measured

| Signal | Why it matters |
|--------|----------------|
| **Pass rate** | Generated code must pass the unit tests |
| **Decode speed (tok/s)** | Inference has to be usable, not only correct |
| **TTFT** | Latency before the first useful token |
| **Peak GPU memory** | Whether the setup fits your hardware |
| **Context** | Behaviour as the window grows |

Default context grid:

```text
50k → 80k → 100k → 120k → 150k → 180k
```

Any context sizes present in the results file are included in the rank plots (not limited to a hard-coded list).

---

### Resources and Links

- **Documentation:** [Website](https://erdogant.github.io/localbenchmark)
- **Issues:** [GitHub Issues](https://github.com/erdogant/localbenchmark/issues)
- **Source:** [github.com/erdogant/localbenchmark](https://github.com/erdogant/localbenchmark)

---

### Maintainer

* Erdogan Taskesen, GitHub: [erdogant](https://github.com/erdogant)
* Contributions are welcome.
* Yes! This library is entirely **free** but it runs on coffee! Feel free to support with a <a href="https://erdogant.github.io/donate/?currency=USD&amount=5">Coffee</a>.

[![Buy me a coffee](https://img.buymeacoffee.com/button-api/?text=Buy+me+a+coffee&emoji=&slug=erdogant&button_colour=FFDD00&font_colour=000000&font_family=Cookie&outline_colour=000000&coffee_colour=ffffff)](https://www.buymeacoffee.com/erdogant)
