Metadata-Version: 2.4
Name: amagi-draftbench
Version: 0.1.0
Summary: Find the draft model (or n-gram lookup) that speeds up your local LLM: speculative-decoding sweeps for llama.cpp and vLLM with tok/s, acceptance rate, tokens per joule and peak VRAM, plus a pick per VRAM budget.
Author-email: "AmagiLabs (Amagisolutionsgroup, Inc.)" <dev@amagisolutionsgroup.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/amagilabs/amagi-draftbench
Keywords: llm,speculative-decoding,draft-model,llama.cpp,vllm,benchmark,gpu,localllama,cli
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Environment :: Console
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Dynamic: license-file

# amagi-draftbench

Find out which draft model, if any, makes your local LLM faster on your own GPU. amagi-draftbench starts llama-server (or vLLM, or any server you give it) once per target and draft pair, then measures decode tok/s, speedup, draft acceptance rate, tokens per joule and peak VRAM. It also tests n-gram drafting, which needs no draft model at all, and recommends a draft for each VRAM budget.

![amagi-draftbench showing RTX 3090 results and a per-VRAM recommendation](https://raw.githubusercontent.com/amagilabs/amagi-draftbench/main/assets/demo.gif)

## Install

```bash
pipx install amagi-draftbench
```

Not on PyPI yet. Until it is, install from a clone with `pipx install .` or `pip install .`

Python 3.9 or newer, no other dependencies. You also need a llama.cpp build with `llama-server` (or vLLM) and GGUF models.

## Usage

Write a sweep config, run it, then read the results:

```bash
amagi-draftbench init-config --target ~/models/Qwen2.5-32B-Instruct-Q4_K_M.gguf \
    --draft ~/models/Qwen2.5-0.5B-Instruct-Q8_0.gguf --draft ~/models/Qwen2.5-1.5B-Instruct-Q4_K_M.gguf \
    --ngram --hardware rtx3090 -o configs/q25-32b.json
amagi-draftbench sweep configs/q25-32b.json
amagi-draftbench report results/ --by-prompt
amagi-draftbench recommend results/
```

`sweep` runs the target alone first, then once with each draft. Every run gets a fresh server, a warm-up request and the same prompts at temperature 0. Results land in `results/<hardware>_<backend>_<name>.json`. If a sweep stops partway, run it again and it carries on from where it stopped.

To benchmark one server you started yourself (upstream issue #2):

```bash
llama-server -m target.gguf -md draft.gguf -ngl 99 -ngld 99 -np 1 --port 8081
amagi-draftbench bench --url http://127.0.0.1:8081
amagi-draftbench bench --url http://127.0.0.1:8081 --compare-url http://127.0.0.1:8080   # 8080 runs without a draft
```

`find-server` shows which `llama-server` it will use and which draft features that build has:

```
$ amagi-draftbench find-server ~/llama.cpp/build/bin/llama-server
~/llama.cpp/build/bin/llama-server
  --spec-type (n-gram drafting): yes
  n-gram types: ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache
  draft size flag: --spec-draft-n-max
  -np (slots): yes
```

## Results on an RTX 3090

One RTX 3090 (24 GB, power limit 300 W), llama.cpp build 11753 with flash attention, temperature 0, five prompts sent twice each. Replies are 512 tokens on the 7B and the Gemma MoE, and 256 tokens on the 27B and 32B (see the note on heat under [How it measures](#how-it-measures)). Qwen3.8 ran with thinking off. The raw files are in [`results/rtx3090/`](results/rtx3090/).

#### qwen25-32b (rtx3090, llamacpp)

| Target | Draft | tok/s | Speedup | Accepted | tok/J | Watts | VRAM GiB | Same output |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-32B Q4_K_M | none (baseline) | 37.0 | 1.00x | - | 0.14 | 267 | 19.6 | - |
| Qwen2.5-32B Q4_K_M | Qwen2.5-0.5B Q8_0 | 62.8 | 1.70x | 76% | 0.24 | 252 | 20.2 | 5/10 |
| Qwen2.5-32B Q4_K_M | Qwen2.5-1.5B Q4_K_M | 62.4 | 1.69x | 85% | 0.24 | 249 | 20.7 | 5/10 |
| Qwen2.5-32B Q4_K_M | Qwen2.5-1.5B Q8_0 | 60.9 | 1.65x | 87% | 0.23 | 255 | 21.3 | 5/10 |
| Qwen2.5-32B Q4_K_M | Qwen2.5-3B Q4_K_M | 54.6 | 1.48x | 84% | 0.22 | 242 | 21.6 | 5/10 |
| Qwen2.5-32B Q4_K_M | ngram-mod | 43.7 | 1.18x | 70% | 0.16 | 272 | 19.6 | 10/10 |

#### qwen25-7b (rtx3090, llamacpp)

| Target | Draft | tok/s | Speedup | Accepted | tok/J | Watts | VRAM GiB | Same output |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7B Q4_K_M | none (baseline) | 140.8 | 1.00x | - | 0.51 | 273 | 4.7 | - |
| Qwen2.5-7B Q4_K_M | Qwen2.5-0.5B Q8_0 | 170.2 | 1.21x | 80% | 0.59 | 282 | 5.3 | 10/10 |
| Qwen2.5-7B Q4_K_M | Qwen2.5-1.5B Q4_K_M | 153.2 | 1.09x | 90% | 0.57 | 265 | 5.8 | 10/10 |
| Qwen2.5-7B Q4_K_M | ngram-mod | 156.2 | 1.11x | 35% | 0.65 | 239 | 4.7 | 10/10 |

#### qwen38-27b (rtx3090, llamacpp)

| Target | Draft | tok/s | Speedup | Accepted | tok/J | Watts | VRAM GiB | Same output |
|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B Q4_K_M | none (baseline) | 42.1 | 1.00x | - | 0.15 | 263 | 15.6 | - |
| Qwen3.8-27B Q4_K_M | Qwen3.5-0.8B Q8_0 | 44.6 | 1.06x | 88% | 0.18 | 239 | 16.5 | 6/10 |
| Qwen3.8-27B Q4_K_M | Qwen3.5-2B Q4_K_M | 44.2 | 1.05x | 89% | 0.17 | 241 | 16.9 | 8/10 |
| Qwen3.8-27B Q4_K_M | ngram-mod | 49.6 | 1.18x | 63% | 0.18 | 266 | 15.6 | 10/10 |

#### gemma4-26b-a4b (rtx3090, llamacpp)

| Target | Draft | tok/s | Speedup | Accepted | tok/J | Watts | VRAM GiB | Same output |
|---|---|---|---|---|---|---|---|---|
| Gemma-4-26B-A4B Q4_K_M | none (baseline) | 122.3 | 1.00x | - | 0.42 | 282 | 16.6 | - |
| Gemma-4-26B-A4B Q4_K_M | Gemma-4-E2B Q4_K_M | 92.2 | 0.75x | 71% | 0.34 | 262 | 18.2 | 1/10 |
| Gemma-4-26B-A4B Q4_K_M | ngram-mod | 122.9 | 1.00x | 35% | 0.45 | 266 | 16.6 | 8/10 |

**A draft model gave a 50-80% speedup on this 3090 only for the dense 32B.**

- **Qwen2.5-32B Q4_K_M:** a Qwen2.5-0.5B Q8_0 draft took it from 37.0 to 62.8 tok/s (1.70x), and 1.5B Q4_K_M did 1.69x. The 3B draft got a similar share of tokens accepted but costs more to run, so it landed at 1.48x. Energy per token improved along with speed, from 0.14 to 0.24 tok/J.
- **Qwen2.5-7B Q4_K_M:** the same 0.5B draft gave 1.21x. The 7B already runs at 141 tok/s on its own, and the draft's own cost is a bigger share of each step.
- **Qwen3.8-27B Q4_K_M with Qwen3.5 drafts:** 1.05x to 1.06x, even though 88-89% of draft tokens were accepted. A separate single request showed the same thing: the server reported 85% acceptance and 3.5 tokens per draft step, and decode stayed at 42.4 tok/s against 42.2 without the draft. I have not found the cause.
- **Gemma-4-26B-A4B (MoE):** the Gemma-4-E2B draft made it slower, 122.3 down to 92.2 tok/s (0.75x). This MoE only reads about 4B parameters per token, so it is already fast, and running the draft costs more than it saves.
- **n-gram drafting** (`ngram-mod`, no draft model, no extra VRAM) gave 1.11x to 1.18x on the Qwen targets and 1.00x on Gemma. All of the gain came from the two edit prompts, where the reply copies most of the prompt: 1.5x to 2.0x there, and 1.0x on fresh code, JSON and lists.

The per-prompt view for the 32B shows which kinds of reply each draft speeds up:

#### qwen25-32b: tok/s per prompt

| Target | Draft | code | json | list | edit_code | edit_text |
|---|---|---|---|---|---|---|
| Qwen2.5-32B Q4_K_M | none (baseline) | 37 | 37 | 37 | 37 | 37 |
| Qwen2.5-32B Q4_K_M | Qwen2.5-0.5B Q8_0 | 66 (1.8x) | 71 (1.9x) | 45 (1.2x) | 71 (1.9x) | 74 (2.0x) |
| Qwen2.5-32B Q4_K_M | Qwen2.5-1.5B Q4_K_M | 63 (1.7x) | 65 (1.8x) | 53 (1.4x) | 64 (1.7x) | 70 (1.9x) |
| Qwen2.5-32B Q4_K_M | Qwen2.5-1.5B Q8_0 | 64 (1.7x) | 63 (1.7x) | 51 (1.4x) | 62 (1.7x) | 67 (1.8x) |
| Qwen2.5-32B Q4_K_M | Qwen2.5-3B Q4_K_M | 55 (1.5x) | 56 (1.5x) | 48 (1.3x) | 57 (1.6x) | 60 (1.6x) |
| Qwen2.5-32B Q4_K_M | ngram-mod | 37 (1.0x) | 37 (1.0x) | 37 (1.0x) | 58 (1.6x) | 73 (2.0x) |

`recommend` turns the same files into a pick per card size (24 GB shown):

```
$ amagi-draftbench recommend results/rtx3090 --budgets 24
24 GB card:
  Qwen2.5-32B Q4_K_M: draft Qwen2.5-0.5B Q8_0 -> 62.8 tok/s (1.70x vs 37.0 plain, 76% accepted), 20.2 GiB
  Qwen2.5-7B Q4_K_M: draft Qwen2.5-0.5B Q8_0 -> 170.2 tok/s (1.21x vs 140.8 plain, 80% accepted), 5.3 GiB
  Qwen3.8-27B Q4_K_M: draft ngram-mod -> 49.6 tok/s (1.18x vs 42.1 plain, 63% accepted), 15.6 GiB
      best draft model: Qwen3.5-0.8B Q8_0 -> 44.6 tok/s (1.06x vs 42.1 plain, 88% accepted), 16.5 GiB
  Gemma-4-26B-A4B Q4_K_M: skip drafts (122.3 tok/s, 16.6 GiB) -- no draft that fits is at least 5% faster; run it plain

n-gram drafting only pays off when the reply repeats text from the prompt (edits, refactors). Check 'report --by-prompt' before relying on it.
```

## FAQ: the six open draftbench issues

**#1, GGUF vs MLX.** amagi-draftbench has a `command` backend that runs any server from a shell line, so `mlx_lm.server --draft-model` works through it ([`examples/mlx_command.json`](examples/mlx_command.json)). I have no Apple Silicon results, so there is no GGUF vs MLX comparison here yet.

**#2, how do I run a single benchmark?** `amagi-draftbench bench --url ...` against a server you started. See [Usage](#usage).

**#3, TPS shows 0.0.** Thinking models (Qwen3, DeepSeek-R1, gpt-oss) stream their reasoning in `delta.reasoning_content`, not `delta.content`. draftbench only counted `content`, so a reply that spent its whole token budget thinking never produced a "first token" and scored 0. Both fields count now. To turn thinking off, add `"extra_body": {"chat_template_kwargs": {"enable_thinking": false}}` to a config, or pass `--no-think` to `bench`.

**#4, llama-server autodetection.** Searches PATH (the approach from PR #4 by alxarch), then the usual build folders, including Windows `build/bin/Release/llama-server.exe`. Set `llama_bin` in the config to skip the search.

**#5, 400 Bad Request.** draftbench threw the server's error message away, so all you saw was "400". amagi-draftbench prints the message the server sent. It also starts llama-server with one slot (`-np 1`) and a context sized to fit the longest prompt plus `max_tokens`. Recent builds default to several slots that split the context, and a request that does not fit gets a 400. I could not reproduce the 400 with draftbench's own flags on llama.cpp build 11753: all four slots got the full 4096 tokens (unified KV cache) and every request returned 200. Older builds may still behave as the issue describes.

**#6, vLLM.** Current vLLM dropped `--speculative-model` in favor of `--speculative-config`. The `vllm` backend uses the new flag for both draft models and n-gram drafting, and reads acceptance from vLLM's `/metrics` counters ([`examples/vllm_qwen3.json`](examples/vllm_qwen3.json)). I have no vLLM results yet; the commands above come from `sweep --dry-run`.

## How it measures

- **tok/s** is decode speed. On llama.cpp it is the server's own `timings.predicted_per_second`, summed over all requests as total tokens / total decode time. Elsewhere it is measured on the client from the stream.
- **Accepted** is draft tokens the target kept divided by draft tokens proposed (`draft_n_accepted / draft_n`), or vLLM's spec-decode counters.
- **tok/J** is generated tokens divided by GPU energy, a trapezoid integral of `nvidia-smi` power readings taken every 200 ms during each request. Time spent cooling down between requests does not count.
- **VRAM GiB** is peak `memory.used` during the run minus the idle reading before the server started.
- **Same output** counts responses whose text hash matches the run without a draft. At temperature 0 a draft should not change the text in theory. In practice the target sees a batch of tokens during verification instead of one at a time, and the floating-point math changes enough to flip a token now and then. Once one token differs, the rest of the reply goes its own way.
- **N-gram runs start a fresh server for every scored request.** llama-server keeps one n-gram table for the life of the process, shared by every request. In the first sweep each prompt ran twice on one server, and the second pass found its own first answer in the table: the 7B ran edit_text at 1442 tok/s on the second pass against 277 on the first. Those files are in [`results/rtx3090-ngram-warm-table/`](results/rtx3090-ngram-warm-table/). In a long chat or agent session the table does carry over, so repeated content gets that boost for real. A benchmark of one request should not count it, so the sweep doesn't.

Hot cards: `--temp-cmd` reads memory temperature (for example a GDDR6X junction reader), `--abort-mem`/`--abort-core` stop the sweep and discard the run in progress, and `--cool-at 92 --cool-to 82` makes it wait before a request until the card cools down. A dense 32B pushed this RTX 3090's memory to 108 C within one 512-token request, which is why the 27B and 32B configs use 256 tokens.

## Config

```json
{
  "name": "qwen25-32b",
  "hardware": "rtx3090",
  "backend": "llamacpp",
  "targets": [{"label": "Qwen2.5-32B Q4_K_M", "path": "/models/Qwen2.5-32B-Instruct-Q4_K_M.gguf"}],
  "drafts": [
    {"label": "Qwen2.5-0.5B Q8_0", "path": "/models/Qwen2.5-0.5B-Instruct-Q8_0.gguf"},
    {"label": "ngram-mod", "type": "ngram"}
  ],
  "settings": {"runs": 2, "max_tokens": 256, "prompts": "all", "extra_args": ["-fa", "on"]}
}
```

`prompts` is `default` (draftbench's code, JSON and list prompts), `edit` (two prompts that hand back given text with a small change, where n-gram drafting does best) or `all`. Other settings: `llama_bin`, `port`, `ctx_size`, `temperature`, `extra_body`, `n_max` per draft, `ngram_cold`. More in [`examples/`](examples/).

## Claude Code skill

[`.claude/skills/amagi-draftbench/SKILL.md`](.claude/skills/amagi-draftbench/SKILL.md) teaches Claude Code when to run a sweep, how to read the numbers and what not to claim from them.

## Limits

- Results come from one RTX 3090, one llama.cpp build (11753) and Q4_K_M targets. Other cards, quants and builds will differ.
- The prompts are short, and each reply is 256 or 512 tokens long. Long-context work is not covered.
- Temperature 0 only. Sampling at higher temperatures lowers acceptance.
- Power and VRAM come from `nvidia-smi`, so NVIDIA only.

## Star

If this saved you a guessing session, a star helps other people find it.

## More from AmagiLabs

[amagi-tokbench](https://github.com/amagilabs/amagi-tokbench) measures how fast a local LLM server runs: generation and prompt tok/s, time to first token and peak VRAM. [github.com/amagilabs](https://github.com/amagilabs) has the other small open source dev tools from the same lab.

## Credit

Based on draftbench by Alex Ziskind (MIT), github.com/alexziskind1/draftbench

The sweep loop, server launcher and config format started from draftbench. The reasoning-token fix, n-gram drafting, power and VRAM sampling, the per-prompt report, the recommendations and the tests are new. The `results/upstream/` files are draftbench's own published results, kept so `report` can read the old format.

## License

MIT, see [LICENSE](LICENSE). Copyright (c) 2026 Alex, and Copyright (c) 2026 Amagisolutionsgroup, Inc. (AmagiLabs).
