Metadata-Version: 2.5
Name: mispick
Version: 0.2.0
Summary: Find out which of your MCP tools the model mixes up - measured, not guessed.
Project-URL: Homepage, https://github.com/msc35/mispick
Project-URL: Repository, https://github.com/msc35/mispick
License: MIT
License-File: LICENSE
Keywords: confusion-matrix,evaluation,llm,mcp,model-context-protocol,tools
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: jsonschema>=4.20.0
Requires-Dist: mcp>=2.2.0
Requires-Dist: openai>=1.60.0
Requires-Dist: pydantic>=2.12.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.9.4
Requires-Dist: typer>=0.16.0
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40.0; extra == 'anthropic'
Description-Content-Type: text/markdown

# mispick

**Find out which of your MCP tools an AI model mixes up.**

## TL;DR

- **The problem.** An AI assistant picks which tool to use by reading each tool's short
  description. If two descriptions sound alike, it quietly picks the wrong one, and nothing
  tells you.
- **What mispick does.** It writes realistic test requests for every tool, asks a real model to
  pick a tool for each one, and counts what it picked. You get a table of which tools get
  mistaken for which (a *confusion matrix*), plus a 0–100 score.
- **It's safe.** It only reads the tool list. It never runs a tool, so you can point it at a
  server that deletes things.
- **It can suggest fixes**, and it re-tests each suggested rewrite. It keeps only the ones that
  measurably helped.
- **Free by default.** It runs on a local model through Ollama. Cloud models are optional.
- **Status:** v0.2.0 is [on PyPI](https://pypi.org/project/mispick/). A
  [public benchmark](#benchmark) of real MCP servers is running now. A
  [research study](#research-study) built on it is pre-registered, with first results in.

<p align="center">
  <img src="docs/media/demo.svg" alt="mispick's terminal report: a confusion matrix showing search_docs and search_issues being mistaken for each other, per-tool precision and recall with confidence intervals, and a 0-100 score." width="900">
</p>

```bash
uvx mispick run --cmd "python -m my_server"
```

**How to read the report.** Each row of the table is a tool. Each column is what the model
actually picked when that tool was the right answer. A perfect server has every count on the
diagonal. Anything off the diagonal is a mix-up worth fixing. Every percentage comes with a range
(a 95% confidence interval), because with few test requests a single number can mislead.

---

## What the benchmark found

We ran mispick on **14 real MCP servers** from the official registry: **3,636 test requests**
on Gemini 3.5 Flash-Lite. Full results: [docs/study.md](docs/study.md#results).

- **Most requests go to the right tool, but not all:** 85% did.
- **Mistakes are concentrated:** only 107 of 6,024 tool pairs were ever mixed up, so a server
  usually has a few specific problem pairs rather than general confusion.
- **Results are repeatable:** 97% of requests got the same answer in all three runs.
- **Overlapping names and similar descriptions go with more mix-ups,** but with 13 servers that
  evidence is fragile.
- **The clearest tip for server authors: say *when* to use each tool.** Tools whose description
  does were less often mistaken for others.
- **One server could not be used with Gemini at all,** because of a broken schema in its tool
  list.

<p align="center">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="results/figures/similarity-dark.png">
  <img src="results/figures/similarity-light.png" alt="Bar chart: the most similar fifth of tool pairs were confused 4.8 times as often as the least similar fifth." width="760">
</picture>
</p>

<p align="center">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="results/figures/what-matters-dark.png">
  <img src="results/figures/what-matters-light.png" alt="Nine factors and how each changes the odds of a wrong pick. The right tool saying when to use it lowers the odds by 28 percent, the only factor that holds up in the strictest check." width="760">
</picture>
</p>

**Three real servers, as confusion matrices.** Each row is the tool a request was meant for,
and each column is what the AI picked. Numbers on the diagonal are correct.

<p align="center">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="results/figures/matrix-ai-wavespeed-mcp-dark.png">
  <img src="results/figures/matrix-ai-wavespeed-mcp-light.png" alt="Confusion matrix for ai.wavespeed/mcp: 60 percent correct. Every run_model request went to list_models." width="640">
</picture>
</p>

- **ai.wavespeed/mcp, 60% correct.** Every request to *run* a model went to `list_models`, and
  so did most requests to get a price.

<p align="center">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="results/figures/matrix-io-github-modelcontextprotocol-server-memory-dark.png">
  <img src="results/figures/matrix-io-github-modelcontextprotocol-server-memory-light.png" alt="Confusion matrix for the Memory reference server: 67 percent correct, with many edits and deletes going to search_nodes first." width="640">
</picture>
</p>

- **Memory (MCP reference server), 67% correct.** Edit and delete requests often went to
  `search_nodes` first. The model looked up the entry before changing it, which is sensible,
  but mispick scores the first call, so it counts as a miss.

<p align="center">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="results/figures/matrix-com-1102tools-acquisition-gov-mcp-dark.png">
  <img src="results/figures/matrix-com-1102tools-acquisition-gov-mcp-light.png" alt="Confusion matrix for Acquisition.gov: every request went to the right tool." width="520">
</picture>
</p>

- **Acquisition.gov, 100% correct.** Five clearly separated tools and no mix-ups. This is what
  a good tool surface looks like.

Every number here comes from [`results/`](results/): `headline.json`, `analysis.json`, and the
frozen run in `analysis/data/benchmark-v1/`. Results are for one model on one date; see the
limitations below.

## Why this rather than a static linter

`mcp-lint`, `mcpolish`, `oh-my-mcp`, `mcp-conform` and `mcp-cve-lint` check tool definitions
against rules, and they are good at it — run one. But a description can pass every rule and still
be indistinguishable from the tool next to it. Rule-based linters predict problems; this measures
them, by putting your tool list in front of a real model and counting what it picks. Two other
projects do measure selection — [`whichtool`](https://github.com/mattagame/whichtool) (local
models, but no rewrites, one server per run) and [`toolfit`](https://pypi.org/project/toolfit/)
(rewrites, but cloud-only, one server per run). Neither looks across servers. The full comparison,
with what each one actually does, is in [docs/research.md](docs/research.md).

## How it works

1. Connect to your server and read `tools/list` (never `tools/call`).
2. For each tool, generate N realistic requests — plus **hard negatives** written to sit as close
   as possible to its nearest neighbour, and some requests no tool should answer.
3. Ask the model to choose a tool for each request, K times, showing it the real tool definitions
   a client would send.
4. Score it: accuracy, per-tool precision and recall, a confusion matrix, argument validity,
   stability across the K runs — every rate with a Wilson 95% interval.
5. Propose rewrites for the worst pairs, re-run the same requests against them, and keep only the
   ones that measurably won.

## Install

```bash
uvx mispick --help          # no install
uv tool install mispick     # or keep it
pip install mispick
```

The default backend is a local model through [Ollama](https://ollama.com), so it costs nothing and
needs no account:

```bash
ollama pull qwen3.5:4b
```

Cloud models are optional, with the matching key in the environment:

```bash
mispick run --snapshot tools.json --model openai/gpt-4.1-mini            # OPENAI_API_KEY
mispick run --snapshot tools.json --model gemini/gemini-3.5-flash-lite   # GEMINI_API_KEY
mispick run --snapshot tools.json --model anthropic/claude-haiku-4-5-20251001

# anything else that speaks the OpenAI dialect - vLLM, OpenRouter, LM Studio
MISPICK_BASE_URL=http://localhost:8000/v1 mispick run --snapshot tools.json \
  --model compatible/my-model
```

**On cost.** Every selection call re-sends the whole tool list, so a cloud run is priced by
tool-list size × queries × runs, not by the number of servers. Measuring one real 64-tool server
at N=8 K=3 is about 30M input tokens on its own. `--dry-run` is not implemented yet; until it is,
read the token estimate mispick prints and multiply.

## Use

```bash
# measure a local server, a remote one, or a captured snapshot
mispick run --cmd "python -m my_server"
mispick run --url https://example.com/mcp
mispick run --snapshot tools.json

# capture a snapshot so later runs are offline and reproducible
mispick snapshot --cmd "python -m my_server" -o tools.json

# every server in your config at once, and what they do to each other
mispick run --config ~/Library/Application\ Support/Claude/claude_desktop_config.json

# rewrites for the worst pairs, each one re-tested
mispick fix --cmd "python -m my_server" --patch fix.diff

# reports and CI
mispick run --snapshot tools.json --format html --out report.html --badge badge.svg
mispick run --snapshot tools.json --fail-under 80          # exit 1 below the threshold
mispick compare base.json head.json --max-drop 5           # exit 1 on a regression
```

Generated requests are cached in `.mispick/queries.yaml`. **Read that file and edit it** — it is
your test set, and it only regenerates for a tool whose name, description or schema changed.
Commit it.

### Proven fixes

`mispick fix` rewrites both descriptions in a confused pair, then re-runs the *same* requests
against the rewrite and reports before/after with an exact McNemar test — the trials are paired,
so an unpaired test would be the wrong one. Rewrites that fail are shown too.

That re-test is the whole point. Hasan et al. rewrote tool descriptions with an LLM across 231
tasks and found success rose by a median 5.85 points — but **regressed in 16.67% of cases**
([arXiv 2602.14878](https://arxiv.org/abs/2602.14878)). An unmeasured rewrite is a one-in-six
chance of making things worse. mispick also caps rewrite length, so a rewrite cannot win by
crowding out its neighbours, and it never edits your source — it prints a suggestion.

**Expect rejections, especially on a small local model.** In our own runs against
`ollama/qwen3.5:4b`, two of three proposed rewrites measurably made selection *worse* — one of them
took a tool from 78% to 58% accuracy, breaking 14 trials and fixing none, and it replicated when
re-run with three times the data. Its stated reasoning read perfectly sensibly. The third improved
by 10 points but over only two changed trials, which is not enough to call, so mispick reported it
as inconclusive rather than banking it. A 4B model is good enough to *find* confusion and often not
good enough to *write its way out of it*; fix mode is more productive with a larger model. Either
way the verdicts are the product — a run where nothing is accepted has told you something true.

### Across servers

Point `--config` at a whole `claude_desktop_config.json` and mispick loads every server at once,
because that is how a client sees them. It reports which bare tool names collide across servers
and — the part that matters — how often a request for one server's tool is answered by another
server's. On the two-server fixture, 17% of trials cross servers and `workspace:search_docs` falls
to 0% recall: every request for it is answered by `wiki:search_docs`. The MCP spec names this
problem and warns that `serverInfo.name` is not unique across servers, so mispick keys on your
config label instead.

### In CI

```yaml
- uses: actions/checkout@v4
- uses: msc35/mispick-action@v1
  with:
    snapshot: tools.json
    model: anthropic/claude-haiku-4-5-20251001
    fail-under: '80'
    compare-base: 'true'
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

It posts the report as a sticky PR comment, and with `compare-base` the comment reads
`84 → 91 (▲7)` and names the confusions that appeared or disappeared. With no model key it skips
with an explanation instead of failing — a check that goes red over a missing secret is a check
people learn to ignore. See [examples/github-action](examples/github-action/). The Action also
lives on its own in [msc35/mispick-action](https://github.com/msc35/mispick-action), for the
GitHub Marketplace.

## Benchmark

`benchmark/` measures real servers from the official MCP registry that start without an API key,
and builds a static leaderboard from the results.

- **Now running.** A daily GitHub Actions job measures 15 servers on Gemini 3.5 Flash-Lite's free
  tier. It needs about 11 days because of the free tier's daily limit, and it resumes where it
  stopped each day.
- **Reproducible.** The captured tool lists are committed in `benchmark/snapshots/`.
- **Safe.** Only `initialize` and `tools/list` are ever sent.
- **Honest about coverage.** The page also lists the servers it could *not* measure, so coverage
  is not overstated.

See [benchmark/README.md](benchmark/README.md) and the run plan in
[docs/benchmark-plan.md](docs/benchmark-plan.md).

## Research study

The benchmark data also feeds a small study. Its analysis plan was written and committed
**before** looking at any results, so the questions can't be bent to fit the answers. The study
asks three questions:

1. **What makes two tools easy to confuse?** For example, similar descriptions, overlapping names,
   or missing "use this when…" guidance.
2. **Does a cheap model's confusion predict a stronger model's?**
3. **Do generated test requests give the same answers as hand-written ones?**

**First results are in.** Similar tool names go with more mix-ups, but with 13 servers the
evidence is fragile. The clearest signal: a tool whose description says *when* to use it gets
its requests taken by other tools less often. Plan: [docs/analysis-plan.md](docs/analysis-plan.md).
Results: [docs/study.md](docs/study.md#results).

## Limitations

Read these before quoting a number.

- **Results depend on the model.** A score is one model's reading of one tool surface on one day,
  not a property of your server. Every report says so, and states the model, N, K and the date.
- **Small local models are noisy.** That is why every rate carries a Wilson 95% interval and why
  `--runs` exists. With K=3 the intervals are wide; read them, not the point estimate.
- **The requests are generated, so they are not your traffic.** They are a stand-in for it. Edit
  `.mispick/queries.yaml` to make them yours.
- **It measures one selection turn.** Not multi-step agent behaviour, not tool output quality, not
  recovery after a bad call. Arguments are checked against `inputSchema` and no further.
- **A low score can be correct.** Two tools may genuinely overlap. mispick tells you the model
  cannot separate them; whether they *should* be separable is your call.
- **`mispick fix` often accepts nothing**, and that is working as intended. See the note above.
- **Token counts are estimates**, and lower bounds — a real tokenizer and the provider's framing
  both add to them.
- **Reasoning models** spend heavily on hidden thinking. mispick turns thinking off by default for
  Ollama (measured: 89s → 2.2s per call on `qwen3.5:4b`, same answers); `--think` restores it.

## Development

```bash
uv sync --all-extras                  # --all-extras so mypy also checks the Anthropic backend
uv run pytest                         # offline; no model or network needed
uv run ruff check . && uv run mypy src
uv run mispick run --snapshot tests/fixtures/confusing_tools.json --model mock
```

`--model mock` is a deterministic offline backend. The test suite uses it, so `pytest` passes with
no Ollama, no keys and no network.

## License

MIT. See [LICENSE](LICENSE).
