Metadata-Version: 2.5
Name: mcp-toolproof
Version: 0.2.0
Summary: Automated tests for MCP servers: schema checks, YAML test cases, JUnit output and a pytest plugin.
Project-URL: Homepage, https://github.com/NilkanthSuthar/toolproof
Project-URL: Issues, https://github.com/NilkanthSuthar/toolproof/issues
Author: Nilkanth Suthar
License-Expression: MIT
License-File: LICENSE
Keywords: ci,mcp,model-context-protocol,pytest,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: anyio>=4
Requires-Dist: hypothesis-jsonschema>=0.23
Requires-Dist: hypothesis>=6.100
Requires-Dist: jsonpath-ng>=1.6
Requires-Dist: jsonschema>=4.21
Requires-Dist: mcp<3,>=2.3
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6
Requires-Dist: rich>=13
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: types-jsonschema; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Description-Content-Type: text/markdown

# toolproof

[![CI](https://github.com/NilkanthSuthar/toolproof/actions/workflows/ci.yml/badge.svg)](https://github.com/NilkanthSuthar/toolproof/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/mcp-toolproof)](https://pypi.org/project/mcp-toolproof/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

**pytest for MCP servers.** Write repeatable tests for any [Model Context Protocol](https://modelcontextprotocol.io) server's tools and run them in CI.

- **Static checks** on every tool definition: missing descriptions, invalid JSON Schemas, duplicate names, required fields that don't exist
- **YAML test cases**: call a tool and assert on errors, text, regex, JSONPath values, latency and the declared `outputSchema`
- **Crash detection**: if the server dies mid-test, toolproof reports it with the server's stderr, restarts the server and keeps going
- **Fuzzing**: generates valid and invalid inputs from each tool's schema with Hypothesis, finds crashes, hangs and leaked tracebacks, and shrinks each one to the smallest failing input
- **Benchmarks**: p50/p95/p99 latency, throughput and error rate under concurrency, with thresholds that fail the build
- **CI-friendly output**: exit code 0/1, JUnit XML and JSON reports
- **pytest plugin**: an `mcp_server` fixture for tests written in Python
- Works with any server over **stdio** or **streamable HTTP**, in any language

## Install

```bash
pip install mcp-toolproof
```

Python 3.11+. The package is called `mcp-toolproof` on PyPI. The command and the import are both `toolproof`.

## 30-second quickstart

Look at a server:

```bash
toolproof inspect -- python my_server.py
toolproof inspect --url http://localhost:8000/mcp
```

Write `toolproof.yaml`:

```yaml
server:
  command: ["python", "my_server.py"]

tests:
  - name: toronto weather
    tool: get_weather
    args: { city: Toronto }
    expect:
      is_error: false
      contains: "Toronto"
      jsonpath: { "$.temp_c": { type: number } }
      max_latency_ms: 2000

  - name: rejects empty city
    tool: get_weather
    args: { city: "" }
    expect: { is_error: true }
```

Run it:

```bash
toolproof run toolproof.yaml --junit report.xml
```

Fuzz and benchmark it, no test cases needed:

```bash
toolproof fuzz -- python my_server.py
toolproof bench --tool get_weather --args '{"city": "Toronto"}' -- python my_server.py
```

## What a failing run looks like

`examples/buggy_server.py` has five planted bugs. Here is `toolproof run examples/buggy.yaml`:

```
toolproof - buggy-weather 0.0.1

Static checks
┌──────┬───────────────┬─────────────────┬──────────────────────────────────────────────────────────────────┐
│      │ Tool          │ Check           │ Problem                                                          │
├──────┼───────────────┼─────────────────┼──────────────────────────────────────────────────────────────────┤
│ FAIL │ search_cities │ description     │ tool has no description                                          │
│ FAIL │ search_cities │ input-schema    │ input schema is not valid JSON Schema: 'strng' is not valid      │
│      │               │                 │ under any of the given schemas                                   │
│ FAIL │ lookup        │ required-fields │ required fields not listed in properties: city_id                │
└──────┴───────────────┴─────────────────┴──────────────────────────────────────────────────────────────────┘

Tests
┌──────┬────────────────────────────────────┬─────────────────┬────────┬────────────────────────────────────┐
│      │ Test                               │ Tool            │   Time │ Details                            │
├──────┼────────────────────────────────────┼─────────────────┼────────┼────────────────────────────────────┤
│ PASS │ weather works for a normal city    │ get_weather     │    5ms │                                    │
│ FAIL │ empty city should be an error, not │ get_weather     │   23ms │ no result from server: connection  │
│      │ a crash                            │                 │        │ lost, server probably crashed      │
│      │                                    │                 │        │ (Connection closed)                │
│ PASS │ server still works after the crash │ get_weather     │    3ms │                                    │
│ FAIL │ report comes back in time          │ slow_report     │ 1001ms │ no result from server: timed out   │
│      │                                    │                 │        │ after 1.0s                         │
│ FAIL │ temperature matches its output     │ get_temperature │    2ms │ output schema: $.temp_c: '12.5' is │
│      │ schema                             │                 │        │ not of type 'number'               │
└──────┴────────────────────────────────────┴─────────────────┴────────┴────────────────────────────────────┘

FAILED  tests: 2 passed, 3 failed  |  checks: 3 errors, 0 warnings  |  7.21s
```

The same run as JUnit XML (`--junit report.xml`), which GitHub, GitLab and Jenkins show as a normal test report:

```xml
<testsuite name="tests" tests="5" failures="3" errors="0" time="1.034">
  <testcase classname="toolproof.get_weather" name="weather works for a normal city" time="0.005" />
  <testcase classname="toolproof.get_weather" name="empty city should be an error, not a crash" time="0.023">
    <failure message="no result from server: connection lost, server probably crashed (Connection closed)">...</failure>
  </testcase>
  ...
</testsuite>
```

## YAML reference

```yaml
server:
  command: ["python", "server.py"]   # start the server over stdio...
  # url: http://localhost:8000/mcp   # ...or connect over streamable HTTP (exactly one)
  env: { API_KEY: test }             # extra environment variables (stdio only)
  cwd: .                             # working directory, relative to this file (default: this file's folder)
  startup_timeout_s: 30

timeout_ms: 10000        # default per-test timeout
retries: 0               # retry failing tests this many times

checks:
  enabled: true
  max_description_length: 1024
  strict: false          # treat warnings as failures

tests:
  - name: a readable name
    tool: tool_name
    args: { any: json }
    timeout_ms: 2000     # optional override
    retries: 1           # optional override
    expect:
      is_error: false
      equals: { result: 42 }         # whole result: structured content, JSON text, or plain text
      contains: "Toronto"            # or a list: ["Toronto", "cloudy"]
      regex: "\\d+ C"
      jsonpath:
        "$.temp_c": { type: number, min: -50, max: 50 }
        "$.city": { equals: Toronto }
        "$.wind": { exists: false }
      max_latency_ms: 2000
      output_schema: true            # validate against the tool's outputSchema (default on)

bench:                   # optional; `toolproof run` includes it when present
  calls: 100             # per target
  concurrency: 10
  warmup: 3
  timeout_ms: 10000
  thresholds:            # all optional
    p50_ms: 50
    p95_ms: 200
    p99_ms: 500
    max_error_rate: 0.01
    min_throughput: 100  # calls per second
  targets:               # default: every test case that expects success
    - tool: get_weather
      args: { city: Toronto }
      thresholds: { p95_ms: 100 }   # per-target override

fuzz:                    # optional; `toolproof run` includes it when present
  max_examples: 50       # per tool, for valid and for invalid inputs
  timeout_ms: 2000
  max_time_s: 60         # per tool
  seed: 1                # fix it for repeatable CI runs
  tools: []              # only these (default: all)
  skip: [send_email]
  include_destructive: false
  valid_must_succeed: false
  invalid_must_fail: true
```

Notes:

- `jsonpath` looks at the tool's structured content if it returned any, otherwise at its text parsed as JSON.
- `type` is a JSON Schema type: `string`, `number`, `integer`, `boolean`, `array`, `object`, `null`.
- If a tool declares an `outputSchema`, every successful call is validated against it automatically.
- Paths in `command` and `cwd` are relative to the YAML file, so `toolproof run path/to/toolproof.yaml` works from anywhere.

### Static checks

| Check | Severity | What it catches |
|---|---|---|
| `name` | error | tool with an empty name |
| `duplicate-name` | error | two tools with the same name |
| `description` | error | tool with no description |
| `description-length` | warning | description longer than `max_description_length` |
| `input-schema` | error | input schema that isn't valid JSON Schema, or whose root isn't `type: object` |
| `required-fields` | error | `required` lists a field that isn't in `properties` |
| `required-fields` | warning | tool has properties but none are marked required |
| `output-schema` | error | `outputSchema` that isn't valid JSON Schema |

Errors fail the run. Warnings only fail it with `--strict`.

## Fuzzing

`toolproof fuzz` calls every tool with inputs generated from its input schema, using [Hypothesis](https://hypothesis.readthedocs.io) and [hypothesis-jsonschema](https://github.com/python-jsonschema/hypothesis-jsonschema):

- **valid inputs** match the schema, mixed with edge cases that still match it: empty, whitespace, huge (100k characters), emoji, right-to-left text, null bytes, SQL and path-traversal strings
- **invalid inputs** take a valid input and break it: drop a required field, set a field to `null`, or give it the wrong type

Each call is judged like this:

| Finding | Meaning |
|---|---|
| `crash` | the server process died or the connection dropped |
| `hang` | no answer within `timeout_ms` |
| `internal-error` | a JSON-RPC error other than "invalid params" |
| `unhandled-exception` | an error result that leaks a traceback or the SDK's generic crash message |
| `output-schema` | a successful result that doesn't match the tool's `outputSchema` |
| `rejected-valid` | a schema-valid input rejected as invalid params |
| `accepted-invalid` | an invalid input that returned success (turn off with `invalid_must_fail: false`) |
| `unexpected-error` | any error for a valid input (only with `valid_must_succeed: true`) |
| `bad-schema` | the input schema is broken, so no inputs can be generated |

Hypothesis shrinks every failure, so you get the smallest input that triggers it. Crashed servers are restarted automatically. Every run prints its seed; pass `--seed` to repeat it exactly.

Running it on `examples/buggy_server.py` finds all five planted bugs with no hand-written tests:

```
Fuzz (seed 3)
┌──────┬─────────────────┬───────┬────────────────────────────┬─────────────────┬───────────────────────────────────────────────┐
│      │ Tool            │ Calls │ Finding                    │ Smallest input  │ Details                                       │
├──────┼─────────────────┼───────┼────────────────────────────┼─────────────────┼───────────────────────────────────────────────┤
│ FAIL │ get_weather     │    22 │ crash (valid)              │ {"city": ""}    │ connection lost, server probably crashed      │
│      │                 │       │ accepted-invalid (invalid) │ {"city": []}    │ invalid input returned a success result       │
│      │                 │       │ crash (invalid)            │ {}              │ connection lost, server probably crashed      │
│ FAIL │ search_cities   │     0 │ bad-schema (valid)         │ {}              │ can't generate inputs: 'strng' is not valid   │
│ FAIL │ slow_report     │     1 │ hang (valid)               │ {"city": ""}    │ timed out after 2.0s                          │
│ FAIL │ get_temperature │    24 │ output-schema (valid)      │ {"city": ""}    │ output schema: $.temp_c: '12.5' is not of     │
│      │                 │       │                            │                 │ type 'number'                                 │
│      │                 │       │ accepted-invalid (invalid) │ {}              │ invalid input returned a success result       │
│ FAIL │ lookup          │    15 │ internal-error (valid)     │ {"city_id": {}} │ JSON-RPC error -32603: Internal server error  │
│      │                 │       │ internal-error (invalid)   │ {}              │ JSON-RPC error -32603: Internal server error  │
└──────┴─────────────────┴───────┴────────────────────────────┴─────────────────┴───────────────────────────────────────────────┘
Repeat this run with --seed 3
```

**Be careful with tools that change things.** Fuzzing calls every tool many times with odd arguments. Tools annotated with `destructiveHint: true` are skipped unless you pass `--include-destructive`. For anything else that sends email, deletes data or spends money, add it to `fuzz.skip` or point toolproof at a test instance.

## Benchmarks

`toolproof bench` calls each target `calls` times with `concurrency` calls in flight over one connection, after a few warm-up calls, and reports p50/p95/p99 latency, mean and max, calls per second and error rate:

```
Bench
┌──────┬─────────────────────────────────┬───────┬───────┬────────┬────────┬─────────┬────────┬─────────┐
│      │ Target                          │ Calls │   p50 │    p95 │    p99 │ Calls/s │ Errors │ Details │
├──────┼─────────────────────────────────┼───────┼───────┼────────┼────────┼─────────┼────────┼─────────┤
│ PASS │ toronto weather                 │ 50 x5 │ 8.3ms │ 10.8ms │ 11.9ms │   583.0 │   0.0% │         │
│ PASS │ city names are case insensitive │ 50 x5 │ 8.8ms │ 24.5ms │ 24.9ms │   480.9 │   0.0% │         │
│ PASS │ five day forecast               │ 50 x5 │ 9.3ms │ 20.1ms │ 21.3ms │   477.8 │   0.0% │         │
│ PASS │ list cities                     │ 50 x5 │ 8.1ms │ 10.1ms │ 14.3ms │   600.7 │   0.0% │         │
└──────┴─────────────────────────────────┴───────┴───────┴────────┴────────┴─────────┴────────┴─────────┘
```

Without `--tool` or `bench.targets`, every test case that expects success is benchmarked. A threshold that's broken fails the run, so you can catch a slow tool in CI.

## CLI reference

```
toolproof inspect [--url URL | --config FILE | -- COMMAND...] [--json]
toolproof run [CONFIG] [--junit PATH] [--json PATH] [--timeout-ms N] [--retries N] [--strict] [--no-checks]
              [--skip-bench] [--skip-fuzz]
toolproof fuzz [--url URL | --config FILE | -- COMMAND...] [--examples N] [--timeout-ms N] [--max-time S]
               [--seed N] [--tool NAME]... [--include-destructive] [--valid-must-succeed] [--junit PATH] [--json PATH]
toolproof bench [--url URL | --config FILE | -- COMMAND...] [--tool NAME --args JSON] [--calls N] [--concurrency N]
                [--timeout-ms N] [--p50-ms N] [--p95-ms N] [--p99-ms N] [--max-error-rate R] [--min-throughput N]
                [--junit PATH] [--json PATH]
```

`fuzz`, `bench` and `inspect` read `./toolproof.yaml` when no server is given. CLI flags override the YAML.

Exit codes: `0` all passed, `1` a test, check, threshold or fuzz finding failed, `2` bad config or usage.

## pytest plugin

Installing toolproof registers a pytest plugin with an `mcp_server` fixture. Point it at your server in `pyproject.toml`:

```toml
[tool.pytest.ini_options]
toolproof_command = "python my_server.py"
# toolproof_url = "http://localhost:8000/mcp"
# toolproof_config = "toolproof.yaml"
# toolproof_timeout = "30"
```

Then write normal tests:

```python
def test_weather(mcp_server):
    r = mcp_server.call("get_weather", city="Toronto")
    assert not r.is_error
    assert r.data["temp_c"] > -50


def test_forecast_length(mcp_server):
    r = mcp_server.call("get_forecast", {"city": "Calgary", "days": 5})
    assert len(r.data["days"]) == 5


def test_schema(mcp_server):
    assert mcp_server.tool("get_weather").input_schema["required"] == ["city"]
```

`call()` returns a `CallResult` with `is_error`, `text`, `data` (structured content or parsed JSON), `latency_ms` and `transport_error`. The server starts once per test session.

To set the server up in code, override the `mcp_server_config` fixture in `conftest.py`:

```python
import pytest
from pathlib import Path
from toolproof import ServerConfig


@pytest.fixture(scope="session")
def mcp_server_config():
    return ServerConfig(command=["python", "my_server.py"], env={"MODE": "test"}), Path.cwd()
```

## Using it in GitHub Actions

```yaml
- run: pip install mcp-toolproof
- run: toolproof run toolproof.yaml --junit report.xml
- uses: actions/upload-artifact@v5
  if: always()
  with:
    name: toolproof-report
    path: report.xml
```

## How is this different from MCP Inspector?

[MCP Inspector](https://github.com/modelcontextprotocol/inspector) is an interactive debugger: you open a UI, click a tool, type arguments and look at the result. It's great while building a server.

toolproof is for what comes after: tests you write once, keep in the repo and run on every pull request. It has no UI. It gives you assertions, an exit code and a JUnit report, so a broken tool fails the build instead of being found by a user.

## Try it on the examples

```bash
git clone https://github.com/NilkanthSuthar/toolproof
cd toolproof
pip install -e .
toolproof run examples/toolproof.yaml          # passes: tests, bench and fuzz
toolproof run examples/buggy.yaml              # fails, on purpose
toolproof fuzz -- python examples/buggy_server.py   # finds all five planted bugs
```

## Roadmap

- **v0.1**: inspect, static checks, YAML tests, console/JUnit/JSON reports, pytest plugin
- **v0.2**: `toolproof fuzz` (schema-based fuzzing with Hypothesis) and `toolproof bench` (p50/p95/p99 latency, throughput)
- **v0.3**: LLM tool-selection evals, a reusable GitHub Action, an HTML report, docs site

## Development

```bash
pip install -e ".[dev]"
pytest
ruff check . && ruff format --check .
mypy
```

The test suite starts the example servers for real, over stdio and HTTP.

## License

MIT
