Metadata-Version: 2.4
Name: argwitness
Version: 0.3.0
Summary: Concrete counterexamples for breaking AI tool contracts. Offline, cross-provider, evidence-first.
Author: GenRamzi
License-Expression: MIT
Project-URL: Repository, https://github.com/GenRamzi/argwitness
Project-URL: Documentation, https://github.com/GenRamzi/argwitness/tree/main/docs
Project-URL: Changelog, https://github.com/GenRamzi/argwitness/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/GenRamzi/argwitness/issues
Keywords: mcp,ai-agents,json-schema,contract-testing,compatibility
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jsonschema<5,>=4.23
Requires-Dist: referencing<1,>=0.35
Dynamic: license-file

# ArgWitness

**Don't just flag a breaking tool change. Show the call it breaks.**

[English](README.md) · [العربية](README.ar.md) · [Schema support](docs/schema-support.md) · [Design](docs/design.md) · [Contributing](CONTRIBUTING.md)

ArgWitness finds concrete JSON arguments that an old AI tool schema accepts and a new one rejects. It imports saved **MCP**, **OpenAI function tools**, and **Anthropic tools**, searches for counterexamples, shrinks them, and checks captured calls offline.

No API keys. No model calls. No server execution. No telemetry.

> **Alpha v0.3.0.** A confirmed counterexample proves one schema incompatibility. Failing to find one does **not** prove compatibility. Unresolved changes fail CI with `review`, not a green checkmark.

## The problem in one example

You change a search tool's maximum `limit` from `100` to `20`. The tool still exists and simple smoke tests pass. Previously valid agent calls now fail.

ArgWitness produces evidence:

```console
$ argwitness compare examples/before.mcp.json examples/after.mcp.json --show-values
ArgWitness 0.3.0 — BREAKING
  AW002 "search_docs": breaking — Concrete call accepted before and rejected after
    OLD accepts -> NEW rejects (generated)
    {"query": "a", "limit": 100}
```

The arguments are validated against **both** schemas before reporting. `--show-values` is explicit because even schema enums and examples can contain sensitive information.

## Try it in a minute

Requires Python 3.10+. Install the public release from PyPI:

```bash
python -m pip install argwitness
argwitness --version
```

To run the repository demo from source:

```bash
git clone https://github.com/GenRamzi/argwitness.git
cd argwitness
python -m pip install .
python examples/demo.py
```

The demo generates a witness, checks two synthetic calls against the new contract, and independently verifies a saved witness. It requires no live services. Installation downloads dependencies; subsequent analysis is offline.

## Four commands

```bash
# Search for calls that break after a schema change. A detected break exits 1.
argwitness compare before.json after.json

# Add sanitized production-shaped examples; values are omitted from reports by default.
argwitness compare before.json after.json --calls calls.jsonl --format json

# Check captured arguments against a catalog without invoking any tool.
argwitness replay after.json calls.jsonl --format markdown

# Independently recheck {"tool": "name", "arguments": {...}} evidence.
argwitness verify before.json after.json witness.json

# Normalize provider-specific envelopes to a shared tool catalog.
argwitness normalize tools.json\n\n# Select one effective view from a multi-protocol MCP Description document.\nargwitness normalize server.mcpdesc.json --protocol-version 2026-07-28
```

Reports support `text`, `json`, and `markdown`. Redirect stdout to save a report. `normalize` always emits JSON and preserves catalog values; it is not a redaction command.

## What it catches

| Change | Evidence |
| --- | --- |
| New required field | A formerly valid call without that field |
| Narrowed enum | A value that was removed |
| Tightened number or length bound | A call on an old/new boundary |
| Number changed to integer | A fractional argument |
| Closed additional properties | A formerly accepted extra property |
| Nested field or array changes | An object containing a rejected nested value |
| Removed tool | A structural removal finding |
| Description, annotations, strict mode, or output schema changed | Human review; input validation cannot establish semantic behavior |

Generated search is deterministic and bounded. It is strongest on common object arguments and scalar/array boundaries. Complex patterns, interacting conditions, large structures, and recursive schemas may need captured calls or human review. See [exact limits](docs/schema-support.md).

## Provider inputs

| Source | Catalog shape | Captured call shape |
| --- | --- | --- |
| MCP | `{"tools":[{"name":"...","inputSchema":{...}}]}` or JSON-RPC `result` | `tools/call` request, or `{tool, arguments}` |
| mcp-contracts snapshot | `.mcpc.json` with `snapshotVersion` and object-map `tools` | Use sanitized calls in ArgWitness format if desired |\n| MCP Description (`mcpdesc`) | JSON `mcpdesc` document with `tools[]`; use `--protocol-version` for authored multi-protocol variants | Use sanitized calls in ArgWitness format if desired |
| OpenAI Chat Completions | `[{"type":"function","function":{"name":"...","parameters":{...}}}]` | `{function:{name,arguments}}` |
| OpenAI Responses | `[{"type":"function","name":"...","parameters":{...}}]` | `{type:"function_call",name,arguments}` |
| Anthropic | `[{"name":"...","input_schema":{...}}]` | `{type:"tool_use",name,input}` |

Save tool arrays rather than entire API request bodies. Calls can be JSONL or a JSON array; OpenAI argument strings are decoded as JSON. Combine all MCP catalog pages first. Duplicate tool names and incomplete paginated catalogs are rejected. For authored multi-protocol `mcpdesc` files that contain multiple variants of the same tool name, select one effective view with `--protocol-version`. Tools from different servers still need explicit namespacing.

These are **schema-envelope adapters**, not live SDK integrations or provider-conformance certification. For mcp-contracts snapshots, ArgWitness reads the tool schemas and metadata only; it does not verify snapshot hashes/signatures or analyze resources/prompts. See [integration examples](docs/integrations.md).

## Honest CI states

| Exit | Meaning | Suggested action |
| --- | --- | --- |
| `0` | Existing schemas and metadata unchanged, or tools only added | Proceed under your policy; runtime behavior is untested |
| `1` | A breaking witness or removed tool; replay failure; unconfirmed verification | Block and inspect |
| `2` | Change needs review or baseline traffic was invalid | Review; do not silently ignore |
| `3` | Bad input, unsupported schema, missing file, or analysis error | Fix the input/configuration |

If any breaking finding exists, comparison exits `1` even when there are also review findings. A widening input schema returns `review` in this release: there is no schema-inclusion prover.

## GitHub Action

ArgWitness can run directly in pull requests. Pin the immutable `v0.1.0` release tag (or an immutable commit SHA) rather than `main`:

```yaml
name: Tool contract compatibility

on:
  pull_request:

permissions:
  contents: read

jobs:
  contracts:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: GenRamzi/argwitness@v0.2.0
        with:
          before: contracts/baseline.json
          after: contracts/candidate.json
          calls: contracts/sanitized-calls.jsonl
          format: markdown
          report: artifacts/argwitness.md
```

The Action installs the package from the pinned Action revision, writes the report inside the caller workspace, and fails closed for `breaking`, `review`, or input-error states. Raw argument values remain omitted unless `show-values: "true"` is explicitly enabled. Paths are constrained to the caller workspace.

## Python API

```python
from argwitness import compare, replay, verify_witness

before = {"tools": [{"name": "search", "inputSchema": {"type": "object"}}]}
after = {"tools": [{"name": "search", "inputSchema": {
    "type": "object", "required": ["query"]
}}]}

report = compare(before, after, show_values=True)
assert report["status"] == "breaking"
value = next(f["witness"]["arguments"] for f in report["findings"] if "witness" in f)
assert verify_witness(before, after, {"tool": "search", "arguments": value})["status"] == "confirmed"
```

## Trust boundaries

- The validator is `python-jsonschema`, using Draft 2020-12. `format` is annotation-only. Unknown keywords, unsupported dialects, and external references are rejected.
- Reports omit argument values by default but include tool names, schema paths, and catalog hashes. Those may still be sensitive. Sanitize before sharing.
- JSON Schema regexes can consume CPU. File/search limits are not a security sandbox; process untrusted schemas in a resource-limited environment.
- A witness says nothing about business logic, model routing, authorization, output compatibility, or whether a live server follows its advertised schema.
- Report digests identify normalized input catalogs. They are not signatures, identity attestations, or proof that a server ever exposed a catalog.

## Development

```bash
python -m pip install -e .
python -m unittest discover -s tests -v
python examples/demo.py
python -m pip install build
python -m build
```

The tests include end-to-end CLI exit behavior, provider envelopes, privacy defaults, and an independent validation check over 144 schema pairs. These are synthetic tests, not evidence of production adoption or a benchmark against other tools.

## Why another tool?

MCP contract diffing already has useful projects, including [mcp-contracts](https://github.com/mcp-contracts/mcp-contracts) and [Cisco mcptoolkit-contract](https://github.com/cisco-open/mcptoolkit-contract). ArgWitness focuses on a small reusable job: **find and independently recheck a concrete breaking input across provider envelopes**, then incorporate sanitized observed calls. It can complement a schema diff pipeline.

We make no first-of-its-kind, performance, or superiority claim. See [positioning and sources](docs/positioning.md). The most useful next contribution is a small real schema change where the generated search misses a counterexample.

## License

MIT. Maintained by GenRamzi. No affiliation with or endorsement from Anthropic, OpenAI, or the MCP project.
