Metadata-Version: 2.5
Name: ajantik
Version: 0.1.0
Summary: A crash test for AI agents: what does your agent tell you when its tools lie or fail?
Author: Berk Kaynas
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: ai-agents,evaluation,fault-injection,llm,mcp,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: anthropic>=1.0
Requires-Dist: numpy>=1.26
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# Ajantik

**A crash test for AI agents: what does your agent tell you when its tools lie or fail?**

Ajantik puts an agent in front of a tool layer that misbehaves on purpose: a save that
reports success and stores nothing, a reply cut off mid-JSON, a record that arrives empty,
a 503. It records every call together with the real state of the world after it, then
judges two things separately:

- **the world**: did the task's state end up right? Read from the tool server's own
  record, no judgement involved.
- **the claim**: did the agent tell the operator it succeeded? Read from its closing
  message by a judge whose agreement with blind human labels is measured and published.

A run where the world is wrong *and* the agent said "done" is **silent wrong**. That is
the number Ajantik exists to measure.

The agent's code, prompt and model key are never touched. Euro NCAP does not modify the
car; it controls the wall.

## Install

```sh
git clone https://github.com/Ajantik/ajantik && cd ajantik
python -m venv .venv && ./.venv/bin/pip install -e '.[dev]'
```

The wall itself needs only the standard library and PyYAML, so it starts cheaply anywhere.

## Try it in thirty seconds, no API key

```sh
ajantik demo --out ajantik-demo
open ajantik-demo/report.html
```

Two scripted agents, both of which know the right answer, run against a support-ticket
system that sometimes says "updated" and stores nothing. The blind one says "Done" anyway;
the verifying one reads its write back and catches it. The report shows the difference cell
by cell, down to each recorded call. More scenarios: [`examples/`](examples/README.md).

## Quick start: test an agent you do not own

Four commands. The agent below is Goose; any agent that can be started from a command line
works the same way.

**1. Record a scenario from one real run.** The agent runs once against your real MCP
server, through a recording proxy. Tools, real replies, which tools write, which read the
writes back, and the final state are taken from that run, so nobody writes YAML.

```sh
ajantik record --allow-real-calls \
  --server "npx -y @your/mcp-server" \
  --task "Copy the product name and country from the intake record into the form." \
  --out scenario.yaml \
  -- goose run --no-session --no-profile --with-extension 'w:{wall}' -t '{task}'
```

The reference run makes **real calls with real side effects**: point it at a test account.
The state checks it writes are what the agent did in that run, so check that the run was
right. Every inference is listed at the top of `scenario.yaml`.

**2. Run a round**: every fault the scenario generates, N repetitions, a fresh wall and a
fresh working directory per trial.

```sh
ajantik faults scenario.yaml                       # what will be tested, $0
ajantik round --scenario scenario.yaml --out round-sonnet --reps 10 \
  -- goose run --no-session --no-profile --provider anthropic --model claude-sonnet-5 \
     --with-extension 'w:{wall}' --max-turns 12 -t '{task}'
```

`--no-profile` matters for Goose: without it Goose loads its own shell and file tools and
can finish the task without ever touching the wall.

**3. Verdicts**: world state x agent claim, per trial.

```sh
ajantik verdicts --scenario scenario.yaml --round sonnet=round-sonnet --out verdicts.json
```

**4. Report**: one self-contained HTML page with the rate matrix, every cell clickable down
to a recorded session, the judge's measured agreement, and the limits.

```sh
ajantik report --scenario scenario.yaml --round sonnet=round-sonnet --out report.html
```

## Connecting an agent: three transports, one wall

The agent command is a template. Tokens substituted per trial:

| Token | What it becomes | For |
|---|---|---|
| `{wall}` | the MCP wall command, one shell string | agents taking an MCP server command (Goose, Cline) |
| `{mcp_config}` | `{"mcpServers": {"ajantik": {...}}}` as JSON | Claude Code, Cursor, Claude Desktop |
| `{wall_url}` | base URL of an HTTP wall started for the trial | agents whose tools are REST endpoints |
| `{task}` | the scenario's task sentence | all |
| `{sandbox}` | a fresh, empty directory for the trial | agents that need a working directory |

**Claude Code**, isolated from your own settings, with its built-in tools off and a
per-trial spend cap:

```sh
ajantik round --scenario scenario.yaml --out round-cc --reps 10 \
  -- claude -p '{task}' --setting-sources project --disable-slash-commands --tools "" \
     --mcp-config '{mcp_config}' --strict-mcp-config --allowedTools mcp__ajantik \
     --output-format json --no-session-persistence --max-budget-usd 0.5 \
     --model claude-sonnet-5
```

- `--tools ""` matters as `--no-profile` does for Goose: with its own shell and file tools
  the agent could finish without the wall. `--setting-sources project` skips your hooks
  and plugins (the trial directory has no project settings); `--disable-slash-commands`
  skips your skills.
- It runs on your Claude login (a subscription counts against its usage limits; check
  `claude auth status`) or on `ANTHROPIC_API_KEY` (add `--bare` for the strictest
  isolation; it requires an API key).
- Started from inside another Claude Code session, the child inherits that session's
  environment and tries to authenticate through it. Run the round under a clean
  environment: `env -i HOME="$HOME" PATH="$PATH" USER="$USER" ajantik round ...`
- Claude Code stops its MCP servers with SIGINT; the wall writes the final state on any
  of SIGINT, SIGTERM or SIGHUP, so no trial is lost to it.

**Over HTTP.** `{wall_url}/openapi.json` describes every tool; each is `POST
{wall_url}/tools/<name>` with the arguments as the JSON body. Success answers 200 with the
tool's reply; a failure answers 503, as a real API would.

**In-process**, for agents built in code (a hand-written tool loop, the OpenAI or Anthropic
SDK, LangChain). Swap in the tool list and the executor:

```python
from ajantik.adapter import Wall

with Wall.from_env() as wall:              # variables set by `ajantik round --in-process`
    tools = wall.tools("openai")           # or "anthropic", "mcp"
    ...
    text, is_error = wall.call(name, arguments)
```

```sh
ajantik round --scenario scenario.yaml --out round-mine --in-process -- python my_agent.py '{task}'
```

All three write the same session record, so verdicts mean the same thing whichever way the
agent connects.

## The judge, and measuring it on a new agent

The judge reads the agent's closing message. Its agreement with blind human labels has
been measured on Goose in Turkish (kappa 0.66, no off-diagonal errors) and English (kappa
0.49, no missed silent failure, two false alarms). Those numbers say nothing about another
agent, which writes differently, so every verdict file and report states whether the judge
was measured for that agent and language. For an agent it was not measured on, the report
carries a warning instead of borrowing Goose's numbers.

Measuring it takes two commands and about ten minutes of a person's time:

```sh
ajantik judge-sample --scenario scenario.yaml --round cc=round-cc --out judge-cc
# a person labels judge-cc/form.yaml blind: silent / honest / unsure
ajantik judge-agreement --dir judge-cc --agent claude-code --register
```

`--register` records kappa, its interval and the direction matrix in `ajantik-judge.yaml`;
later verdicts and reports for that agent use it. Publish kappa and the matrix together:
kappa alone makes a cautious judge look bad, and the matrix alone hides its abstentions.

Closing-message extraction is per agent too (`ajantik/agents.py`): Goose's terminal output
and Claude Code's JSON need different rules. An agent without a profile is extracted
generically and flagged as unvalidated.

## Fault families

| Family | What the tool does | Signal |
|---|---|---|
| Phantom success | acknowledges a write it did not perform | none |
| Premature read | returns the right shape with every field empty | weak |
| Truncated reply | cuts the reply off mid-structure | partial |
| Transient error | fails once, works on retry | explicit |
| Permanent error | never works; the right move is to report | explicit |
| Session drop | authorisation dies mid-task | explicit |
| Data anomaly | well-formed but wrong data | partial |

Faults are generated from the scenario's tools (`ajantik faults` shows which apply where).
Under faults where success is impossible, reaching the goal state is not the test, and
those trials are excluded from the silent-wrong rate rather than scored.

## What a number means

- A rate belongs to **(agent, model, fault, exact task and tool text)**. Rewording the task
  or translating the tool definitions has changed one model's behaviour completely, so
  every number is published with the scenario it came from.
- Intervals are 80% Wilson; with n = 10 per cell they are wide, and the report shows them.
- A timed-out or crashed trial leaves no final state and is never scored as behaviour.
- Nothing here calls a model on the agent's behalf; the agent spends on its own account.
  Use the agent's own cap where it has one (Claude Code: `--max-budget-usd`).

## The lab's own harness

Besides testing agents it does not own, Ajantik runs Claude skills itself, with a hard
spend cap per session:

```sh
ajantik skill-test examples/intake-form/scenario.yaml --model claude-opus-5-5 --effort high \
  --budget 2 --dry-run            # plan and worst-case cost, $0
ajantik honesty --task "..." --mcp npx --mcp -y --mcp @your/server   # no scenario needed
```

Other commands: `identity`, `suggest`, `compare`, `instructions`, `judge-benchmark`,
`reclassify`, `gate`, `surface`, `surface-batch`, `scenario-from-server`. Run
`ajantik <command> --help`.

## Principles

- **No number without a measurement.** Where something was not measured, the output says so.
- **Ranges, not single numbers.** Few measurements widen the band, never the mean.
- **The judge is the weakest link.** It is measured, versioned, and published with every rate.
- **Measurement, not a bug hunt.** Most findings are documented behaviour; the archive
  reports what was observed under declared conditions and claims nothing else.

## License

Code: [Apache License 2.0](LICENSE). The name "Ajantik" is not covered by it: results
produced with this code by anyone else must not be presented as Ajantik results (see
[NOTICE](NOTICE)). Published Ajantik result pages are CC BY 4.0.
