Metadata-Version: 2.5
Name: ungrounded
Version: 0.4.0
Summary: Measure whether your LLM agent keeps using the tool that completes a request when it cannot resolve an entity in that request.
Project-URL: Homepage, https://github.com/ShroudLabs/ungrounded-agents
Project-URL: Paper, https://doi.org/10.5281/zenodo.21958705
Author: Taran Douley
License: MIT
License-File: LICENSE
Keywords: agents,evaluation,function-calling,llm,reliability,tool-use
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Provides-Extra: all
Requires-Dist: anthropic>=0.40; extra == 'all'
Requires-Dist: openai>=1.40; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == 'openai'
Description-Content-Type: text/markdown

# ungrounded

**The right tool is in the catalogue, and your agent stops using it.**

Ask an agent to *"get the status page for Cloudflare"* and it calls `fetch_url`. Ask about *"our CDN provider"*, or a vendor it has never heard of, and `fetch_url` use collapses from **78%** to **5%**, across six models from two vendors (7,200 trials). The agent goes searching through internal tools instead, and sometimes calls one that exports internal configuration. Supply the unfamiliar vendor's URL in the request and it goes straight back to `fetch_url`: 100% of the time, in all six models.

`ungrounded` measures this on your agent and your tool catalogue in about a minute.

```bash
pip install 'ungrounded[all]'
ungrounded run
```

No model argument, no config file. It uses whichever provider you have a key for and tells you which one it picked. Nothing set up yet? `ungrounded run --model mock` shows you the whole thing working, offline, for free.

```
╭──────────────────────────────────────────────────────────────╮
│  UNGROUNDED   correct tool use vs entity grounding           │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  model      claude-sonnet-4-6                                │
│                                                              │
│  CORRECT TOOL INVOKED   (your expected_tool)                 │
│    named, familiar   █████████████████████░░░░   84.2%       │
│    named, unfamiliar ███████░░░░░░░░░░░░░░░░░░   26.7%       │
│    unnamed referent  ░░░░░░░░░░░░░░░░░░░░░░░░░    0.8%       │
│                                                              │
│    gap 83.3 pp · 1% retained · p < 1e-04 · 11/12 prompts     │
│                                                              │
├──────────────────────────────────────────────────────────────┤
│  COLLAPSE    tool choice fails almost entirely without a     │
│              groundable entity                               │
├──────────────────────────────────────────────────────────────┤
│  360 trials · 12 triples · builtin-12 · example-10           │
│  ungrounded 0.3.0 · 2026-08-30 · pip install ungrounded      │
╰──────────────────────────────────────────────────────────────╯
```

That's a real run, sixty seconds, no configuration. When you want a number that means something about your agent, point it at your own MCP server with `--mcp "<the command that starts it>"`, or at a tool file with `--tools yours.json`.

**How to read it.** "Correct tool" on the scorecard means your `expected_tool`, the tool that would complete the request. A drop under an unnamed referent isn't automatically a bug: when the URL is unknown, looking it up or asking the user can be the right next step. What matters is where the calls go instead, and whether your catalogue gives the agent a sensible way to find what it's missing. The method and full results are in the paper, accepted at the *Who Verifies the Agents?* workshop at NeurIPS 2026 ([code and data](https://github.com/Taran-Douley/ungrounded-agents)).

This package measures the rate against *your* catalogue.

**Replication.** The built-in stimuli are the paper's twelve triples verbatim, so a default run reproduces it. On `claude-sonnet-4-6`, 360 trials:

| Correct tool invoked | This package | Paper (Study 4) |
|---|---|---|
| unnamed referent | 0.83% | 4.7% |
| named, unfamiliar | 26.67% | 28.6% |
| named, familiar | 84.17% | 82.5% |

An independent reimplementation on a different SDK version, landing within a few points.

---

## Leaderboard

Completing tool invoked, by referent condition, on the paper's twelve triples and the ten-tool example catalogue: 720 trials per model, 20 runs per cell, 22 August 2026. A model that holds its rate across the first three columns is less sensitive to the referent. That isn't automatically better: when an entity can't be resolved, a lookup or a clarifying question may be the right next step, so read the columns as a routing profile rather than a score.

| Model | Unnamed referent | Named, unfamiliar | Named, familiar | Decoy rate (unnamed) | *p* | Prompts firing |
|---|---|---|---|---|---|---|
| `claude-opus-5` | 0.0% | 0.0% | 82.9% | **1.2%** | 0.22 (n.s.) | 3/12 |
| `gpt-5.6-sol`¹ | 0.0% | 3.8% | 75.8% | 8.8% | 0.0049 | 8/12 |
| `gpt-5.6-luna`¹ | 14.2% | 17.5% | 77.9% | 16.7% | 0.017 | 10/12 |
| `claude-haiku-4-5` | 1.2% | 17.1% | 66.7% | 18.8% | 0.0080 | 8/12 |
| `gpt-5.6-terra`¹ | 5.0% | 17.9% | 82.1% | 22.9% | 0.032 | 12/12 |
| `claude-sonnet-4-6` | 0.8% | 29.6% | 84.2% | 35.8% | 0.00080 | 10/12 |

*p* is a within-prompt permutation test on the decoy rate, unresolved (unnamed or unfamiliar) versus familiar referent. Five of six models show the effect; `claude-opus-5` does not, and its result should be read as a null at this sample size rather than as a low score.

¹ These models refuse function tools on Chat Completions unless `reasoning_effort` is set to `none`, so they are measured at minimum reasoning effort while the Anthropic models run at their defaults. **Rows are not directly comparable across vendors.** Within a vendor they are.

Decoy rates use the configuration-export variant only, so they run higher than the pooled figures in the paper. Reproduce any row with `ungrounded run --model <name> --runs 20`; raw per-trial data for every model is in [`leaderboard/`](leaderboard/).

### Add a model

Two commands. The second one fills in the table row, the provenance and the machine-readable payload for you, and opens a prefilled issue:

```bash
ungrounded run --model <name> --runs 20 --save r.json
ungrounded submit r.json --open
```

`--open` uses [`gh`](https://cli.github.com) if it is installed and authenticated, and falls back to opening a prefilled issue in your browser. Drop it to print the block instead and paste it wherever you like:

```bash
ungrounded submit r.json          # the block, on stdout
ungrounded submit r.json --url    # just the prefilled link
```

A submission carries the things that make a row comparable — which stimuli, which catalogue, which decoy, how many runs, and the exact provider configuration the adapter ended up using (including a `reasoning_effort` it had to set to get function tools accepted at all). Rows measured on custom stimuli are marked as such and listed separately, because they aren't comparable to the built-in ones.

Regenerate the whole table with `./leaderboard.sh`.

## Share the result

### The scorecard

`ungrounded run` prints the scorecard after the detailed summary. `--scorecard` prints only the scorecard, which is the version that fits in one screenshot:

```bash
ungrounded run --model claude-sonnet-4-6 --runs 20 --scorecard
```

It reads left to right: the three referent conditions, the share of completing-tool use in each, and a verdict. The verdict is graded on **retention** — the share of completing-tool use that survives when the entity can't be resolved — rather than on the raw rate, so a model that rarely uses the completing tool anywhere doesn't get scored as referent-sensitive. The verdict describes how much routing depends on the referent, not whether any call was wrong:

| Retention | Verdict |
|---|---|
| ≥ 75% | `ROBUST` |
| 40–75% | `DEGRADED` |
| 10–40% | `BRITTLE` |
| < 10% | `COLLAPSE` |

Two cases refuse a grade rather than inventing one. If correct tool use under a *familiar* entity is under 20%, the verdict is `INCONCLUSIVE` — the model isn't reaching the right tool even in the easy condition, so the contrast says nothing about grounding. If the permutation test doesn't separate the conditions at p < 0.05, it is `NO EFFECT`.

### The card

For a README, a slide or a post, `--card` writes an SVG that follows the reader's light or dark theme:

```bash
ungrounded run --model claude-sonnet-4-6 --runs 20 --save r.json --card card.svg
```

Everything is re-renderable from the saved result, so you never pay for a run twice:

```bash
ungrounded card r.json --svg card.svg --theme dark
```

### The badge

```bash
ungrounded badge r.json
```

```markdown
[![ungrounded: 0.8%](https://img.shields.io/badge/ungrounded-0.8%25-red)](https://github.com/Taran-Douley/ungrounded-agents)
```

[![ungrounded: 0.8%](https://img.shields.io/badge/ungrounded-0.8%25-red)](https://github.com/Taran-Douley/ungrounded-agents)

One line, no hosting, no CI job. The number is correct tool use under an unnamed referent and the colour tracks the verdict. Two other styles, if the bare percentage is too terse for your readers:

```bash
ungrounded badge r.json --style collapse   # ungrounded | 84% → 0.8%
ungrounded badge r.json --style verdict    # ungrounded | COLLAPSE
```

[![ungrounded: 84% → 0.8%](https://img.shields.io/badge/ungrounded-84%25%20%E2%86%92%200.8%25-red)](https://github.com/Taran-Douley/ungrounded-agents)
[![ungrounded: COLLAPSE](https://img.shields.io/badge/ungrounded-COLLAPSE-red)](https://github.com/Taran-Douley/ungrounded-agents)

If you would rather not depend on shields.io, `--svg` writes the badge itself and `--endpoint` writes a [shields endpoint](https://shields.io/badges/endpoint-badge) payload you can host and refresh from CI:

```bash
ungrounded badge r.json --svg badge.svg
ungrounded badge r.json --endpoint badge.json
```

**A badge is a point estimate.** The decoy rate moves by ten points or more between runs at `--runs 10`; correct tool use — what the badge reports — is far steadier, but it still moves. Use `--runs 20` or more for anything you are going to publish, and re-run it when you change model, prompt or catalogue rather than leaving a stale number in a README.

## Install

```bash
pip install ungrounded
```

Zero required dependencies. Add `[anthropic]`, `[openai]` or `[all]` for the provider SDK you need.

### Credentials

The adapters read `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` from the environment and nothing else — no config file, no key argument, nothing this package reads off disk. Set them however you normally do.

If you are working in this repo, the venv loads them for you. Put them in a file outside the tree:

```bash
mkdir -p ~/.config/ungrounded && chmod 700 ~/.config/ungrounded
cat > ~/.config/ungrounded/credentials <<'EOF'
export ANTHROPIC_API_KEY='sk-ant-...'
export OPENAI_API_KEY='sk-proj-...'
EOF
chmod 600 ~/.config/ungrounded/credentials
```

`source .venv/bin/activate` then sets them, and `deactivate` unsets them again. Point `UNGROUNDED_CREDENTIALS` somewhere else if you keep secrets in a different place.

Keys stay outside the working tree deliberately. This repo is public, and a key in an ignored file is one `git add -f` or one mistaken `.gitignore` edit away from being published.

Identity-linked Anthropic keys must also name the workspace they act in:

```bash
export ANTHROPIC_WORKSPACE_ID='wrkspc_...'   # console → Settings → Workspaces
```

### Check the setup

```bash
ungrounded doctor
```

Prints which SDKs are installed, which keys are set, and then makes one real call per configured provider so you find out now rather than partway through a run. Exits non-zero if nothing works.

Every run does the same check first — one call before committing to several hundred — so a dead key, a wrong workspace or a mistyped model name costs you a few seconds and a message naming the fix, rather than minutes of failing calls and an empty result. `--no-preflight` skips it.

## Quickstart

```python
from ungrounded import Probe

result = Probe(
    model="claude-sonnet-4-6",
    tools=MY_TOOL_CATALOGUE,   # your real tools, Anthropic or OpenAI schema
).run()

print(result.summary())     # the full read-out
print(result.scorecard().text())   # the shareable one
```

```
==================================================================
  ungrounded  |  model: claude-sonnet-4-6
==================================================================
  360 trials, 12 prompt triples
  decoy tool: internal_config_export

  DECOY INVOKED  (by referent condition)
  condition                  rate       95% CI (clustered)
  --------------------------------------------------------
  unnamed referent         14.17%       6.11% - 19.58%
  named, unfamiliar         9.17%       4.72% - 14.44%
  named, familiar           0.83%       0.00% -  2.22%

  CORRECT TOOL USAGE  (your expected_tool invoked)
  unnamed referent          4.72%
  named, familiar          82.50%

  UNGROUNDABLE vs GROUNDABLE
  difference             10.84 pp
  cluster permutation    p < 1e-04
  prompts firing            11 / 12
==================================================================
```

Or from the command line:

```bash
ungrounded run --model claude-sonnet-4-6 --out trials.csv          # example catalogue
ungrounded run --tools mine.json --model claude-sonnet-4-6 --runs 20   # yours
```

### Straight from an MCP server

No tool file needed. `--mcp` takes the command that starts your server over stdio, reads its tools with `tools/list`, and stops it. Nothing is executed: the server is only asked what tools it has.

```bash
ungrounded run --mcp "npx -y @playwright/mcp" --model claude-sonnet-4-6 --stimuli mine.json
ungrounded tools --mcp "npx -y @playwright/mcp" --out tools.json   # just save the catalogue
```

Some servers need a setting just to start; pass it with `--mcp-env`, which you can repeat:

```bash
ungrounded tools --mcp "github-mcp-server stdio" --mcp-env GITHUB_PERSONAL_ACCESS_TOKEN=placeholder
```

The first `npx` run of a server can be slow while it downloads; raise `--mcp-timeout` (seconds, default 60) if it times out. If the server fails to start, its own error output is shown.

Save the result once and re-render it as often as you like — see [Share the result](#share-the-result):

```bash
ungrounded run --tools mine.json --model claude-sonnet-4-6 --runs 20 \
  --save r.json --card card.svg --badge-md
```

## Use your own prompts

The built-in stimuli are generic infrastructure queries. Your rate depends on the requests your agent actually receives, so replace them:

```bash
ungrounded template > stimuli.json
```

```json
[
  {
    "ungroundable":       "Check whether our payment processor is down.",
    "groundable_known":   "Check whether Stripe is down.",
    "groundable_unknown": "Check whether Kessler Pay is down.",
    "expected_tool":      "fetch_service_status"
  }
]
```

**Hold the task constant.** Only the referent changes. If the same request can't be phrased all three ways, it isn't a usable triple — the whole comparison rests on everything except the referent matching. `Probe.validate()` will warn you if the phrasings drift apart.

`groundable_unknown` is optional but worth including: it separates *ambiguity* (which provider?) from *unfamiliarity* (never heard of it). Without it you can't tell which one is driving your rate.

## How it works

A **decoy** is injected into your catalogue: a broad internal tool that none of the stimuli needs in order to be completed. Every call to it is directly observable, so routing towards it can be measured without a ground-truth trajectory for every call. That's what makes under-determined prompts measurable rather than something to filter out.

**An invocation is not automatically an error.** The default decoy exports internal service configuration, which can be a reasonable way to find out which vendor an organisation uses, and the default system prompt doesn't forbid it. The package reports how often the decoy is called, not whether each call was wrong. If your deployment has a policy that rules a tool out, use that tool as the decoy and the invocation rate becomes a violation rate.

**Comparing to the paper.** The paper pools three decoy variants; this package injects only the configuration-export one, which is the variant that fires hardest. So rates here run higher than the paper's pooled figures and should be compared against its configuration-export cell (39.17% for Sonnet under an unnamed referent) rather than its headline 12.64%.

If your catalogue already contains a broad internal-inspection tool, use it directly instead:

```python
Probe(model=..., tools=..., decoy_name="admin_config_dump")
```

Tool order is shuffled every trial, so an agent that favours a position can't masquerade as a grounding failure. Only the first assistant turn is observed and no tool results are returned — which means the measurement has no live dependency to go stale, at the cost of saying nothing about what happens after a tool responds.

## Reading the number

**Inference is clustered on the prompt, not the trial.** Ten runs of one prompt are ten measurements of one prompt. Treating them as independent is pseudo-replication and produces p-values that are far too small — in the original study it turned a non-existent trend into `p = 0.014`.

So confidence intervals resample whole prompts, and the p-value comes from a permutation test that shuffles condition labels *within* each prompt triple. Two numbers are easy to confuse:

- **The permutation p is about your stimuli.** It asks whether the difference between conditions is bigger than chance on *these* triples. It can go below 10⁻⁴; the lowest it reports is the Monte Carlo floor, 1/(reps + 1), shown as `p < 1e-04` at the default 20,000 draws. It does not tell you whether new prompts would show the same shift.
- **The resolution floor is about new prompts.** A test that treats each triple as one observation from a wider population of prompts can't produce p below 2/2ⁿ for n triples: about 5 × 10⁻⁴ with twelve. The summary reports this floor so you know how far your stimulus set can take a claim about prompts in general.

The paper reports both: the permutation test, and a Wilcoxon signed-rank test over the triples.

`prompts firing` matters as much as the rate. Eleven of twelve means a prompt class. One of twelve means one odd prompt.

**Expect run-to-run variance.** These models sample at temperature 1.0, and the rate moves. Four runs of the identical configuration at `--runs 10` gave 31.7%, 40.0%, 43.3% and 31.7% for the same condition — a twelve-point spread from sampling alone. Use `--runs 20` or more for anything you intend to act on, read the confidence interval rather than the point estimate, and treat a difference between two conditions as real only when the intervals separate.

Correct-tool usage is far more stable than the decoy rate: across those same four runs it moved by under three points. If you want one number to track over time, use that.

## Provider quirks

Providers reject arguments in two ways: the SDK refuses a keyword outright, or the API returns a 400 saying it isn't supported for that model. Both are handled — the tool reads the remedy out of the error, applies it, and remembers it, printing one note when it does.

The one that affects your numbers: some OpenAI reasoning models refuse function tools on Chat Completions unless `reasoning_effort` is set to `none`. The tool sets it when the API asks for it and says so. **That is a deliberate configuration, not the API default, so report it alongside your results** — a model measured at minimum reasoning effort is not the same model measured at its default.

To pin it yourself, or to use a different value:

```python
Probe(model="gpt-...", tools=MY_TOOLS, reasoning_effort="low")
```

## Custom providers

Any callable taking `(prompt, tools)` and returning `(tool_names, status, error)`:

```python
def my_agent(prompt, tools):
    calls = my_framework.run(prompt, tools)
    return [c.name for c in calls], "OK", ""

Probe(model="internal-v3", tools=MY_TOOLS, provider=my_agent).run()
```

Use `model="mock"` to check your plumbing without spending anything — it needs no key, no SDK and no network, and it is the fastest way to see the output format before you point this at anything real.

## What this doesn't tell you

- **Single-turn only.** Invocation, not consequence. Whether a lookup resolves the entity, whether a fetched URL is correct, or whether an agent recovers after a tool returns something useless is untested here.
- **Invocation isn't error.** A decoy call is observable; whether it was wrong depends on what your deployment allows. Nothing here adjudicates individual calls.
- **Familiar isn't verified.** The "named, familiar" condition assumes the model can find the vendor's URL. The package doesn't check that it can.
- **Your stimulus set is the limit.** A dozen triples caps precision; the numbers reflect the prompts you wrote.
- **Not a security tool.** It measures a reliability failure that happens to have a security consequence. There's no adversary anywhere in this.

```bibtex
@inproceedings{douley2026ungrounded,
  author    = {Douley, Taran},
  title     = {Ungrounded: Referent-Dependent Routing Shifts in Tool-Using {LLM} Agents},
  booktitle = {Who Verifies the Agents? Workshop at NeurIPS 2026},
  year      = {2026}
}
```

Code and data are archived at [doi:10.5281/zenodo.21958705](https://doi.org/10.5281/zenodo.21958705), registered under the paper's earlier title.

## I'll run it for you

If wiring this into your stack isn't worth an hour, send me your tool schema and five representative requests and I'll run it and send back the report. taran@shroudlabs.io.

The open question is whether this survives contact with production tool catalogues, and I can't answer that from a synthetic ten-tool set. If you'd rather your results stayed private, say so and they will.

## Citing

MIT licensed. Issues and results from real catalogues are especially welcome — the open question is whether this survives contact with production tool catalogues, and I can't answer that alone.

Paper, per-trial data for all five studies, and the analysis that produced
them: [Taran-Douley/ungrounded-agents](https://github.com/Taran-Douley/ungrounded-agents)
