Metadata-Version: 2.4
Name: toolleash
Version: 0.1.1
Summary: Synthesize and enforce least-privilege tool-call policies from OpenTelemetry GenAI traces.
Project-URL: Homepage, https://github.com/Avanishk05/AgentProbe
Project-URL: Repository, https://github.com/Avanishk05/AgentProbe
Project-URL: Issues, https://github.com/Avanishk05/AgentProbe/issues
Project-URL: Changelog, https://github.com/Avanishk05/AgentProbe/blob/main/CHANGELOG.md
Author: ToolLeash contributors
License: Apache-2.0
License-File: LICENSE
Keywords: cedar,genai,least-privilege,opentelemetry,policy,rego
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: cedarpy<5,>=4.8.7
Requires-Dist: mcp<2,>=1.28
Requires-Dist: opentelemetry-api<2,>=1.43
Requires-Dist: opentelemetry-exporter-otlp<2,>=1.43
Requires-Dist: opentelemetry-proto<2,>=1.43
Requires-Dist: opentelemetry-sdk<2,>=1.43
Requires-Dist: pydantic<3,>=2.13
Requires-Dist: regopy<2,>=1.5.2
Requires-Dist: typer<1,>=0.26
Description-Content-Type: text/markdown

# ToolLeash

Synthesize and enforce **least-privilege tool-call policies** from OpenTelemetry GenAI traces.

ToolLeash ingests OTel GenAI `execute_tool` spans emitted by a real agent run, synthesizes a
deny-by-default policy that allows only observed behaviour (with inferred per-argument
constraints), emits it in multiple formats (Cedar, OPA/Rego, an MCP allow-list, or its own
canonical `policy-ir` JSON), and enforces it deterministically at the tool-call boundary. Mental
model: `iamlive` / AWS Access Analyzer, but for AI-agent tool calls.

> **Status: pre-1.0, under active construction.** Every claim in this README is backed by a test
> or a command you can run yourself — see `STATUS.md` for the exact test IDs and CI run per
> capability, `PLAN.md` for the roadmap, and `CLAUDE.md` for the engineering rules this project
> holds itself to. Nothing is claimed here that a test does not prove.

## Install

```bash
pip install .          # from a clone; no PyPI release yet
toolleash --help
```

Zero API keys, zero external binaries (no Node, no Docker) — every dependency is pip-installable.

## Quickstart

```bash
python examples/run_quickstart.py
```

One command, no traces to record by hand. It runs a real Python MCP server, records a handful of
real `execute_tool` spans, synthesizes a policy from them in-process, then enforces that policy on
the same live session: an allowed call proceeds, a policy-violating call (never seen during
training) is denied. See `examples/README.md` for what it does and why. This exact script is run
as a real CI integration test (`tests/integration/test_examples_quickstart.py`).

## CLI

```bash
toolleash record --listen 127.0.0.1:4318      # OTLP/HTTP receiver: capture a real staging run
toolleash inspect ./traces                    # sessions, tools, calls, argument-capture coverage
toolleash synthesize --traces ./traces --format cedar -o policy --strictness balanced
toolleash diff --policy policy-ir.json --traces ./new-traces    # day-2 drift report (always exits 0)
toolleash validate --policy policy-ir.json --traces ./ci-run    # CI gate: exits 1 on any denial
toolleash evaluate ./traces                   # self-graded / de-biased holdout table
```

Every command accepts `--json` for machine-readable output (all but `record`, which is a blocking
server, not a value-returning command). Formats available from `synthesize`: `cedar`, `rego`,
`mcp-allowlist`, `policy-ir` (Progent-JSON is not implemented — see Limitations).

## Library

```python
from toolleash.enforce.engine import PolicyEngine, ToolCallRequest
from toolleash.enforce.adapters import guarded_tool

engine = PolicyEngine(policy)                    # a PolicyIR from toolleash.policy.ir.build_policy_ir
decision = engine.decide(ToolCallRequest(agent="my-agent", tool="write_file", arguments={...}))

@guarded_tool(engine)                            # framework-agnostic: no framework imports
def write_file(path: str, content: str) -> None: ...
```

`PolicyEngine.guard`/`enforce` raise `PolicyViolation` on a policy-denied call in `enforce` mode;
`shadow` mode records what *would* have been blocked without blocking it — the safe rollout path.
See `src/toolleash/enforce/engine.py` and `examples/run_quickstart.py` for the full contract.

## Limitations

These are not boilerplate — read them before you rely on a synthesized policy.

- **Every enforcement mechanism here has only ever been probed by this project's own
  adversarial-reviewer, never an independent security researcher.** The traversal and
  injection bypasses fixed this phase (see `docs/THREAT_MODEL.md`'s summary table) were each
  found by this project deliberately widening its own search after the previous one — not by
  an external red team. "No known bypass" and "no bypass" are different claims; today this
  project can only make the first one. Treat every "Yes" in `docs/THREAT_MODEL.md`'s summary
  table as "this project's own tests say yes," not as an independently audited guarantee.
- **Progent-JSON is not implemented.** Its real DSL can't be pinned from code without cloning
  and running the actual Progent repo — outside this project's zero-asset guarantee, and
  inferring the schema from the paper instead was explicitly ruled out (honesty over feature
  count). So it is absent from `--format` entirely, not stubbed, not silently degraded.
  `cedar`, `rego`, `mcp-allowlist`, and `policy-ir` are the only real formats; see `PLAN.md`
  T-3.5 for the full locked decision.
- **Policies reflect *observed* behaviour, nothing more.** An under-exercised staging run yields
  an over-*restrictive* policy (legitimate calls the agent will make in production were never
  observed, so they're denied); a staging run that's too broad — say, one that happens to touch
  paths or hosts wider than production ever will — yields an over-*permissive* policy. ToolLeash
  cannot tell which case it's in from the traces alone; the recorded window's coverage is on you.
- **False-deny is a measured floor, not a product guarantee.** On the two available free-tier
  model-driven corpora (`gemini-flash-lite-latest`, `tests/fixtures/recorded/model_driven/`), the
  held-out envelope false-deny rate at ToolLeash's default (`balanced`) strictness measures
  **11.1%–14.9%**; at the tightest (`exact`/`tight`) presets it measures **24.6%–35.6%**.
  Reproduce it yourself: `toolleash evaluate tests/fixtures/recorded/model_driven/open` (or
  `.../shared`). These numbers come from three toy agent personas over two toy tool suites — they
  are expected to improve with richer, more diverse training data, not a bound on what ToolLeash
  can achieve in your environment.
- **Permissiveness figures are lower bounds.** `toolleash evaluate`'s "security-vacuous" /
  permissiveness column measures how much of a *sampled foreign corpus* a constraint admits; a
  thin corpus makes a wide-open constraint look artificially tight. A low permissiveness score is
  "not yet shown to be permissive," never "shown to be tight" — see the CLI's own `LOWER BOUND`
  caption on every `evaluate` run.
- **Cedar output is a conservative *under*-approximation, not equivalent to the IR.** The Cedar
  emitter cannot express `string_length`, `char_class_regex`, or `url_constraint` families; a rule
  containing one of these is **omitted entirely** (fail-closed — deny-by-default then denies those
  calls, never silently rendered as accept-any). A Cedar user therefore gets a **strictly stricter**
  policy than the canonical IR and MUST check which rules were dropped: `synthesize` prints
  `OMITTED: ...` to stderr and the emitted `.cedar` file carries `// OMITTED` header comments;
  reproduce with `toolleash synthesize --traces tests/fixtures/recorded/filesystem --format cedar
  -o /tmp/policy --strictness balanced`. Rego is more expressive (only `url_constraint` is its own
  gap) but is not equivalent to the IR either — always check `omissions()` / the CLI output, for
  either format, before trusting the emitted policy's coverage.
- **Tool-argument capture is opt-in, and inference is only as good as what it's given.** If a
  traced agent's instrumentation does not capture call arguments (`arguments=None` on the
  recorded span), ToolLeash has nothing to constrain — the synthesized policy degrades to a
  tool-name-only allow-list (any arguments accepted for an observed tool). `toolleash inspect`
  prints a loud `WARNING: ... content capture appears OFF` when this is detected; check that
  warning before trusting a policy's argument-level constraints.
- **Dependency upper bounds cap majors, not minors — the versions below are what was actually
  verified, not a guarantee every version in range behaves identically.** Every dependency is
  capped at its next major (ADR-0020) after a real TestPyPI round-trip resolved `mcp>=1.28` (no
  ceiling) to a genuinely breaking `mcp==2.0.0` on a fresh install — a class of bug no gate run
  ever exercises, since CI installs from the pinned `uv.lock`, never a fresh resolve. Last verified
  against: `cedarpy` 4.8.7, `regopy` 1.5.2, `mcp` 1.28.1/1.29.0, `pydantic` 2.13.4, `typer`
  0.26.8/0.27.0, `opentelemetry-*` 1.43.0/1.44.0. **`regopy` is the sharpest risk of this group**:
  its RE2 regex-matching semantics are load-bearing for the char-class containment guarantee
  (ADR-0015), and a future major changing that would not crash the way `mcp` did — it could
  silently change which policies validate. A ceiling is only ever raised after re-running the full
  round-trip against the new major (ADR-0020), never preemptively.
- **The OTel GenAI semantic conventions this project reads are Development/experimental status**
  (the `_incubating` namespace) — attribute names may still change upstream. ToolLeash pins the
  exact version it targets in `docs/specs/otel_genai_semconv.md` and `src/toolleash/semconv.py`;
  a future semconv release could require a re-pin.

See `docs/THREAT_MODEL.md` for what ToolLeash defends against and — just as important — what it
explicitly does **not**.

## Requirements

Python ≥ 3.11. Zero API keys, zero external binaries, zero Node, zero Docker — every dependency
is pip-installable (see `PLAN.md` "Zero-asset guarantee").

## Developing

```bash
uv sync                          # create the environment from pyproject + uv.lock
uv run python scripts/check.py   # run the full quality gate (the definition of "green")
```

`make check` is equivalent where `make` is available; both just invoke `scripts/check.py`.

## License

Apache-2.0. See `LICENSE`.
