Metadata-Version: 2.4
Name: toolseen
Version: 0.1.0
Summary: Compare raw tool return values with what the harness passed downstream.
Author: toolseen contributors
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# toolseen

Your tool returned 50KB. Your harness forwarded only 9KB. toolseen shows what changed between the two.

A harness can truncate, filter, summarize, or alter a tool result before passing it downstream. toolseen compares the original value with the downstream value you measured.

![toolseen comparing a tool's raw return with what the harness passed downstream](https://raw.githubusercontent.com/yoruhenry/toolseen/main/assets/demo.gif)

That is the whole problem in eleven seconds. A fake tool returns ~50KB of JSON with `"delayed": true` in the middle. A fake harness keeps only the head and the tail. The agent checks whether the word `delayed` reached it, finds nothing, and reports that the order is on schedule.

Nothing errored. Nothing warned. The fake agent was answering from text that no longer contained the key fact, and `toolseen report` is what makes that visible.

## Install

```bash
pip install toolseen

# from a checkout
pip install -e ".[dev]"
```

Requires Python 3.10+. No runtime dependencies. Comparison is local and deterministic. No network calls or model API.

## Compare two values you already have

If you can get both values—two log lines, two captured payloads, or two request bodies—`compare()` is the whole API. For files, use the CLI below.

```python
from toolseen import compare

verdict = compare(original_text, seen_text)
print(verdict.kind)
```

Optional arguments: `threshold=64` (minimum dropped characters for drop/truncation kinds), `ignore_ws=False`.

```bash
toolseen compare original.txt seen.txt [--threshold 64] [--ignore-ws]
```

```text
CONTENT_DROPPED (middle)
original: 50,000 chars
seen:      9,000 chars
dropped:  41,000 chars
cause:    unknown
mode:     raw
keys not observed in seen (approximate): delayed, carrier_notes
```

## Instrument a Python harness

If you control the code that calls your tools, wrap the call instead of carrying the strings around.

```python
from toolseen import watch_call, saw

call_id, result = watch_call(check_order)      # records what the tool returned
delivered = my_harness(result)                 # whatever your pipeline does
saw(call_id, delivered)                        # records what went downstream
```

```bash
toolseen report
```

```text
check_order#1  CONTENT_DROPPED (middle)
original 50,115 -> seen 9,000
41,115 chars dropped
keys not observed in seen (approximate): delayed, items
```

By default, `toolseen report` reads the newest session under `.toolseen/` in the current working directory.

`report()` is available from Python too, returning `(call_id, Verdict)` pairs:

```python
from toolseen import report

for call_id, verdict in report():
    print(call_id, verdict.kind)
```

`watch_call` preserves the wrapped function's return value and exception behavior. A `str` return is recorded as-is; other allowed types are serialized with `json.dumps(..., sort_keys=True)`. Events go to `.toolseen/session-<uuid>.jsonl`, created `0700`/`0600` with a `.gitignore` alongside, and toolseen refuses to record when that is not the case.

Phase 2 requires `O_NOFOLLOW` and is unavailable on platforms that do not provide it. `compare()` and `toolseen compare` are unaffected.

## Run the demos

```bash
# the recording above
python examples/demo_instrumented.py
toolseen report

# file comparison, no instrumentation needed
python examples/demo_agent.py
toolseen compare examples/original.txt examples/seen.txt
```

These demos do **not** claim to model any commercial product's truncation rules.

## Exit codes

| Code | Meaning |
|------|---------|
| 0 | Every verdict is `OK` |
| 1 | One or more non-OK verdicts |
| 2 | Input / session / schema error |

## Verdict kinds

| Kind | Meaning (short) |
|------|-----------------|
| `OK` | Exact, JSON-semantic, or (optional) whitespace match |
| `EMPTY_SEEN` | Original non-empty-like; seen empty-like (`""`, `null`, `[]`, …) |
| `SUSPECTED_STRINGIFIED` | Seen fully matches closed patterns like `[object Object]` |
| `TRUNCATED` | Exactly one known truncation marker; pieces fit original |
| `CONTENT_DROPPED` | Large head/tail/middle drop without a marker; cause unknown |
| `SCHEMA_MISMATCH` | Both sides JSON objects; top-level key sets differ |
| `MODIFIED` | Any other difference (redaction insert, rewrite, …) |

`CONTENT_DROPPED` is **not** always truncation—it may be filtering or intentional removal.

## Scope

toolseen compares the two boundaries you provide or measure. It does not guarantee what the model ultimately received, and it does not auto-hook Codex, Claude Code, Gemini CLI, or any other harness.

## Limits

- If you pass the wrong two values, the verdict is still correct for those two values.
- `keys not observed in seen (approximate)` is a substring heuristic on top-level JSON keys, not proof a field was deleted.
- The capture layer records one process and does not share a session across processes.
- Phase 2 is intended for ordinary local development, not as a tamper-evident audit log.

## Security

Tool results can contain secrets, PII, or tokens, and session files hold what your tools returned. Keep `.toolseen/` and production payloads out of public commits and issues.

## License

MIT
