Metadata-Version: 2.4
Name: trace-to-fixture
Version: 0.1.0
Summary: Turn real agent runs into safe, small, repeatable tests: scrub private data, keep one example of each behaviour, write pytest fixtures for faultgate-zero.
Author: Faultline-lab
License-Expression: MIT
Project-URL: Homepage, https://github.com/Faultline-lab/trace-to-fixture
Keywords: ai,agents,testing,fixtures,opentelemetry,pytest
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: faultgate<0.5,>=0.4.2
Requires-Dist: faultgate-zero<0.2,>=0.1
Requires-Dist: pytest>=8
Provides-Extra: pii
Requires-Dist: faultgate[pii]<0.5,>=0.4.2; extra == "pii"
Provides-Extra: dev
Requires-Dist: pytest-xdist>=3; extra == "dev"
Dynamic: license-file

# Trace-to-Fixture

Turn real agent runs into safe, small, repeatable tests.

Read a trace export, mask the private data it can detect, keep one example of each distinct behaviour, and write
tests that run in CI with [faultgate-zero](https://github.com/Faultline-lab/faultgate-zero). The offline tests pass on
day one with no model calls. The live tests re-run your agent on each real prompt and check it.

```
T2F_SALT=<a secret> trace-to-fixture build export.json --out tests/agent_fixtures
```

Free and MIT-licensed. Runs on your machine and sends nothing anywhere.

> **What it does and does not do.** It *masks the private data its detectors can find* (secrets, emails, US SSNs,
> card numbers, IP addresses, phone numbers; names and places with the `pii` extra), plus anything you list in a
> deny file or as a sensitive key. It is not an anonymisation or compliance product, and detectors miss things such as
> internal customer ids. Always run `scan`, read what it found, and review the output before you commit it.

## Install

```bash
pip install trace-to-fixture            # needs Python 3.11+
pip install 'trace-to-fixture[pii]'     # adds Presidio for names and places
```

It builds on `faultgate` (0.4.2 or newer) (format loaders, private-data detectors) and `faultgate-zero` (the checks).

## Quickstart (5 minutes)

The `examples/` folder holds a small refund agent, a "production" export of three of its runs, and a policy.

```bash
git clone https://github.com/Faultline-lab/trace-to-fixture && cd trace-to-fixture
pip install -e .
cd examples

# 1. See what is in the export. Nothing is written (scan needs no salt).
trace-to-fixture scan export.otlp.json

# 2. Build the fixture set. build needs a salt: a secret in real use, never committed.
#    On Windows PowerShell use  $env:T2F_SALT = "quickstart-demo-salt"  or pass --salt-file FILE.
export T2F_SALT="quickstart-demo-salt"
trace-to-fixture build export.otlp.json --out ../tests_out --policy policy.json --agent refund_agent:run

# 3. Run the offline tests. No model calls, free.
pytest ../tests_out
```

`tests_out/` now holds `manifest.json`, `fixtures/*.otlp.json`, `test_agent_fixtures.py` and `substitutions.json`.

### Live replay

Live tests re-run your agent on each real prompt, so they need a test-environment value for every placeholder:

```bash
pytest ../tests_out -m agent_live -rs      # skipped: "needs substitutions for <EMAIL_b1beab65>, <PHONE_29a793ca>"
```

Put test data (never real values) in `substitutions.json`, then run again:

```json
{ "<EMAIL_b1beab65>": "test.user1@example.com", "<PHONE_29a793ca>": "+1 555 0100" }
```

The known failure (a $900 refund over the $500 limit) now **xfails**: it still fails like production. Fix the bug
(`FIXED = True` in `refund_agent.py`) and the test fails with:

```
f-xxxxxxxx no longer fails. Promote it to a guard:
  trace-to-fixture promote ../tests_out f-xxxxxxxx
```

After `promote`, the fixture is a guard: if the bug ever comes back the test fails with the policy reason.
Live tests run only with `-m agent_live` or `T2F_LIVE=1`, so a plain `pytest` never spends tokens.

## Commands

| Command | What it does | Writes |
|---|---|---|
| `scan INPUT` | Format, precheck, counts of detected values by kind and by field, dropped attribute keys. | nothing |
| `build INPUT --out DIR` | Load, precheck, scrub, prove verdicts are unchanged, group, select, write, leak scan. | `DIR` |
| `verify DIR` | Leak scan of every file, hash checks, loading in faultgate-zero and faultgate, snapshot verdicts. | nothing |
| `explain INPUT --trace ID` | One trace: changed fields (path, kind, lengths, never values), raw vs scrubbed verdicts, signature, why kept. | nothing |
| `promote DIR ID` | Change a fixture from `known_failure` to `guard`. | `manifest.json` |

Options: `--format` (default `auto`; OTLP, OpenInference, Langfuse, LangSmith, Braintrust), `--policy FILE`
(faultgate-zero policy format), `--checks a,b,c`, `--agent module:function`, `--per-group N` (1), `--max-clean N` (20),
`--pii builtin|presidio`, `--deny-literals FILE`, `--sensitive-key NAME` (repeatable), `--salt-file FILE` or `T2F_SALT`,
`--drop-mismatched`, `--bundle FILE`, `-v` / `T2F_DEBUG=1` (stage counts and timings on stderr, never values).

Exit codes: `0` success, `1` a problem was found (a leak, a verdict mismatch, snapshot drift), `2` a usage or input
error (a bad file, a missing salt, precheck errors).

## How it works

1. **Load** the export with faultgate's readers; precheck errors stop the build.
2. **Scrub.** Each detected value becomes `<KIND_xxxxxxxx>`, an HMAC of the exact value under your secret salt: the same
   value always gets the same placeholder, so checks that compare values keep working. Attributes the checks never read
   are dropped; trace and span ids are remapped; time is rebased. JSON inside strings (up to two levels) is walked.
3. **Prove nothing changed.** Every check runs on each trace before and after scrubbing; verdicts must match.
4. **Select.** Runs with the same behaviour (tool path, fired checks, outcome, tool errors, size) form a group. Every
   group where a check fired is kept; the largest clean groups fill the rest. Selection is deterministic.
5. **Write** fixtures, a manifest, a generated pytest file, and a leak scan of what was written.

Same input, same salt and same options give byte-identical output. Use one stable salt per project (a different salt
changes every placeholder and fixture id). `manifest.json` records only a fingerprint of the salt.

## Privacy rules

- Raw values live only in memory: never in files, logs, exceptions or the manifest.
- The salt is a secret. Keep it in a secret store or CI secret, never in git. One salt per customer audit.
- `substitutions.json` holds test data only.
- If a detector misses something, add it to `--deny-literals` (one literal per line) or `--sensitive-key`.

## Develop

```bash
pip install -e '.[dev]'
pytest                   # unit tests
python evals/run.py      # acceptance table (add --perf for the 5,000-trace speed row)
```

## Limits

Multi-turn conversations are kept as offline fixtures only (`replayable: false`). Tool responses are not mocked:
live replay calls your own tools in your own test environment. Input is an export file; there is no vendor API puller.
