Metadata-Version: 2.5
Name: tracelint
Version: 0.10.0
Summary: A linter for agent runs — reads the execution trace and flags structural bugs, deterministically.
Project-URL: Homepage, https://tracelint.com
Project-URL: Repository, https://github.com/AshwinUgale/tracelint
Project-URL: Issues, https://github.com/AshwinUgale/tracelint/issues
Project-URL: Changelog, https://github.com/AshwinUgale/tracelint/blob/main/CHANGELOG.md
Author: Ashwin Ugale
License-Expression: MIT
License-File: LICENSE
Keywords: agents,linter,llm,observability,reliability,static-analysis,tool-calling,traces
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: jsonschema>=4.0
Provides-Extra: capture
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'capture'
Provides-Extra: capture-crewai
Requires-Dist: openinference-instrumentation-crewai>=0.1; extra == 'capture-crewai'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'capture-crewai'
Provides-Extra: capture-langchain
Requires-Dist: openinference-instrumentation-langchain>=0.1; extra == 'capture-langchain'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'capture-langchain'
Provides-Extra: capture-smolagents
Requires-Dist: openinference-instrumentation-smolagents>=0.1; extra == 'capture-smolagents'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'capture-smolagents'
Provides-Extra: dev
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: langfuse
Requires-Dist: langfuse>=3.0; extra == 'langfuse'
Provides-Extra: real-agent
Requires-Dist: openai>=1.0; extra == 'real-agent'
Description-Content-Type: text/markdown

# tracelint

> **tracelint checks an agent's execution trace for structural defects, locally or in CI.** It can
> read trace files directly or import traces from your existing observability platform.

It reads what a tool-calling agent *actually did* — the execution trace — and flags structural bugs
deterministically, with the exact evidence and a CI exit code. It runs *after* the run, on the
trace, not on your code, and **no second model ever judges it**. Model-as-judge detection of these
defects is unreliable (published trace-error benchmarks show low localization accuracy); many of
them are *structurally decidable* and need no judge — that is the entire premise of the tool.

**Website: [tracelint.com](https://tracelint.com)** — what it catches, how it works, and how to
wire it into CI.
**[See a live demo report](https://ashwinugale.github.io/tracelint/demo/)** — the constructed
validation suite (one planted instance of every defect, clean controls, and legitimate-but-suspicious
cases) plus the robust-vs-buggy recovery scorecard, generated by `tracelint demo`.

## Quick start — capture → lint → CI

The core loop is the whole product:

```
run agent in tests → capture the trace → run tracelint → fail CI on provable defects
```

**1. Capture a trace.** Most people don't have a trace file yet, so `tracelint.capture` records one.
It wraps your framework's stock OpenInference instrumentation, so the captured file lints with no
extra conversion:

```bash
pip install "tracelint[capture-smolagents]"   # or [capture-langchain] / [capture-crewai]
```

```python
from tracelint.capture import capture

with capture("trace.json", framework="smolagents"):
    agent.run("...")            # your agent, unchanged
```

If your app already instruments the framework (a Phoenix or Langfuse setup), `capture` leaves that
tracing alone and records from the global tracer provider, which `phoenix.otel.register()` sets. A
capture that records nothing raises and says why, instead of writing an empty trace.

**2. Lint it.**

```bash
tracelint check trace.json --format openinference
```

Exit codes: `0` clean · `2` a structurally-provable defect (`hard_defect`) · `3` an input error:
a missing or unreadable file, an empty one, or input that doesn't match `--format` (the error
names the format it looks like), so a misconfigured step never passes having checked nothing.
Heuristic candidates never fail CI on their own; a suppression (a rule that couldn't run) is
disclosed but is not a defect.

**3. Fail CI on a defect.** `check` returns `2` on a provable defect, so it gates a build directly:

```yaml
# .github/workflows/agent-tests.yml
      - run: pip install tracelint
      - run: tracelint check trace.json --format openinference   # exit 2 fails the job
```

See [Add to CI](#add-to-ci) for the ready-made GitHub Action, SARIF code-scanning, and pre-commit.

> The in-process capture helper covers the three in-process frameworks — **smolagents**,
> **LangGraph / LangChain**, and **CrewAI**. Langflow is a running product, not a library you call,
> so it uses the
> [OTel-to-file recipe](https://github.com/AshwinUgale/tracelint/blob/main/docs/integrations/langflow.md)
> instead (point its export at a file, then `check` it). Already collect traces elsewhere? Any
> [supported format](#supported-formats) works with no capture step.

Prefer to see it work first, with no API key or agent? The keyless demo runs a validation suite and
recovery scorecard end to end:

```bash
pip install tracelint
tracelint demo --html demo.html
```

## Limitations (read first)

1. Deterministic rules catch **structural** defects, not whether the final answer was correct.
2. Hallucinated-argument, loop, and redundant-call findings are **candidates** unless structurally
   proven — legitimate value transforms and intentional retries can trip them; each is shown with
   its evidence for human review, never asserted as a verdict. High-confidence hallucination
   detection requires the tool schema to declare field origins (`x-value-origin`).
3. The recovery scorecard needs labeled task outcomes (success oracles); without them it measures
   **behavioral** recovery only ("did not crash"), a weaker claim than correctness.
4. A trace is only as complete as its instrumentation. A rule whose required field is missing is
   **suppressed with a stated reason** — `tracelint` never lints a partial trace as if complete.
   That includes a tool call's arguments: when the trace records them redacted, positionally, or
   as a bare value, and doesn't hold the model's own call either, they are *unknown* (never read
   as `{}`), and the checks that need them report those calls as not checked.

## Supported formats

`check` reads native tracelint JSON by default; `--format` points it straight at the trace your
stack already emits — no manual schema conversion:

```bash
tracelint check trace.json    --format native         # canonical tracelint JSON (default)
tracelint check spans.json    --format openinference   # OTel/OpenInference: Phoenix, OTLP, TRAIL
tracelint check messages.json --format openai          # an OpenAI chat message list
tracelint check trace.json    --format langfuse        # a Langfuse trace export
tracelint check run.json      --format langsmith       # a LangSmith run tree export
```

**OpenTelemetry / [OpenInference](https://github.com/Arize-ai/openinference) is a *format*, not a
platform** — a set of OTel span conventions. So `--format openinference` works from *any*
instrumentation that emits them (Arize Phoenix, OpenLLMetry, Langfuse-via-OTel, datasets like TRAIL)
with no account, no platform, and no vendor lock — and it's the format the capture helper writes.
One shared adapter reaches the whole ecosystem instead of one vendor.

Most rules need no tool schemas, so this works keyless; add `--tools tools.json` to light up the
schema-dependent rules (R1, and R3's high-confidence tier). Don't have one? `tracelint init
spans.json --format openinference -o tools.json` bootstraps a starter contract from the trace —
schemas discovered where the telemetry carries them, behavior fields left as placeholders to review.
A multi-trace input (a JSON array, an OTLP export, or a `.jsonl` file) fans out to one report per
run. Spans are grouped into runs by trace id, so a `.jsonl` file can hold one span per line
(Phoenix's `to_json(..., lines=True)`, an OTel file exporter), whole runs, or export batches, and a
run split across files is linted whole. From the library:

```python
from tracelint import lint_otel_traces

for report in lint_otel_traces(spans):   # spans: your OpenInference span export (a list of dicts)
    print(report.exit_code)              # 0 or 2, one report per trace
```

(`lint_otel_trace`, singular, lints one run's spans and raises if they span several traces.)

## The rules

| Rule | Finding | Tiers |
|------|---------|-------|
| R1 | schema violation — args fail the tool's JSON Schema | `hard_defect` |
| R2a | tool returned an error | `hard_event` (structured signal) / `candidate` (heuristic) |
| R2b | a value only a failed result supplied, used by a later side-effecting call | `hard_defect` / `candidate` |
| R3 | hallucinated argument — value not derivable from provenance | `candidate`; `hard_defect` if the field is annotated `provided` |
| R4 | loop — N identical no-progress calls (polls/retries excluded) | `candidate` |
| R5 | redundant call — identical call + identical result, no mutation between | `candidate` |
| R6 | malformed arguments — tool-call arguments that are not valid JSON | `hard_defect` if the model emitted them; `candidate` in a tool's own record |
| R7 | unknown tool — a call to a tool absent from the declared toolset (possible hallucinated tool) | `candidate` |
| R8 | duplicate side effect — a non-idempotent side effect repeated with the same args after it succeeded | `hard_event`; `candidate` if the first result is unknown |

`hard_event` and `hard_defect` are orthogonal to the finding kind: a tool-error event is a
`hard_event` from a structured signal (the span's or run's own error status, a LangChain
`ToolMessage` with `status: "error"`, or a non-empty `error` field in the result) but a `candidate`
from a convention or an exception-like string. Fields inside a result, such as `status` or
`status_code`, are the tool's data: a link checker reports `status_code: 404` for a page it checked
successfully. They stay candidates until `failure_when` declares what failure looks like.

R2b follows the data, not repeated text. A failed result often echoes its inputs, so a value counts
only when nothing else the agent saw supplied it: an order id the user gave is not from the failure,
even when the error repeats it. Retrying the failed tool is handling the error, not misusing it. If
a call the value was passed to returns it again (a lookup that may have confirmed it), the use is a
`candidate`.

## Input format

A native trace is a JSON object (`.json`, or `.jsonl` for many):

```json
{
  "run_id": "run-1",
  "steps": [
    {"type": "message", "role": "user", "content": "cancel order 4521 if it hasn't shipped"},
    {"type": "tool_call", "call_id": "c1", "name": "get_order_status", "args": {"order_id": "4521"}},
    {"type": "tool_result", "call_id": "c1", "content": {"status": "processing"}, "status": "ok"},
    {"type": "tool_call", "call_id": "c2", "name": "cancel_order",
     "args": {"order_id": "4521", "reason": "not_shipped"}}
  ],
  "final": "Order 4521 has been cancelled."
}
```

`tools.json` supplies the ground truth the rules check against:

```json
{
  "tools": {
    "cancel_order": {
      "schema": {"type": "object", "properties": {"order_id": {"type": "string"}},
                 "required": ["order_id"]},
      "metadata": {"side_effecting": true}
    }
  }
}
```

A tool can also declare **what failure looks like** in its result, so a domain failure returned as
a transport success (HTTP 200 carrying `{"status": "declined"}`) is caught structurally instead of
slipping through:

```json
{
  "tools": {
    "charge_card": {
      "metadata": {
        "side_effecting": true,
        "failure_when": {"pointer": "/status", "in": ["declined", "failed"]}
      }
    }
  }
}
```

`failure_when` is a JSON Pointer into the result plus a match (`in` / `equals` / `exists`); a match
is a structured error for R2 (feeding R2a and, on reuse into a side-effecting call, R2b). A
side-effecting tool with **no** `failure_when` and an unclassifiable result is *suppressed with a
reason* — never counted as a clean pass.

The rules run against **one canonical trace schema**; a thin **adapter** translates each source's
format into it, so the rules never change: `from_openai_messages`, `from_langfuse_trace`,
`from_langsmith_run`, and `from_otel_spans`. The adapters are validated against live data, not just
the spec — `from_otel_spans` on real [TRAIL](https://huggingface.co/datasets/PatronusAI/TRAIL)
benchmark traces, where tracelint deterministically localized real tool errors, a malformed call,
and excessive-retry loops with no model in the loop. Real exports vary, so a new source may need a
small adapter tweak — and when a field a rule needs is absent, that rule **suppresses** (says so)
rather than guessing, so an unhandled quirk degrades safely.

## Integrations & recipes

tracelint reads the telemetry your stack already emits — one shared adapter reaches the whole
ecosystem. [`docs/integrations/`](https://github.com/AshwinUgale/tracelint/blob/main/docs/integrations/README.md)
has short, reproducible one-pagers, each validated on a **real captured trace**:

- **Frameworks (capture a trace in a test):**
  [smolagents](https://github.com/AshwinUgale/tracelint/blob/main/docs/integrations/smolagents.md) ·
  [LangGraph / LangChain](https://github.com/AshwinUgale/tracelint/blob/main/docs/integrations/langgraph.md) ·
  [CrewAI](https://github.com/AshwinUgale/tracelint/blob/main/docs/integrations/crewai.md) —
  each captures via the helper above and lints via `--format openinference` with no
  framework-specific code. [Langflow](https://github.com/AshwinUgale/tracelint/blob/main/docs/integrations/langflow.md)
  is a running product, so it uses the OTel-to-file recipe instead of the in-process helper.
- **Platforms (already have traces?):** the always-works baseline is to **export the trace to a file
  and `check` it** —
  [Arize Phoenix](https://github.com/AshwinUgale/tracelint/blob/main/docs/integrations/phoenix.md) ·
  [Langfuse](https://github.com/AshwinUgale/tracelint/blob/main/docs/integrations/langfuse.md) ·
  [LangSmith](https://github.com/AshwinUgale/tracelint/blob/main/docs/integrations/langsmith.md) ·
  [OpenLLMetry / Traceloop](https://github.com/AshwinUgale/tracelint/blob/main/docs/integrations/otel.md) ·
  [OpenAI / ShareGPT message lists](https://github.com/AshwinUgale/tracelint/blob/main/docs/integrations/openai.md).

For Langfuse, `pull` is convenience sugar on top of that baseline — it fetches a trace straight to a
file, so `pull` → `check` composes and the file doubles as a saved fixture:

```bash
pip install "tracelint[langfuse]"                 # reads your LANGFUSE_* env vars
tracelint langfuse pull <trace-id> -o trace.json   # writes native tracelint JSON
tracelint check trace.json                          # lint it (native is the default format)
```

These are compatibility validations on real traces, not benchmarks or endorsements.

Straight from a running [Arize Phoenix](https://phoenix.arize.com) instance, from the library:

```python
from phoenix.client import Client
from tracelint import lint_otel_traces

spans = Client().spans.get_spans_dataframe(project_name="my-agent")
for report in lint_otel_traces(spans.to_dict("records")):   # one report per run
    print(report.exit_code)
```

Both Phoenix shapes are handled: the span-export JSON (top-level `span_kind`) and the
`get_spans_dataframe()` records (attributes as `attributes.*` columns, message lists nested).

## Add to CI

`tracelint check` returns exit `2` on a structurally-provable defect, so it gates a build directly.
Point it at the traces your agent test job already produces — a defect fails the job; heuristic
candidates never do.

**GitHub Actions** — the ready-made action:

```yaml
name: lint-agent-traces
on: [push, pull_request]
jobs:
  tracelint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # ... your step that runs the agent and writes traces to ./traces ...
      - uses: AshwinUgale/tracelint@v0.10.0
        with:
          traces: "traces/*.jsonl"
          format: "openinference"     # or native / openai / langfuse / langsmith
          tools: "tools.json"          # optional — lights up R1, R3, R2 predicates
```

**Any CI, without the action** — it's one pip install and one command:

```bash
pip install tracelint
tracelint check traces/*.jsonl --format openinference --tools tools.json
```

**GitHub code scanning (SARIF)** — surface findings in the repo's *Security → Code scanning* tab
and as inline pull-request annotations. `--sarif` writes a SARIF 2.1.0 file (mapping `hard_defect`
→ `error`, `hard_event` → `warning`, `candidate` → `note`). It's written *before* the exit-`2`
gate, so an `if: always()` upload step runs even when a defect fails the job:

```yaml
      - run: tracelint check traces/*.jsonl --format openinference --tools tools.json --sarif tracelint.sarif
      - uses: github/codeql-action/upload-sarif@v3
        if: always()          # upload even when tracelint exits 2 on a hard_defect
        with:
          sarif_file: tracelint.sarif
```

**pre-commit** — lint only the trace files a commit touches:

```yaml
repos:
  - repo: https://github.com/AshwinUgale/tracelint
    rev: v0.10.0
    hooks:
      - id: tracelint
        files: ^traces/.*\.jsonl$
        args: ["--format", "openinference", "--tools", "tools.json"]
```

## Library

```python
from tracelint import ToolRegistry, default_rules, lint_trace, load_source

registry = ToolRegistry.load("tools.json")
for trace in load_source("trace.json", "openinference"):   # or native / openai / langfuse / langsmith
    report = lint_trace(trace, default_rules(), registry)
    print(report.exit_code)      # 0 or 2
    for f in report.active_findings:
        print(f.rule, f.tier.value, f.summary)
```

`load_source` returns every run in the file (a span export can hold several), in any supported
format. `Trace.load` reads only tracelint's native trace format.

See `examples/lint_openinference_phoenix.py` for an offline, keyless end-to-end run (Phoenix-shaped
spans → findings, with and without a tool registry).

## Advanced

These layers sit around the core loop and are opt-in.

**Native test assertion (pytest).** Opt in with `pytest_plugins = ["tracelint.pytest_plugin"]` in
your `conftest.py`; the `trace_capture` fixture then captures a run and lints it inline, so a hard
defect fails the test (candidates never do):

```python
def test_agent(trace_capture):
    with trace_capture(framework="smolagents"):
        agent.run("refund order A100")   # a hard defect in the trace fails this test
```

Pass `assert_clean=False` to inspect `cap.report` yourself instead of auto-failing (`cap.reports`
has one report per run when the block runs the agent more than once). A capture with nothing to
lint fails the test rather than passing as clean. It's a thin wrapper over the capture helper, so
the same `capture-<framework>` extra applies.

**Recovery scorecard & fault injection.** Measure how an agent behaves under injected faults, scored
against deterministic success oracles:

```bash
tracelint scorecard --demo --faults timeout,error,rate_limit --runs 5
```

The baseline must satisfy the oracle first (else recovery is not measured). Each fault type reports
a correctness-recovery rate with a Wilson confidence interval; with no oracle it falls back to
behavioral recovery, labeled as weaker.

**Write findings back into Langfuse.** Beyond reading a trace, tracelint can write its verdict back
into Langfuse's own **Score** model so it appears beside your normal evals. This is the one part
that overlaps with what a platform does itself, so it's an advanced recipe, not the primary path —
the baseline (export or `pull` a trace, then `check` it) always works without it:

```bash
pip install "tracelint[langfuse]"        # v3 SDK; reads your LANGFUSE_* env vars

tracelint langfuse check --trace <trace-id> --tools tools.json               # read-only: prints the plan
tracelint langfuse check --trace <trace-id> --tools tools.json --write-back  # writes Scores
```

With `--write-back`, tracelint writes `tracelint.passed` (BOOLEAN) and `tracelint.hard_defects`
(NUMERIC) at the trace level, plus one score per *certain* finding attached to the exact offending
observation. Heuristic candidates are shown in the terminal but never written back. Scores are keyed
by a stable finding id, so re-running updates them in place rather than duplicating.

## Development

```bash
python -m pytest
ruff check src tests
```

The core is dependency-light (`jsonschema` + stdlib) and the whole test suite is deterministic and
offline. A real OpenAI trace-generating agent lives behind the opt-in `[real-agent]` extra and is
never part of the linter. Python 3.10–3.12.
