Metadata-Version: 2.5
Name: tokenaudit
Version: 0.1.0
Summary: Root-cause diagnosis of wasted LLM tokens, from the traces your agents already emit
Project-URL: Repository, https://github.com/yashmyeole/tokenaudit
Project-URL: Issues, https://github.com/yashmyeole/tokenaudit/issues
Author-email: Yash Manohar Yeole <yashmyeole@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: cost,diagnosis,llm,observability,opentelemetry,prompt-caching,tokens,waste
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.11
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pre-commit>=3.8; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: ruff>=0.6.9; extra == 'dev'
Description-Content-Type: text/markdown

# tokenaudit

**Root-cause diagnosis of wasted LLM tokens — from the traces your agents already emit.**

> Status: early development, pre-0.1. Interfaces will change.

Every LLM cost tool can tell you *how much* you spent and *who* spent it. Almost none can tell you *why a specific request wasted money*. The biggest lever on an LLM bill — prompt caching, up to a 10× discount on repeated input — fails silently: change one byte of a cached prefix, drop below a model's minimum cacheable size, let a timer expire, and the provider just bills you full price. No error, no warning. There are [documented](https://barndoor.ai/what-happens-to-your-llm-bill-when-prompt-caching-fails/) [cases](https://github.com/TauricResearch/TradingAgents/issues/750) of agents running at ~0% cache hit rate for months, at 30–40% unnecessary cost, because of a single line of prompt construction.

tokenaudit reads the traces agents already produce and, for each expensive request, answers four questions: **what was wasted, why, what it cost, and what to change.**

## Quickstart

    pip install tokenaudit
    tokenaudit diagnose --sample          # diagnose the bundled demo trace
    tokenaudit diagnose mytraces/ --html report.html

Zero runtime dependencies; reads OTLP/JSON files and OTel SDK JSONL. Try it
on this repo's real captured runs: `tokenaudit diagnose corpus/captured/runs`.

The plan, in one line each:

- Ingest standard traces — OpenTelemetry GenAI in all three of its cache-field naming generations, OpenInference, raw provider SDK logs — into one normalized accounting model.
- Run a library of deterministic detectors for the silent failure modes: prefix drift, sub-minimum cache blocks, model switches, TTL expiry, breakpoint and lookback limits, tool-schema churn and bloat, context bloat, retry storms.
- Price each finding against a counterfactual — what the bill would have been under the fix — as a bounded range, never a point guess.
- Publish per-detector precision and recall against a labeled corpus, so you know how much to trust each finding.

## Prior art

This space is crowded, and being precise about what already exists is the point of this section.

| Tool | What it does |
|---|---|
| [CacheLens](https://github.com/ChingEnLin/CacheLens) | Closest prior art. Dev-time SDK wrapper that meters cache hit rate and prefix stability per prompt layer in apps you instrument. Doesn't read existing traces, and prefix stability is its one diagnosis. |
| [Helicone](https://helicone.ai) | Observability proxy: cost and token accounting per user/session/feature, including cache token counts |
| [Langfuse](https://langfuse.com) | Open-source LLM engineering platform: tracing, evals, per-trace cost accounting |
| [Datadog LLM Observability](https://docs.datadoghq.com/llm_observability/) | Enterprise monitoring: per-request token and cost tracking, quality-issue clustering |
| [Braintrust](https://braintrust.dev) | Evals and observability; its Loop assistant root-causes *quality* failures |
| [LiteLLM](https://github.com/BerriAI/litellm) | Gateway with budgets, rate limits and spend tracking across providers |
| [ccusage](https://github.com/ryoppippi/ccusage) | CLI usage and cost reporting from Claude Code's local logs |
| [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/) | Platform-level caching, spend caps and alerts |

All of these answer "how much", and several now count cache tokens. The ones that do automatic root-cause analysis target response *quality*, not cost. The closest tool diagnoses one failure mode, for one integration style, going forward only. What none of them do is take the telemetry any agent already emitted — no code changes, any vendor — and say: *these 44 requests missed cache because the system prompt embeds a timestamp; here is the evidence, and here is what it cost.* That attribution step is this project.

If you know a tool that already does this, please open an issue — I would genuinely rather find out now.

## What tokenaudit is not

- **Not a dashboard.** The tools above already report spend well.
- **Not a proxy, gateway, or wrapper.** It reads traces after the fact and never sits in the request path, so it can't change what it measures and doesn't need your code.
- **Not an LLM reading your logs.** Verdicts come from explicit rules with unit tests and a published false-positive rate ([decision 001](DECISIONS.md)).
- **Not a savings guarantee.** Estimates are ranges with stated assumptions, and requests whose telemetry fails a plausibility check are flagged as unpriceable, not silently priced.

## Planned layout

    src/tokenaudit/
      ingest/          trace readers + schema normalization
      model/           normalized requests/sessions + token accounting
      detect/          the diagnosis engine (deterministic detectors)
      counterfactual/  what the bill would have been under a fix
      report/          CLI, JSON and HTML output
      ci/              cost-regression gate for pipelines

## Roadmap

- [x] Prior-art survey, positioning, project scaffold
- [x] Labeled trace corpus: synthetic known-answer fixtures + captured runs from real public agents
- [x] Trace reader and normalizer with a data-quality gate (bad telemetry is refused, not zero-filled)
- [x] Provider caching rulebook: minimums, TTL semantics, invalidation rules — versioned, with sources
- [x] Detectors, starting with prefix drift, sub-minimum blocks, and model/version switch
- [x] Evaluation harness: per-detector results published in [eval/results.md](eval/results.md)
- [x] `tokenaudit diagnose` CLI, HTML report, cache health score
- [ ] CI cost-regression mode: fail a build that tanks the cache hit rate
- [ ] Case studies on public agents, with reproduction commands

## Development

    python -m venv .venv && source .venv/bin/activate
    pip install -e ".[dev]"
    pre-commit install
    pytest

## License

Apache-2.0
