Metadata-Version: 2.4
Name: cachecheck
Version: 0.2.0
Summary: Find token accounting that silently undercounts Anthropic's prompt cache
Project-URL: Homepage, https://github.com/arthi-arumugam-git/cachecheck
Project-URL: Findings, https://github.com/arthi-arumugam-git/wrong-numbers
Author: Arthi Arumugam
License: MIT License
        
        Copyright (c) 2026 Arthi Arumugam
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: anthropic,billing,claude,cost,linter,llm,observability,prompt-cache,static-analysis,tokens
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# cachecheck

**Five agent frameworks silently undercounted Anthropic's prompt cache. This finds it in your repo in one command.**

```bash
uvx cachecheck .
```

No dependencies. Exit code 1 when it finds something, so it drops straight into CI.

## The defect

Anthropic reports `input_tokens` **net of the prompt cache**, and bills `cache_read_input_tokens` and `cache_creation_input_tokens` separately on top. It does not send a total.

So this line, which appears in a lot of shipped code, is wrong:

```python
total = usage.input_tokens + usage.output_tokens
```

Every token served from cache falls out of the number. In a voice agent that re-reads its history each turn, a turn billed for 2,100 tokens with 2,000 cached reports 150. Nothing raises, the number looks plausible, and anything costing off it under-reports.

## Where this came from

Each rule is a defect found by hand in a shipped library, every one with a pull request and a test that fails without the fix:

| Framework | Pull request | State |
|---|---|---|
| Pipecat (Daily) | [#5163](https://github.com/pipecat-ai/pipecat/pull/5163) | merged |
| LiveKit Agents | [#6663](https://github.com/livekit/agents/pull/6663) | merged, by LiveKit's co-founder |
| Haystack (deepset) | [#3717](https://github.com/deepset-ai/haystack-core-integrations/pull/3717) | merged |
| mcp-use | [#2127](https://github.com/mcp-use/mcp-use/pull/2127) | open |
| LlamaIndex | [#22548](https://github.com/run-llama/llama_index/pull/22548) | closed, subsystem being retired |

The full write-up of each, with the wrong number and the right one, is at [wrong-numbers](https://github.com/arthi-arumugam-git/wrong-numbers).

## How many of those it actually catches

Four of the five. That is measured, not asserted: each framework's source was pulled at the parent of its fix commit and scanned, and the fixed version was scanned too.

| | pre-fix | post-fix |
|---|---|---|
| Pipecat | caught | silent |
| LiveKit Agents | caught | silent |
| Haystack | caught | silent |
| mcp-use | caught | silent |
| LlamaIndex | **missed** | n/a |

LlamaIndex's defect is in provider-agnostic core code that names no provider, and the gate requires one. That is the price of not reporting every OpenAI file in a repository, and it is a real miss rather than a rounding error.

The first published version found **two** of the five. It missed Pipecat, the defect this is named after, and it reported LiveKit's merged fix as a defect. Both are fixed. If you are reading this because a badge told you the tool was thorough, the numbers above are the ones to trust.

On the three repositories at HEAD, where every one of these fixes has landed and the answer should be silence, it reports 2 findings across 2,380 files. Before these changes it reported 14, of which all 14 were wrong.

## What it checks

**`total-excludes-cache`** · a total of input plus output that leaves the cache out, in a file that reads the cache counters elsewhere. The breakdown is right and only the total is wrong, which is far harder to spot than ignoring the cache outright. This is Pipecat's shape, and it is the most common one.

**`net-input-as-gross`** · assigns `input_tokens` to `prompt_tokens`. Those names do not mean the same thing: `prompt_tokens` includes the cached tokens and `input_tokens` is net of them, so the cache is lost in the rename with no arithmetic anywhere to give it away. This is Haystack's shape.

**`cache-blind-total`** · a total of input plus output, in a file that never reads `cache_read_input_tokens`. Those tokens are billed and are not inside `input_tokens`.

**`cache-read-never-read`** · computes with `input_tokens` and never reads the cache counter at all, so every cached token is invisible to the accounting.

**`cache-creation-ignored`** · reads cache reads but not cache writes. Cache writes are billed **above** the base rate, so they cost more per token than the ones they save. Bedrock calls this field `cacheWriteInputTokens`.

## What it deliberately does not flag

OpenAI reports `cached_tokens` **inside** `prompt_tokens`. Adding it there would double count, so a file handling OpenAI-shaped usage is correct as written and is left alone.

Comments, docstrings and type declarations name these fields without computing anything. An earlier version flagged `inputTokens?: number;` and a docstring showing sample output. Rule 2 now requires evidence of arithmetic, not a mention.

Test files build fixtures rather than billing anyone, and are skipped unless you pass `--tests`.

A checker that cries wolf on correct code gets switched off, and then it catches nothing at all.

## Usage

```bash
cachecheck .                 # human readable
cachecheck . --json          # machine readable
cachecheck . --quiet         # exit code only, for CI
cachecheck . --tests         # include test files
cachecheck path/to/file.py   # a single file
```

Scans `.py`, `.ts`, `.tsx`, `.js`, `.mjs`, `.jsx`, `.go`, `.rb`, `.java`, `.kt`.

## GitHub Action

```yaml
- uses: arthi-arumugam-git/cachecheck@v1
  with:
    path: .
```

Fails the job when anything is found. Set `fail-on-findings: false` to report without failing.

## Install

```bash
uvx cachecheck .        # no install
pip install cachecheck  # or install it
```

Python 3.9 or newer. No dependencies, on purpose: this has to run in someone else's CI without an argument about their lockfile.

## Honest limits

It is a static checker built on patterns, not a type checker. It reads code, not runtime behaviour.

**It only reads files that name Anthropic, Claude or Bedrock.** Every question here has an answer on those providers and nowhere else, because only they report the input count net of the cache. The cost is that provider-agnostic code which aggregates usage without naming anyone is skipped, and LlamaIndex's core token counter is exactly that shape. That miss is the direct consequence of this line, not an accident.

**A file naming one of those can still be a false positive.** Bedrock's Nova Sonic reports speech and text tokens that no prompt cache serves, and it is still a Bedrock file. Read the evidence line before acting on a finding.

**Rule 5 reads a window around the assignment**, so a file that folds the cache in more than eight lines away from the rename will be reported. That is a heuristic and it is the weakest thing here.

Silence is a result, not proof. It means nothing matched these five shapes in the files it scanned.

## Licence

MIT.
