Metadata-Version: 2.5
Name: cache-pressure
Version: 0.1.0
Summary: KV-cache retention pressure benchmark for OpenAI-compatible endpoints
Author: Conrad
License-Expression: MIT
License-File: LICENSE
Keywords: benchmark,kv-cache,llm,openai,prefix-cache
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.9
Requires-Dist: requests>=2.31
Description-Content-Type: text/markdown

# cache-pressure

Measure how much context a prefix cache **actually retains** under overflow
pressure — not what the engine *advertises*.

The advertised KV cache size is an **active-request planning budget**, not a
ceiling on cacheable content. Hybrid layouts (MLA + sparse-attention +
recurrent-state groups) pack cached blocks denser than that reservation,
and cache-management defects can waste the headroom with duplicate blocks
and unreachable replay tails. This tool quantifies the real number, so you
can tell a healthy cache from a leaking one on any OpenAI-compatible
deployment.

## How it works

```
1. capacity   the engine's advertised KV size in tokens, provided manually
              with --kv-size (required)
2. calibrate  tokens/char for this tokenizer at the target length
              (length-dependent, iterates to convergence), then the
              hit/miss threshold: one probe context sent cold (miss
              baseline) and again from cache (hit baseline) — the
              threshold is their midpoint
3. hydrate    N unique ~8K-token contexts sequentially (max_tokens=1,
              prefill-only — the full context is committed regardless)
              N = ceil(capacity/8000)+5, so the cache overflows
4. verify     re-send each context in reverse order (newest first),
              classify hit/miss by TTFT against the calibrated threshold
              → the first miss is the oldest evicted context = the real
              retained capacity. Stops there: under LRU everything older
              is evicted by construction.
```

Contexts are seeded random word-orders (unique per salt within a run,
byte-identical across runs) so no context rides on another's cached
content, and two runs (e.g. before/after a fix) replay the same text.

The default 8K context size keeps the measurement granularity fine; pass
`--context-tokens` to coarsen or refine. The hit/miss threshold is
calibrated at runtime from measured cold-prefill and cache-hit TTFT, so it
stays valid regardless of engine speed — override with `--hit-threshold`
only if you have a reason.

## Usage

Run directly from PyPI — no install needed:

```bash
# point --base-url at your OpenAI-compatible endpoint; the model is
# auto-detected via GET /models (pass --model to override)
uvx cache-pressure --base-url http://my-server:8000/v1 \
    --kv-size <ADVERTISED_KV_CACHE> --output run.json

# A/B two runs
uvx cache-pressure --compare fix.json control.json
```

Small sanity check (3 contexts, all hits):

```bash
uvx cache-pressure --base-url http://my-server:8000/v1 \
    --kv-size <ADVERTISED_KV_CACHE> --num-contexts 3
```

### `needle-test` — companion correctness check

The retention number means nothing if the engine is broken. This sweeps
increasing context lengths (default 50K/100K/200K/300K/450K), hides a
needle sentence at 0.8 depth in each unique haystack, and requires the
model to output the exact secret code — proving long-context retrieval is
intact end-to-end.

```bash
uvx needle-test --base-url http://my-server:8000/v1  # full sweep (50K -> 450K)
uvx needle-test --base-url http://my-server:8000/v1 \
    --lengths 50000,100000                            # subset
```

`needle-test` calibrates against its own target lengths and does not need
the advertised capacity (it only needs to stay under the engine's
`max_model_len`). The model is auto-detected like in `cache-pressure`; pass
`--base-url`/`--model` to point it at your endpoint.

## Development

```bash
git clone <repo> && cd cache-pressure
uv sync                        # create .venv with the project installed
uv run python tests/test_cache_pressure.py    # no cluster needed

# exercise the console scripts against the local checkout
uvx --from . cache-pressure --help
uvx --from . needle-test --help
```

## Interpreting results

`retained % capacity` can exceed 100% — that is **not a bug**. The
advertised capacity is a worst-case reservation per active context token;
cached content (constant-size recurrent state, bounded sliding-window
groups, deduplicated blocks) packs denser, so the cache can physically hold
more context than the budget implies. One fixed deployment retained ~147%
of advertised capacity with zero evictions at 39K granularity; the
pre-fix engine kept ~52-65%.
