Metadata-Version: 2.4
Name: chunk-invariance
Version: 0.3.0
Summary: Test that a streaming redactor or LLM guardrail cannot leak a secret split across chunks.
Author-email: Ninad Phalak <ninadphalak@gmail.com>
Maintainer-email: Ninad Phalak <ninadphalak@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/ninadphalak/LLM-Shield-Proxy/tree/main/chunk-invariance
Project-URL: Repository, https://github.com/ninadphalak/LLM-Shield-Proxy
Project-URL: Bug Tracker, https://github.com/ninadphalak/LLM-Shield-Proxy/issues
Keywords: streaming,testing,redaction,guardrails,chunk-boundary,property-based-testing,mcp,mcp-server,pii,secrets,llm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Framework :: Hypothesis
Classifier: Framework :: Pytest
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: hypothesis
Requires-Dist: hypothesis>=6; extra == "hypothesis"
Provides-Extra: tokens
Requires-Dist: tiktoken>=0.7; extra == "tokens"
Provides-Extra: mcp
Requires-Dist: mcp<3,>=2.3.0; python_version >= "3.10" and extra == "mcp"
Dynamic: license-file

# chunk-invariance

A streaming filter (a redactor, a guardrail, a rewriter) is correct only if, for every way of
splitting an input into chunks, the concatenated streamed output equals the output of filtering
the whole input at once. This package turns that rule into one assertion.

```python
from chunk_invariance import assert_chunk_invariant

def redact(text):                 # your filter on the whole input
    return text.replace("sk-AbCd1234", "[KEY]")

def redact_stream(chunks):        # your filter on a stream: takes chunks, yields chunks
    for chunk in chunks:
        yield redact(chunk)       # redacts each chunk alone

def test_streaming_redaction():
    assert_chunk_invariant(redact_stream, redact, "use sk-AbCd1234 now")
```

That test fails, and names the smallest split that breaks it:

```
chunk_invariance.core.ChunkInvarianceError: stream filter is not chunk-invariant
  input:    'use sk-AbCd1234 now'
  split:    ['use s', 'k-AbCd1234 now'] (cut at 5)
  whole:    'use [KEY] now'
  streamed: 'use sk-AbCd1234 now'
  split #6 in the order tried; generated splits run fewest parts first
```

```bash
pip install chunk-invariance
pip install "chunk-invariance[hypothesis]"   # optional Hypothesis strategies
pip install "chunk-invariance[tokens]"       # optional: real token boundaries (tiktoken)
pip install "chunk-invariance[mcp]"          # optional MCP server for AI assistants
```

## What it catches

- A filter that evaluates each chunk alone, so a value cut across two chunks matches neither half.
- A filter that prepends a fixed carryover from the previous chunk. The match fires, but the start
  of the value was already emitted with the previous chunk.
- A hold-back that is shorter than the pattern, a missing end-of-stream flush, and output that is
  corrupted rather than leaked (any difference from the whole-input output fails).

- A filter that is right whenever a value spans two chunks but wrong when it spans many, as a
  one-token-per-event stream delivers it, or when a chunk ends right after a `.`, `?` or `!`
  inside the value.

`chunk_invariance.examples` has one correct filter (`hold_back`) and three wrong designs for the
same pattern, as a worked comparison: `design_a`, `design_b`, and `lookahead_one`, which passes
every single cut and fails a token stream.

## Stream shapes

Real providers do not cut a reply once. Many send about one tokenizer token per event, so a
value arrives in many small pieces (`555`, `.`, `867`, `.`, `530`, `9`), and a piece often ends
right after a mark inside the value. Others batch a few tokens per event. The default check
therefore runs, after the whole input and every single cut:

| Named set | Splits |
|---|---|
| `"tokens"` | one token per chunk, then 2, 3 and 4 tokens per chunk |
| `"punctuation"` | for each `.`, `?` or `!` followed by a non-space character: a chunk that ends right after it, alone and inside the token stream |
| `"fixed-size"` | chunks of 1, 2, 3, 4 and 8 characters (1 is one character per chunk) |
| `"n-part"` | seeded random splits into exactly 3, 4, 6 and 8 parts |
| `"streams"` | all four, duplicates removed |

Pass any of them as `splits=` to run only that set, or `splits="all-two-part"` for the previous
default (the whole input and every single cut). The extra shapes add a few dozen splits per
input, so the default stays fast. The same generators are importable (`token_splits`,
`punctuation_splits`, `fixed_size_splits`, `n_part_splits`, `stream_splits`, `tokenize`).

Token boundaries come from tiktoken (`o200k_base`) when it is installed, and otherwise from a
small built-in tokenizer that splits words, digit groups of up to three and punctuation the same
way. The base install stays standard library only.

## What it does not catch

It tests a filter you can call in-process. It does not test a running gateway, where the chunks
are decided by an upstream model, a network and an SSE parser. For that, use
[`pii-leak-benchmark`](https://pypi.org/project/pii-leak-benchmark/), which sends
fragmented responses through the gateway and checks what reaches the client.

It checks the splits you ask for. The default is the whole input, every single cut, and the
stream shapes above. `splits="all"` tries every split of a short input (up to 16 characters);
`splits=3` tries every split into at most three parts; an explicit list of chunk lists runs
exactly those. It cannot prove a filter correct for inputs you did not give it, and the token
shapes come from one tokenizer, not from every provider's.

## Use it from your AI assistant

<!-- mcp-name: io.github.ninadphalak/chunk-invariance -->

Do you use Claude Code, Cursor or Claude Desktop? The package has an optional MCP server. With it
you can ask the assistant to run the every-split check on your own streaming function, and then
to add the test to your repo.

```bash
pip install "chunk-invariance[mcp]"      # Python 3.10 or later
claude mcp add chunk-invariance -- chunk-invariance-mcp
```

For Claude Desktop, add this to `claude_desktop_config.json` (give the full path to the command
if Claude Desktop cannot find it on your PATH):

```json
{
  "mcpServers": {
    "chunk-invariance": { "command": "chunk-invariance-mcp" }
  }
}
```

Then ask, for example: "Check whether `redact_stream` in `myapp/filters.py` leaks values split
across chunks, and add a regression test for it."

There are three tools. `check_stream_filter` imports your function (`myapp.filters:redact_stream`),
streams each sample through it in every two-part split and tells you which split broke it, for
example that `'sk-AbCd1234'` came out unredacted when the input was cut into `'use s'` and
`'k-AbCd1234 now'`. `write_regression_test` returns a pytest file built on `assert_chunk_invariant`,
so the check stays in your CI. `explain` is one paragraph on why per-chunk filtering leaks.

The default samples (an email, an SSN, a card number and a few API-key shapes) are all fake. Pass
your own made-up samples in the formats your filter redacts, such as a key you typed yourself in
your provider's format. A sample the filter does not change tests nothing, and the tool says so.

Never pass a real secret or real personal data as a sample. The samples, and any value your filter
leaks, come back in the tool's result, and your assistant sends that result to its model provider
like the rest of the conversation. So does the message of any exception your code raises, but
nothing your code prints. The same samples also end up in the generated test file.

Checking a function runs your code. The server imports your module on your own machine, in a
separate process that is stopped after a timeout (60 seconds by default). If your project has a
`.venv` (or `venv`), that is the Python it uses; the `python` argument accepts only that
interpreter or the server's own. The target, `flush_target` and `whole_target` must be Python
functions defined in source files under `project_dir`: built-ins, the standard library and
installed packages (anything in the interpreter's site-packages, including your `.venv`'s) are
refused, and so is a `project_dir` that is or contains the interpreter's own library. The check
process gets a minimal environment (`PATH`, `HOME`, temp and locale variables), not the server's,
so a filter that reads its configuration from other environment variables will not see them.
`project_dir` is chosen by the assistant's tool call, so whatever code lives under it is what can
run: approve calls only for your own project. The server itself makes no network calls, sends no
telemetry and writes no files, though the code it runs can do whatever that code does. Saving the
test file is up to you and your assistant.

## API

| Name | Does |
|---|---|
| `assert_chunk_invariant(stream_filter, whole_filter, text, *, splits="default")` | Raises `ChunkInvarianceError` (an `AssertionError`) on the first failing split, in the order tried (the whole input and single cuts first). Returns the number of splits checked. Works on `str` and `bytes`. |
| `assert_chunk_invariant_async(...)` | The same, for an async generator filter, inside an async test. |
| `two_part_splits(text)` | Every split into two non-empty parts. |
| `all_splits(text, max_parts=None)` | Every split into at most `max_parts` non-empty parts, the whole input first. |
| `token_splits`, `punctuation_splits`, `fixed_size_splits`, `n_part_splits`, `stream_splits` | The stream-shape generators behind the named sets. |
| `tokenize(text, encoding="auto")` | Token boundaries: tiktoken when installed, the built-in tokenizer otherwise. |
| `per_chunk(process, *, state=dict, flush=None)` | Adapts `process(chunk, state) -> output` and `flush(state) -> output`, with fresh state per stream. |
| `from_async(async_stream_filter)` | Adapts an async generator filter for the synchronous assertion. |
| `strategies.splits_of(text, max_parts=None)` | Hypothesis strategy: random splits of `text`, shrinking toward fewer cuts. |
| `strategies.text_and_splits(texts, max_parts=None)` | Hypothesis strategy: `(text, chunks)` pairs. `texts` is itself a strategy, such as `st.sampled_from([...])`, not a list. |

`stream_filter` is called once per split, so it must start from fresh state each time.

## Background

The invariant and the two wrong designs are described in Phalak, N. (2026), *Split-Boundary
Leaks in Streaming Guardrails*, Zenodo, [10.5281/zenodo.22909585](https://doi.org/10.5281/zenodo.22909585).

Standard library only (the `mcp` extra adds the MCP SDK, for the server alone; `tokens` adds tiktoken). Apache-2.0. A TypeScript version with the same core API (without the stream shapes yet) is
published on npm as `chunk-invariance`.
