Metadata-Version: 2.4
Name: replaycheck
Version: 0.2.0
Summary: Find the bugs that only appear when an event is delivered twice.
License: MIT License
        
        Copyright (c) 2026 replaycheck contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/venwork-dev/replaycheck
Project-URL: Repository, https://github.com/venwork-dev/replaycheck
Project-URL: Issues, https://github.com/venwork-dev/replaycheck/issues
Keywords: event-driven,idempotency,property-testing,replay
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: setuptools>=61; extra == "dev"
Requires-Dist: wheel; extra == "dev"
Dynamic: license-file

# replaycheck

[![CI](https://github.com/venwork-dev/replaycheck/actions/workflows/ci.yml/badge.svg)](https://github.com/venwork-dev/replaycheck/actions/workflows/ci.yml)

Find the bugs that only appear when an event is delivered twice.

## The bug it finds

Your handler charges a card, then marks the order paid:

```python
world.effect("charge", order=order, amount=event["amount"])
world.effect("paid", key=order, order=order)
```

Crash between those two lines and the retry charges the card again. Every backend
developer knows this bug exists. Almost nobody tests for it, because testing it
means simulating a crash at every single step.

## What it does

`replaycheck` runs your handler once cleanly to learn what the correct end state
looks like. Then it runs the same events again and again — each time killing the
process the instant a different side effect becomes durable, and resuming from
the last committed position, exactly as an at-least-once broker would. It also
delivers each event twice with no crash at all.

Every one of those runs has to end in the same durable state as the clean run.
When one doesn't, the failing schedule is shrunk to the shortest input that still
breaks.

## Try it

```
make demo
```

## Install

Install the released package from PyPI:

```console
python -m pip install replaycheck
```

For development, install the repository with its test tools:

```console
python -m pip install -e ".[dev]"
```

The 0.2 release supports Python 3.9 through 3.13. The supported public Python
surface is `check`, `sweep`, `World`, `Crash`, `Report`, `Failure`, `Schedule`,
`Plan`, `Stalled`, and `hazards`, plus the `replaycheck` console command. The
handler contract is `handler(event, world)`; applications should treat the
event and durable-effect data as snapshots and should not connect adapters to
live services.

Patch releases preserve this surface. New minor releases may add APIs and CLI
options, but existing documented behavior remains supported unless a release
note calls out a deliberate breaking change.

```
--- before ---
FAIL  crashed after charge() on event 0
      order-772 charged twice
      shortest failing input (1 event(s)):
        {'event_id': 'e2', 'order_id': 'order-772', 'amount': 1500}

--- after ---
PASS  7 schedules, 4 durable writes, no divergence
```

The only difference between the two handlers is one argument — `key=order` on
the charge sink, which makes it idempotent.

## Using it

Three things: your handler, some events, and one rule that must always hold.

```python
from replaycheck import check

def process_order(event, world):
    order = event["order_id"]
    if world.has("paid", order):
        return
    world.effect("charge", key=order, order=order, amount=event["amount"])
    world.effect("paid", key=order, order=order)

def charged_at_most_once(world):
    for _, data in world.effects("charge"):
        assert world.count("charge", order=data["order"]) <= 1

report = check(process_order, events, invariant=charged_at_most_once)
assert report, report.text()
```

For the implementation map and scaling model, see [ARCHITECTURE.md](ARCHITECTURE.md).

The handler writes through a `World` instead of a real database:

- `world.effect(name, key=None, data=None, **fields)` — a durable write. Pass
  `key` and the sink deduplicates, the way a real idempotent endpoint would.
  `name`, `key` and `data` are taken by the method, so a recorded field of your
  own with one of those names goes through `data=` — which is the usual case for
  a Kafka record key: `world.effect("dlq", key=idem, data={"key": record.key})`.
- `world.has(name, key)` — has this already been written?
- `world.count(name, **match)` / `world.effects(name)` — for invariants.

The harness owns the commit position: an event is committed when your handler
returns for it, and a crash resumes from the last one that did.

## What counts as "the same" write

Nothing is matched by dollar amount. Two effects are the same only if the name
and *every* recorded field match, and the comparison is a multiset — so two
orders for $50 are two writes, and a third $50 charge is a duplicate. Amount
collisions do not confuse it.

What matters is which fields you record. `world.effect("charge", amount=50)`
records only the amount, so "o1 and o2 each charged $50" and "o1 charged $50
twice, o2 never charged" produce the same durable state. Same total, wrong
customer, invisible. Recording the order id separates them.

The report says so rather than leaving you to work it out:

```
PASS  7 schedules, 4 durable writes, no divergence
NOTE  2 identical charge(amount=50) writes -- a misattribution between them
      would be invisible; record a distinguishing field
```

It is a note, not a failure — two identical writes can be perfectly correct. It
only tells you the comparison has a blind spot there.

Note that `key=` and the recorded fields do different jobs. `key` decides whether
the sink suppresses a second write; the recorded fields decide whether the
comparison can tell two writes apart. A keyed sink can still leave a blind spot
if what it records is not distinguishing.

## Out-of-order arrival

Off by default, because a reordering is not automatically a bug — if your source
guarantees order, a divergence here is a property your handler was never required
to have. Turn it on when the source really can deliver out of order, and say how
far an event may slip:

```python
check(fulfil, events, reorder=1, compare=["paid", "shipped"])
```

```
FAIL  event 0 arrives 1 position(s) late
      never wrote 1x: shipped(order='order-771')
```

That handler drops a shipment that overtakes its payment. One that parks the
early shipment and releases it when the payment lands passes.

`compare` matters here. Reordering can legitimately leave different internal
bookkeeping behind while reaching the same business outcome — the tolerant
handler writes a `pending_ship` marker the canonical order never writes. Name the
effects that constitute the outcome and the rest is ignored. Without it you get a
failure for a handler that is doing the right thing.

```
make ordering
```

## Starting from state you already have

A consumer usually restarts against a database that is already partly populated,
and some replay bugs only appear from there. `setup` writes that state into every
run — baseline and schedules alike — before delivery starts. Writes made during
setup are pre-existing rows, never crash points.

```python
def already_paid(world):
    world.effect("paid", key="order-771", order="order-771")

check(process_order, events, setup=already_paid)
```

## When the handler raises

If your handler raises anything that is not a simulated crash, the event is never
committed, so an at-least-once broker redelivers it forever. That is a poison
pill, and it is reported as a failure naming the event rather than escaping as a
traceback:

```
FAIL  handler raised on an event it never commits
      handler raised KeyError on event 1: 'order_id' -- this event is never
      committed, so it is redelivered forever
```

If raising is what you intend, catch it in the handler and write a dead-letter
effect instead.

## Large streams

Cost is quadratic: every durable write is a crash point, and every schedule
replays the whole stream. Measured on one machine, fully enumerated:

| events | schedules | time |
|---|---|---|
| 100 | 301 | 0.09s |
| 400 | 1,201 | 1.70s |
| 800 | 2,401 | 7.69s |

Extrapolating, 10,000 events is roughly twenty minutes and 100,000 is out of
reach. So don't enumerate a huge stream — replay bugs are local, and every
failure this tool finds shrinks to one or two events. A few hundred short random
streams cover the same transitions for a fraction of the work:

```python
from replaycheck import sweep

def make_events(rng):
    return [
        {"order_id": f"o{rng.randint(0, 3)}", "amount": rng.randrange(100, 9999)}
        for _ in range(rng.randint(1, 5))
    ]

sweep(process_order, make_events, runs=300)
```

`sweep` reports the coverage its inner runs actually had, so a capped budget
still surfaces as PARTIAL rather than PASS.

Same bug, found in 0.00s instead of 2.53s, and shrunk to a single event.

If you do run a long stream, `max_schedules` caps the work and the result says so
rather than pretending:

```
PARTIAL  no divergence in 201 of 1998 schedules (sampled, seed 0);
         raise max_schedules to cover the rest
```

Sampling is stratified across crash, duplicate and reorder schedules and seeded,
so a partial run still covers every family reproducibly. The minimum
`max_schedules` is therefore one slot per enabled family that has candidates:
normally 2 for crash and duplicate schedules, or 3 when `reorder` is enabled on
a stream with at least two events. The clean baseline run is additional and does
not consume this budget. If a handler makes no durable writes, there is no crash
family, so the corresponding minimums are 1 without reordering and 2 with it. An
empty stream has no candidate schedules and permits 0. A smaller value is
rejected with an error that names the enabled families and the minimum required
value.

The cap also bounds plan memory. Schedule generation shares one defensive copy
of the events and instantiates only the selected descriptors; it does not build
every candidate before sampling. A 10,000-transaction fixture with a budget of
50 schedules therefore executes 51 runs (the clean run plus 50 sampled fault
schedules), rather than retaining all 20,000 crash and duplicate descriptors.

## Testing another repository

Install `replaycheck` in that repository's test environment, then add a small
adapter that maps its decisions and side effects onto `World`. For example,
`replaycheck_adapter.py` at the repository root:

```python
def handle_transaction(event, world):
    transaction = event["transaction_id"]
    if world.has("posted", transaction):
        return
    world.effect(
        "posted",
        key=transaction,
        transaction_id=transaction,
        amount=event["amount"],
    )

def balances_stay_non_negative(world):
    for _, posting in world.effects("posted"):
        assert posting["amount"] >= 0
```

Run it from the other repository's root, where its modules are importable:

```console
replaycheck check \
  --handler replaycheck_adapter:handle_transaction \
  --invariant replaycheck_adapter:balances_stay_non_negative \
  --events tests/fixtures/transactions.jsonl \
  --max-schedules 200 \
  --seed 0
```

The command exits 0 when no selected schedule diverges, 1 when it finds a replay
failure, and 2 for invalid input or adapter configuration, so it can run directly
in CI. `--setup`, `--reorder`, and repeatable `--compare EFFECT` flags expose the
same controls as the Python API.

The adapter must be a model: never point it at production services. Replaycheck
deliberately duplicates delivery and injects crashes after durable effects.

## Fixture hazards

Replay bugs hide behind well-behaved sample data. This says so before you trust
a fixture:

```
make hazards
```

```
ABSENT   duplicate delivery: 0 repeated event_id value(s) across 4 events
ABSENT   out-of-order arrival: 0 event(s) arrive with a timestamp before their predecessor
ABSENT   sequence gap: 0 gap(s) in offset

4 events, and none of them exercise: duplicate delivery, out-of-order arrival, sequence gap
```

## What it does not do

**The clean run is the oracle.** This checks that every failure schedule ends
where a clean run ends — it does not check that the clean run is *correct*. A
handler that charges the wrong amount every time is perfectly replay-invariant
and will pass:

```python
check(charges_999_always, events)                       # PASS
check(charges_999_always, events, invariant=amount_matches)   # FAIL
```

Divergence catches bugs that appear under replay. Everything else has to come
from the invariant you write. Nothing here measures conformance to a spec.

- It does not run your real database, broker, or network. You model side effects
  through `World`; the fidelity of the result is the fidelity of that model.
- It injects one crash per run. Interleaved failures across concurrent consumers
  are out of scope — that is Jepsen's problem, not this one.
- Reordering is bounded and opt-in: one event moved up to `reorder` positions.
  Arbitrary permutations and concurrent interleavings are not covered.
- A passing report means every schedule it tried ended in the same state. It is
  evidence, not a proof.

## Tests

```
make test
```

To compare replay cost on the current machine:

```console
make benchmark
```

The benchmark is informational and is not included in the default CI test job.

For package and PyPI release steps, see [RELEASING.md](RELEASING.md). The
project's release history is in [CHANGELOG.md](CHANGELOG.md).
