Metadata-Version: 2.5
Name: replysignal
Version: 0.1.0
Summary: Catch broken LLM generations in production, without ground-truth labels or an LLM judge.
Project-URL: Homepage, https://github.com/mathewOracle/replysignal
Project-URL: Repository, https://github.com/mathewOracle/replysignal
Project-URL: Issues, https://github.com/mathewOracle/replysignal/issues
Author-email: Mathew Kadambatt <matmart5@gmail.com>
License: MIT
License-File: LICENSE
Keywords: evaluation,generation,guardrails,hallucination,llm,mlops,monitoring,observability,prometheus,quality,rag,sre
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: hypothesis>=6.0; extra == 'dev'
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: prometheus-client>=0.15; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: prometheus
Requires-Dist: prometheus-client>=0.15; extra == 'prometheus'
Description-Content-Type: text/markdown

# replysignal

**Catch broken LLM generations in production — without ground-truth labels or an LLM judge.**

[![CI](https://github.com/mathewOracle/replysignal/actions/workflows/ci.yml/badge.svg)](https://github.com/mathewOracle/replysignal/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/replysignal)](https://pypi.org/project/replysignal/)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](https://pypi.org/project/replysignal/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
[![Typed](https://img.shields.io/badge/typing-strict-blue)](https://peps.python.org/pep-0561/)

Your model returns `200 OK`. Latency is flat. Error rate is zero.

And it's been looping the same sentence for six hours, or refusing every
third request, or answering in JSON wrapped in "Sure! Here's the JSON:".

```python
from replysignal import Monitor

monitor = Monitor(labels={"model": "gpt-4o"})

health = monitor.observe(reply_text)

if not health.ok:
    for report in health.degraded:
        log.warning("%s: %s", report.signal, report.reason)
```

```
repetition: 63% of 3-grams are repeats; the model is looping
format_violation: JSON was requested but the reply does not parse, starting 'Sure! Here's the JSON:'
```

## Why this exists

Generation quality is measured offline, against labels, or by paying for a
second LLM call to judge the first one. Both are fine for evaluating a model
change. Neither tells you a production model started looping at 2am.

| The usual approach | What goes wrong |
|---|---|
| BLEU / ROUGE against references | Needs ground truth. In production you have none. |
| LLM-as-judge (ragas, deepeval, trulens) | Accurate, but another inference call per response — its own latency, its own cost, its own failure mode |
| HTTP status and latency alerts | A perfectly fast, perfectly successful loop or refusal is invisible |
| Manual spot checks | You look after someone complains |

`ragas`, `deepeval`, and `trulens-eval` are excellent at what they do —
judging correctness and faithfulness with labels or a judge model. They are
built for offline evaluation and batch runs. `replysignal` is the other
half: **online, per-response, and judge-free.** Use both — this is what runs
on every single response; save the judge for a sample.

## Install

```bash
pip install replysignal                  # zero dependencies
pip install "replysignal[prometheus]"    # metrics export
```

Python 3.9+. The core has **no runtime dependencies**.

## The idea

You cannot know a reply is *correct* without a judge. You can know a great
deal about whether it's **shaped like the broken ones**:

| Signal | Fires when | Catches |
|---|---|---|
| `Truncation` | The reply was cut off | Token limit too low, streaming cut short |
| `Repetition` | The decoder loops | Sampling misconfiguration, degenerate decoding |
| `Emptiness` | Blank or too short | Silent failures upstream of the text itself |
| `Refusal` | The model declines | Prompt regression, over-tuned safety filter |
| `FormatViolation` | JSON/markdown requested, not delivered | Broken downstream parsers |
| `PromptEcho` | The reply parrots the question | Small/quantised models, bad prompts |
| `Ungrounded` | A RAG answer ignores its sources | Retrieval that ran but wasn't used |

None of these prove an answer is *wrong*. They prove it's shaped like the
ones that were — which is exactly what a paging threshold is for.

### Every signal is honest about what it needs

Only `Truncation`, `Repetition`, `Emptiness`, and `Refusal` run by default —
they need nothing but the text itself. `PromptEcho`, `Ungrounded`, and
`FormatViolation` need extra context and `SKIP` cleanly without it, rather
than guessing:

```python
from replysignal import Reply, Ungrounded, check

reply = Reply(
    text=answer_text,
    prompt=user_question,  # unlocks PromptEcho
    sources=retrieved_chunks,  # unlocks Ungrounded
    expected_format="json",  # unlocks FormatViolation
    finish_reason=api_response.choices[0].finish_reason,
)

health = check(reply, signals=[Ungrounded(warn_below=0.3)])
```

### "I couldn't measure that" is not "that's fine"

A signal that cannot run reports `SKIPPED`, never `OK`. A reply nobody could
judge does not quietly count as healthy, and a skipped signal is never
exported as a number — a fabricated value on a dashboard is worse than a gap
in it.

```python
health.signal("ungrounded").severity  # Severity.SKIPPED, no sources given
health.ok  # unaffected by skips
```

### Truncation trusts the provider first

If the API tells you `finish_reason="length"`, that's a fact, not a
heuristic — `Truncation` reports it as `CRITICAL` immediately. Only without
that does it fall back to checking whether the text ends on terminal
punctuation, and even then only warns, since a bare list item or code block
legitimately lacks a full stop.

### Refusal is a rate, not an incident

One refusal is often the right call. A rising rate of them across your fleet
is the interesting signal — a prompt regression, a model swap, an
over-tuned filter — which is why this reports `WARNING`, not `CRITICAL`:
the value is the trend line, not the individual response.

## Prometheus

```python
monitor = Monitor(labels={"model": "gpt-4o", "endpoint": "chat"})
```

| Series | Type | Use |
|---|---|---|
| `replysignal_evaluations_total{signal,severity}` | Counter | Verdict volume |
| `replysignal_value{signal}` | Gauge | Current quality level |
| `replysignal_degraded_total{signal,severity}` | Counter | **Alert on the rate of this** |

```promql
# Page when repetition spikes for five minutes.
rate(replysignal_degraded_total{signal="repetition",severity="critical"}[5m]) > 0

# Watch the refusal rate trend, not any single refusal.
rate(replysignal_degraded_total{signal="refusal"}[1h])
```

**`prometheus_client` is optional.** Without it, `Monitor` still runs every
signal and still returns a `Health` — it just does not export. A monitoring
library must never be the reason a request fails, and that includes failing
on an import.

## Guarantees

Verified by property-based tests across hundreds of generated replies
([`tests/test_properties.py`](tests/test_properties.py)):

- Evaluating a reply **never raises** for any well-formed input, including
  adversarial unicode, empty strings, and control characters.
- Evaluation **never mutates** the reply it is given.
- Identical inputs produce **identical** verdicts.
- Every signal returns **exactly one** report — nothing vanishes unrecorded.
- A `SKIPPED` signal **never** carries a value, and **never** exports one.
- The bounded signals (truncation, repetition, format violation, echo,
  grounding, refusal) always land in `0..1` — they measure a fraction of
  the reply itself, which cannot be over-delivered the way a page of search
  results can.

Because checking is a pure function returning a plain value, you can assert on it:

```python
def test_empty_reply_is_critical():
    health = check("")
    assert health.signal("emptiness").severity is Severity.CRITICAL
```

## Development

```bash
git clone https://github.com/mathewOracle/replysignal && cd replysignal
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest && ruff check . && ruff format --check . && mypy
```

Contributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). New signals are
especially welcome and shouldn't require touching anything but your own class.

## License

MIT — see [LICENSE](LICENSE).
