Metadata-Version: 2.5
Name: legal-rag-verifier
Version: 0.5.0
Summary: Sentence-level claim and NLI verification with KV-cache rollback during generation, for legal RAG
Project-URL: Homepage, https://github.com/azterizm/legal-rag-verifier
Project-URL: Repository, https://github.com/azterizm/legal-rag-verifier
Project-URL: Issues, https://github.com/azterizm/legal-rag-verifier/issues
Project-URL: Changelog, https://github.com/azterizm/legal-rag-verifier/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/azterizm/legal-rag-verifier/blob/main/docs/API.md
Project-URL: Verdict, https://github.com/azterizm/legal-rag-verifier/blob/main/docs/VERDICT.md
Project-URL: Consultancy, https://memonsystems.com/engagements
Author-email: Abdullah Memon <abdullah@memonsystems.com>
Maintainer-email: Memon Systems Ltd <abdullah@memonsystems.com>
License-Expression: AGPL-3.0-only
License-File: LICENSE
Keywords: hallucination,kv-cache,legal,nli,rag,verification
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Legal Industry
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.11
Provides-Extra: hf
Requires-Dist: accelerate>=1.0; extra == 'hf'
Requires-Dist: bitsandbytes>=0.46.1; (platform_system != 'Windows') and extra == 'hf'
Requires-Dist: torch>=2.4; extra == 'hf'
Requires-Dist: transformers>=4.56; extra == 'hf'
Provides-Extra: nli
Requires-Dist: torch>=2.4; extra == 'nli'
Requires-Dist: transformers>=4.56; extra == 'nli'
Provides-Extra: sglang
Requires-Dist: transformers>=4.56; extra == 'sglang'
Description-Content-Type: text/markdown

# legal-rag-verifier

**Checks each sentence of an LLM answer against its source while the model writes it, and rewrites only the sentences that fail. Tested on UK law.**

[![PyPI](https://img.shields.io/pypi/v/legal-rag-verifier)](https://pypi.org/project/legal-rag-verifier/)
[![CI](https://github.com/azterizm/legal-rag-verifier/actions/workflows/ci.yml/badge.svg)](https://github.com/azterizm/legal-rag-verifier/actions/workflows/ci.yml)
[![Python 3.11–3.14](https://img.shields.io/badge/python-3.11%E2%80%933.14-blue)](https://github.com/azterizm/legal-rag-verifier/blob/main/pyproject.toml)
[![Licence: AGPL-3.0-only](https://img.shields.io/badge/licence-AGPL--3.0--only-blue)](https://github.com/azterizm/legal-rag-verifier/blob/main/LICENSE)
![Runtime dependencies: none](https://img.shields.io/badge/runtime%20dependencies-none-brightgreen)

## Results

240 UK law questions, Qwen 2.5 7B answering, a Qwen 27B judge. Answers containing at least one
wrong sentence:

| | No check | With the verifier | Refused |
|---|---|---|---|
| Answering from its own knowledge | 198 | 28 | 7 % |
| Source documents in the prompt | 25 | 13 | 4 % |

On Claude Opus 4.6, with Gemini 3.8 Flash checking, rewriting a rejected sentence instead of
deleting it kept 71 of the 80 correct sentences that deleting lost, for one more wrong sentence
across 240 answers.

**How to check it.** Each test's method was written down and hashed before the run
([`docs/GRID.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/GRID.md)). [`docs/RESEARCH.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/RESEARCH.md) maps every test
to its figures, scripts and result files. The answers quote statute text and stay private; their
grades, reports and hashes are committed.

**What it does not show.** Status: proof of concept. The gains come from an LLM judge, now in
the package as `legal_rag_verifier.judge` ([Use the judge](#use-the-judge)). The DeBERTa head
the package ships did not improve answers. UK statute questions only. The [verdict](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/VERDICT.md) states what holds, what does not and
what is required first.

`legal-rag-verifier` checks each sentence of an answer against the statutory provisions it was
given, as the sentence is written. A deterministic claim check covers figures, dates, citations,
instrument titles, modal verbs and qualifiers. An entailment check covers the prose. A rejected
sentence is rolled back and regenerated on the inference server's prefix cache, so only that
sentence is thrown away. The retry can insert the provision the sentence was checked against at
the cut point.

Built by [Abdullah Memon](https://memonsystems.com) at [Memon Systems Ltd](https://memonsystems.com).
Pairs with [`legal-rag-router`](https://github.com/azterizm/legal-rag-router), which can refuse a
question before it reaches the model.

## Install

```bash
pip install legal-rag-verifier            # claim check only: no runtime dependencies
pip install "legal-rag-verifier[nli]"     # + DeBERTa-v3 NLI head (torch, transformers)
pip install "legal-rag-verifier[sglang]"  # + SGLang client for inject (allow, ban: + torch, sglang)
pip install "legal-rag-verifier[hf]"      # + in-process Hugging Face generation
```

Python 3.11 or later. Built on United Kingdom legislation. Prose is read in English; figures (numbers,
prices, dates and durations) in English and French formats.

## Check a sentence

```python
from legal_rag_verifier.premise import Premise
from legal_rag_verifier.verifier import Verifier

premise = Premise.from_text(
    "The limit of a compensatory award is the lower of £123,543 and 52 multiplied by a week's pay.",
    titles=("Employment Rights Act 1996",),
    citations=("s124",),
)
verifier = Verifier()  # Verifier(SentenceNLIVerifier("cross-encoder/nli-deberta-v3-base")) adds NLI

verdict = verifier.check_sentence(premise, "The compensatory award is capped at £85,000.")
verdict.verdict  # Verdict.ROLLBACK
verdict.reasons  # (Reason.UNGROUNDED_FIGURE,)
verdict.ungrounded  # ['£85,000']
```

`check_text` runs the same checks over a whole answer, for post-hoc use.

`SentenceNLIVerifier` now warns when built: the DeBERTa head did not improve answers, and the
[judge](#use-the-judge) is recommended (see [`docs/VERDICT.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/VERDICT.md)).

## Generate with rollback

```python
from legal_rag_verifier.backends.sglang import SGLangBackend
from legal_rag_verifier.engine import InFlightGenerator
from legal_rag_verifier.judge import Judge, JudgedVerifier

backend = SGLangBackend.connect("http://127.0.0.1:30000", "Qwen/Qwen2.5-7B-Instruct")
judge = Judge.from_preset("qwen27-ar4", "http://127.0.0.1:30001")  # the judge's own server
verifier = JudgedVerifier(judge)
generator = InFlightGenerator(backend, verifier, steering="inject")

verifier.reset()  # before each answer: the engine does not reset the judge
answer = generator.generate_verified("What is the cap on the compensatory award?", premise)
answer.text  # the verified answer
answer.trace  # per sentence: verdicts, retries, what was inserted, prefix-cache hits, timings
```

The engine sends one request per sentence. A rejected sentence is not resubmitted, so the next
request reuses the verified prefix from the server's cache (SGLang's RadixAttention). Each sentence
gets at most three retries and each answer at most eight; after that the engine writes a fixed
refusal for that point and moves on. Retry modes:

- `allow`: constrain a rejected figure to the grounded values of the same type, then `ban`;
- `ban`: restart the sentence with its rejected first token banned;
- `inject`: insert the provision the sentence was checked against at the cut point (a new user
  turn; `inject_template`, which may restate the question with `{query}`) and regenerate. The
  provision stays in the context and never enters the answer.

The SGLang server needs `--enable-custom-logit-processor` for `allow` and `ban`. They also need
`torch` and `sglang` on the client, which the `[sglang]` extra does not install; without them the
first `allow` or `ban` retry raises an `ImportError` that says what to do: install them, build the
mask where they are installed (`MaskProcessor().to_str()`) and pass it as
`SGLangBackend(..., processor=...)`, or use `inject`, which needs neither. Full API:
[`docs/API.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/API.md).

## Use the judge

`legal_rag_verifier.judge` is the LLM judge behind the results above, now in the package: stdlib
only, against any OpenAI-compatible endpoint (SGLang, vLLM). It takes the NLI head's place as the
entailment check, with the measured prompt fixed.

Measured presets:

| Preset | Model and revision | Quantisation, server, reply | Right on (stop 61) | Per clause, p50 | Figures |
|---|---|---|---|---|---|
| `qwen27-ar4` | `Vishva007/Qwen3.8-27B-W4A16-AutoRound-GPTQ` at `1b9d56b4` | W4A16 AutoRound; SGLang 0.5.21, thinking off; short reply | 93.8 % of 354 sentences, 90.1 % of 151 | 0.54 s | [`docs/measurements.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/measurements.md#a-judge-reply-without-the-reason-stop-61-2026-10-12-method-in-docsgridmd) |

The 0.54 s per clause is from the first test only: a timing run of 40 prompts sent one at a time.
Any other model is unmeasured. A judge built by hand, not from a preset, carries `preset: None` in
its call log. On answers written with the source in the prompt, this judge scored below no auditor:
88.9 % against 94.4 % on 72 sentences (stop 73).

Serve the preset (SGLang 0.5.21, one L40S-class GPU):

```bash
python -m sglang.launch_server --model-path Vishva007/Qwen3.8-27B-W4A16-AutoRound-GPTQ \
  --revision 1b9d56b428e9f841d7ad566f87660b5f47553c97 \
  --reasoning-parser qwen3 --context-length 65536 --mem-fraction-static 0.85
```

Then check an answer against it:

```python
from legal_rag_verifier.judge import Judge, JudgedVerifier

# premise (a Premise, as in "Check a sentence"), answer and question come from your RAG pipeline
judge = Judge.from_preset("qwen27-ar4", "http://localhost:30000")
verifier = JudgedVerifier(judge)  # the measured config, STRICT_FIGURES
verdicts = verifier.check_text(premise, answer, query=question)
verifier.calls  # one record per judge call: label, passage, attempts, ms, tokens, sentence, preset
```

For a hosted endpoint, `api_key_env="NAME"` sends that environment variable as a bearer token; it
is read at call time and never stored or logged. In the engine, pass the `JudgedVerifier` as the
verifier and call `verifier.reset()` before each answer.

A judge that cannot be reached scores every sentence not stated, so every sentence rolls back:
check the endpoint before a run. Full interface:
[`docs/API.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/API.md) §2b.

## Evidence

Every figure below is measured. Figures: [`docs/measurements.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/measurements.md).
Methods, fixed before each run: [`docs/GRID.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/GRID.md). Raw results:
[`results/`](https://github.com/azterizm/legal-rag-verifier/tree/main/results).

**Detector, sealed held-out battery** (346 rows, sealed before it ran once; claim check +
`nli-deberta-v3-base` + enrichment): 80.0 % of planted errors caught (124/155), 7.9 % of correct
sentences rolled back (15/191), 89.2 % rollback precision. Weakest: dropped qualifiers (6/18).

**Rollback on Qwen 2.5 7B, SGLang, one NVIDIA L4:** every retry reused the verified prefix from the
cache. A retry request costs 117 ms in bf16 (69 ms in FP8), against 270 ms (170 ms) to re-prefill.
Checking sentence by sentence released the first verified sentence after 3.8 s, against 8.1 s when
the whole answer was checked first.

**Inserting the provision at the cut point** (48 rejected sentences retried from the same state):
the retry passed the auditor 39 times with the provision inserted, against 19 with decode-level
steering (exact McNemar p = 1.1e-5). The same text at the top of the prompt passed 17 of 46; a new
turn without the provision, 14. With the provision inserted, the regenerated sentence became more
likely in 46 of 46 states, the rejected sentence less likely in 45 of 46, and attention on
provision text rose in 46 of 46.

**Live output, DeBERTa head as the auditor:** its rejections were mostly wrong: 1 of 12 by hand on
software questions, 18 of 46 by a frontier model's labels on statutes. Graded by hand, the loop's
answers were no better than one plain answer from the same model. Inserting the provision makes
the retry pass the auditor; it does not make it better.

**An LLM judge in the auditor's place** (Qwen 27B at 4 bits as generator and judge on one SGLang
server; the judge is the entailment scorer, in `scripts/live_loop.py`): right on 93.8 % and 90.1 %
of the sentences of two hand-graded tests (354 and 151 sentences), against 68.7 % and 58.3 % for
the DeBERTa head. Run live on 40 questions with the source in the prompt, 48 of 49 sentences were
committed at the first try, none wrong, at 1.9 s per answer at the median. When the answer started
from memory, 36 of 65 sentences were wrong without the loop and 3 of 50 with it, all three
refusals.

**The loop on 240 questions** (the 27B as judge): closed-book, it cut answers holding a wrong
sentence from 132 to 23 for the 27B, 198 to 28 for Qwen 2.5 7B and 92 to 53 for MiniCPM5-1B. With
the source in the prompt it cut the 7B's from 25 to 13 (Holm p = 0.045).

**Against frontier models** (the same 240 questions, source in the prompt): answers holding a wrong
sentence, Qwen 27B with the loop 4, Gemini 3.8 Flash 3, Claude Opus 4.6 8, Qwen 2.5 7B with the
loop 13. The 27B is within the fixed margin of 4 points of both frontier models; the 7B is not.
The 27B answers in 2.0 s at the median against 3.1 to 4.4 s, and states less per answer.

**Other judges on the same two tests:** a 1B model trained on the 27B's labels, 89.0 % and 82.1 %
at 0.06 s per call ([adapters](https://huggingface.co/azterizm/minicpm5-1b-clause-judge),
DOI [10.57967/hf/10885](https://doi.org/10.57967/hf/10885)); Jev
1.13, 90.7 % and 88.1 %; the OpenAI Decisions API, 89.0 % and 86.8 %. None is within 3 points of
the 27B on both tests. ConSens, Iris.ai's context-reliance metric with Llama 3.2 1B and no judge reply,
scored 81.5 % on the second test (threshold fixed on the first) at 30 ms per sentence. It tells
closed-book answers from grounded ones, and caught none of the 4 wrong sentences written with the source.

See [`docs/VERDICT.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/VERDICT.md) for the proof-of-concept verdict and its limits.

## Reproduce the evidence

The repository, not the package, holds the evidence behind the figures: the batteries and their
seals (`batteries/`), the per-provision enrichment (`enrichment/`), the labels, grades and reports
of every test (`results/`) and the scripts that produced them (`scripts/`). From stop 43 on,
answers and traces quote statute text, so they stay local. Each test's seal and report keep their
hashes and counts.
[`docs/RESEARCH.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/docs/RESEARCH.md)
maps each test to its method, figures, scripts and results. The rollback-point tables and the
placement and attention report re-derive from the committed results, with no GPU or API key:

```bash
git clone https://github.com/azterizm/legal-rag-verifier && cd legal-rag-verifier && uv sync
uv run python scripts/grid_inject.py --cells 4B 4B-inject 4B-inject-v2
uv run python scripts/replay_report.py
```

Re-scoring the grid (`scripts/grid_score.py`) and re-running the batteries also need the statute
corpus, built with [`legal-rag-router`](https://github.com/azterizm/legal-rag-router) and copied
to `data/`; it is not redistributed here. Generation runs used Modal (`scripts/modal_m4.py`).

## Development

```bash
uv sync
uv run ruff check && uv run ruff format --check && uv run mypy && uv run pytest
```

Contributing: [`CONTRIBUTING.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/CONTRIBUTING.md). Security: [`SECURITY.md`](https://github.com/azterizm/legal-rag-verifier/blob/main/SECURITY.md).

## Citation

If you use the verifier or its evaluation, cite it with [`CITATION.cff`](https://github.com/azterizm/legal-rag-verifier/blob/main/CITATION.cff)
("Cite this repository" on GitHub).

## Contact

`legal-rag-verifier` is built and maintained by **Abdullah Memon** at
**[Memon Systems Ltd](https://memonsystems.com)**. Bug reports and questions are welcome as
[GitHub issues](https://github.com/azterizm/legal-rag-verifier/issues).

## Licence

AGPL-3.0-only (see [`LICENSE`](https://github.com/azterizm/legal-rag-verifier/blob/main/LICENSE)). Contains public sector information licensed under the
Open Government Licence v3.0 (see [`NOTICE`](https://github.com/azterizm/legal-rag-verifier/blob/main/NOTICE)).
