Metadata-Version: 2.5
Name: gasp-rag
Version: 0.2.1
Summary: Span-level detection of ungrounded content in retrieval-augmented generation by grounding sensitivity.
Project-URL: Homepage, https://github.com/drbouke/gasp-rag
Project-URL: Repository, https://github.com/drbouke/gasp-rag
Project-URL: Issues, https://github.com/drbouke/gasp-rag/issues
Author-email: Mohamed Aly Bouke <bouke@ieee.org>
License: MIT
License-File: LICENSE
Keywords: explainable-ai,faithfulness,hallucination-detection,large-language-models,rag,retrieval-augmented-generation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: numpy>=1.21
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: torch
Requires-Dist: torch>=2.0; extra == 'torch'
Requires-Dist: transformers>=4.40; extra == 'torch'
Description-Content-Type: text/markdown

# GASP

[![PyPI](https://img.shields.io/pypi/v/gasp-rag.svg)](https://pypi.org/project/gasp-rag/)
[![Python](https://img.shields.io/pypi/pyversions/gasp-rag.svg)](https://pypi.org/project/gasp-rag/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Grounding-Aware Sensitivity by Perturbation** — a span-level detector of ungrounded
content in retrieval-augmented generation (RAG).

GASP scores each answer sentence by its *grounding sensitivity*: the change in the
sentence's likelihood when the retrieved context is perturbed. A grounded sentence loses
much of its likelihood when its supporting passage is removed; an unsupported sentence
barely reacts. GASP needs only a probabilistic scorer, no trained verifier and no labeled
data, and it returns, for each sentence, the chunk that best supports it.

## Install

```bash
pip install gasp-rag          # core
pip install gasp-rag[torch]   # with PyTorch and transformers, needed to run a scorer
```

## Quickstart

```python
from gasp import GASP

detector = GASP("Qwen/Qwen2.5-1.5B-Instruct", k_chunks=5)

context = "..."          # the retrieved passages, as one string
answer = "..."           # the generated answer to check
question = "..."         # the query (optional)

result = detector.detect(context=context, answer=answer, query=question)

for s in result:
    print(f"{s.sensitivity:+.2f}  {s.text}")
    if s.supporting_chunk:
        print(f"        supported by: {s.supporting_chunk[:80]}...")
```

Higher sensitivity means the sentence depends more on the retrieved evidence and is more
likely grounded. Lower sensitivity means it barely reacts to removing evidence and is more
likely unsupported. To flag sentences, pass a threshold:

```python
result = detector.detect(context, answer, threshold=0.5)
for s in result.flagged():
    print("likely unsupported:", s.text)
```

Thresholds are corpus dependent and are best calibrated on held-out data; the continuous
`sensitivity` score is the primary output.

## Options

Everything is configurable on the detector:

```python
GASP(
    model_id,                       # any Hugging Face causal LM, small or large, CPU or GPU
    k_chunks=5,                     # number of context chunks
    threshold=None,                 # flag sentences below this sensitivity
    economical=False,               # two-pass variant: faster, no attribution
    sensitivity_feature="max_drop", # or "gap", "mean_drop", "top2_drop", "max_jsd"
    max_ctx_tokens=1800,            # context truncation
    max_ans_tokens=256,             # answer truncation
    device=None,                    # "cpu" or "cuda"
    dtype=None,                     # "float16", "bfloat16", "float32"
)
```

Change the scorer at any time by constructing a new detector with a different `model_id`.

## Many answers and files

Score a list of answers, or read a `.jsonl`/`.csv` file:

```python
items = [
    {"context": ctx1, "answer": ans1, "query": q1},
    {"context": ctx2, "answer": ans2},
]
detections = detector.detect_batch(items, threshold=0.5)

# results as plain dicts, ready for pandas or JSON
import pandas as pd
rows = [r for d in detections for r in d.to_records()]
df = pd.DataFrame(rows)
print(detections[0].summary())   # {'n_sentences': ..., 'n_flagged': ..., 'mean_sensitivity': ...}
```

## Command line

No code needed to score a file of RAG outputs:

```bash
# input.jsonl: one object per line with "context", "answer", and optional "query"
gasp detect --input outputs.jsonl --model Qwen/Qwen2.5-1.5B-Instruct \
            --k-chunks 5 --threshold 0.5 --output results.jsonl

# fast two-pass variant on CPU
gasp detect --input outputs.jsonl --model Qwen/Qwen2.5-0.5B-Instruct \
            --economical --device cpu --output results.jsonl
```

## Metrics

If you have reference labels (`1` for an unsupported span, `0` for a grounded one), score
the detector with the full metric suite, ROC-AUC, PR-AUC, and point metrics at a threshold:

```python
from gasp import evaluate
m = evaluate(labels, sensitivities_negated, threshold=None)   # {'roc_auc': ..., 'pr_auc': ...}
```

Or from the command line, on a results file that carries a `label` column:

```bash
gasp eval --input labeled_results.csv --label-col label --score-col sensitivity --threshold 0.5
```

## How it works

For each answer sentence GASP re-scores the fixed answer under three conditions, the full
context, no context, and each context chunk removed in turn, and reads the log-likelihood
drops and Jensen-Shannon divergences at the sentence's tokens. The largest per-chunk drop
is the sentence's grounding sensitivity, and the chunk that produced it is returned as the
candidate supporting passage. Every method sees the same character-span chunks and
sentences, so the segmentation is defined once and never re-tokenized.

## API

- `GASP(model_id, k_chunks=5, threshold=None, economical=False, sensitivity_feature="max_drop", ...)` — the detector.
- `GASP.detect(context, answer, query="", threshold=None) -> Detection` — score one answer.
- `GASP.detect_batch(items, threshold=None) -> list[Detection]` — score many answers.
- `Detection` — iterable of `SentenceResult`, with `.flagged()`, `.to_records()`, `.summary()`.
- `SentenceResult` — `index`, `text`, `sensitivity`, `supporting_chunk`,
  `supporting_chunk_index`, `features`, `flagged`, `.to_dict()`.
- `evaluate`, `roc_auc`, `pr_auc`, `threshold_metrics` — evaluation metrics.
- `read_items`, `write_records` — read a `.jsonl`/`.csv` of items, write result records.
- `Scorer` — the lower-level scorer, if you want the raw per-sentence features.
- `Case`, `sentence_spans`, `chunk_spans` — the canonical segmentation, usable without a model.

## Reproducing the evaluation

The experiment pipeline, the corrected source-level results, the figures, and the human study
live in the project repository at [github.com/drbouke/GASP](https://github.com/drbouke/GASP)
(see `pipeline/` and `results/`). In short, GASP beats entailment and attribution baselines and
matches the per-chunk trained verifiers, while a full-context fact-checker and an LLM judge rank
spans more accurately at higher compute. Adding GASP to a verifier helps the weaker entailment,
attribution, and per-chunk verifiers but not the strongest full-context fact-checker or the LLM
judge, so it is best used as a cheap, training-free standalone detector with built-in
attribution and as a complement to weaker verifiers.

## Citation

If you use GASP, please cite:

```bibtex
@article{bouke2026gasp,
  title   = {Grounding-Aware Sensitivity by Perturbation for span-level hallucination
             detection in retrieval-augmented generation},
  author  = {Bouke, Mohamed Aly},
  year    = {2026},
  note    = {Preprint}
}
```

## License

MIT. See [LICENSE](LICENSE).
