Metadata-Version: 2.4
Name: ragxray
Version: 0.1.0
Summary: A lightweight RAG debugging toolkit: understand why your RAG application gave a bad answer.
Author: RAGXRay Contributors
License: MIT
Project-URL: Homepage, https://pypi.org/project/ragxray/
Keywords: rag,llm,debugging,retrieval-augmented-generation,evaluation,diagnostics
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.0
Requires-Dist: numpy>=1.24
Requires-Dist: scikit-learn>=1.3
Provides-Extra: rich
Requires-Dist: rich>=13.0; extra == "rich"
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Dynamic: license-file

# RAGXRay

**RAGXRay** is a lightweight RAG debugging toolkit that answers one question:

> **Why did my RAG application give this answer?**

It takes the `query`, the generated `answer`, and the retrieved `contexts` from
your RAG pipeline and produces an explainable diagnostic report: scores for
retrieval quality, context sufficiency, and answer grounding; a single primary
root cause; and concrete recommendations for fixing it.

RAGXRay is **not** a RAG framework, vector database, LLM wrapper, or
dashboard. It doesn't call any LLM, download any embedding model, or require
an API key. Everything runs offline using explainable heuristics (TF-IDF
similarity, token overlap, and pattern matching for numbers/dates).

## Install

```bash
pip install -e .
```

(Once published: `pip install ragxray`.)

## Quickstart

```python
from ragxray import diagnose

report = diagnose(
    query="What is the waiting period for pre-existing diseases?",
    answer="The waiting period is 48 months.",
    contexts=[
        "Pre-existing diseases are covered after 36 months.",
        "Other conditions have a waiting period of 24 months.",
    ],
)

report.show()
```

```text
RAGXRAY DIAGNOSTIC REPORT

Overall Score: 58/100
Status: Poor

Scores:
Retrieval Quality: 94/100
Context Sufficiency: 100/100
Answer Grounding: 0/100

Primary Root Cause:
UNSUPPORTED_ANSWER
(heuristic confidence: 0.80 - not a statistical probability)

Issues:
- Answer mentions a number not found in the matched context: The waiting period is 48 months.
- The answer is not supported by the retrieved evidence.

Recommendations:
- Add grounding checks and restrict generation to retrieved evidence.
- Return "I don't know" when evidence is insufficient.
```

## What it checks

| Analyzer | Question it answers |
|---|---|
| **Retrieval Quality** | Are the retrieved contexts relevant to the query? |
| **Context Sufficiency** | Is there enough information in the contexts to answer the query? |
| **Answer Grounding** | Is the generated answer actually supported by the retrieved contexts? |
| **Root Cause Engine** | What's the single most likely reason for a bad answer? |
| **Recommendations** | What should you change in your pipeline? |

### Root causes

RAGXRay always names exactly one primary root cause:

```text
RETRIEVAL_FAILURE       - retrieval found little/nothing relevant
LOW_RETRIEVAL_RELEVANCE - retrieval is weak but not a total failure
INSUFFICIENT_CONTEXT    - the retrieved contexts don't contain the needed facts
UNSUPPORTED_ANSWER      - the answer states things the context doesn't support
CONTRADICTORY_ANSWER    - the answer directly contradicts the retrieved evidence
UNKNOWN                 - no significant issue detected by current heuristics
```

> **Note:** `confidence` on the root cause is a **heuristic diagnostic score**
> in `[0, 1]`, not a calibrated statistical probability. It reflects how
> strongly the detected signals point at the chosen root cause.

## Working with the report

```python
report.score               # 0-100 overall score
report.status               # "Excellent" | "Good" | "Needs Attention" | "Poor" | "Critical"
report.retrieval_score
report.context_score
report.grounding_score
report.primary_root_cause   # RootCause enum
report.issues               # list[str] of detected problems
report.recommendations      # list[str] of actionable next steps

report.to_dict()            # plain JSON-serializable dict
report.to_json()            # JSON string
report.show()               # pretty-print to the terminal
```

Each analyzer's full detail is also available:

```python
report.retrieval_analysis.per_context     # per-context relevance scores
report.context_analysis.missing_terms     # query terms not found anywhere
report.grounding_analysis.unsupported_claims  # specific unsupported/contradicted claims
```

## Context input formats

Plain strings:

```python
contexts = ["Some text", "Another text"]
```

Structured, with metadata:

```python
contexts = [
    {
        "text": "Pre-existing diseases are covered after 36 months.",
        "source": "policy.pdf",
        "page": 12,
    }
]
```

Both (and a mix of the two) are normalized internally into a `Context` model.

## Scoring

```text
Overall Score = 30% Retrieval Quality
              + 30% Context Sufficiency
              + 40% Answer Grounding

90-100  Excellent
75-89   Good
60-74   Needs Attention
40-59   Poor
0-39    Critical
```

All weights and thresholds are configurable via `RAGXRayConfig`:

```python
from ragxray import diagnose, RAGXRayConfig

config = RAGXRayConfig(
    weight_retrieval=0.2,
    weight_context=0.2,
    weight_grounding=0.6,
    grounding_sentence_support_threshold=0.4,
)

report = diagnose(query=..., answer=..., contexts=..., config=config)
```

## How the heuristics work (v1)

- **Retrieval Quality**: TF-IDF cosine similarity between the query and each
  context (falls back to token overlap if TF-IDF's vocabulary is empty),
  rescaled against a calibrated "strong match" reference point.
- **Context Sufficiency**: fraction of the query's meaningful (non-stopword)
  terms that appear anywhere in the retrieved contexts.
- **Answer Grounding**: each answer sentence is matched against the most
  similar context sentence. If nothing is similar enough, the claim is
  unsupported. If something *is* similar but numbers/dates disagree, that's
  flagged as unsupported with the specific mismatch. Explicit negation
  clashes (e.g. "is covered" vs. "is not covered") are flagged as direct
  contradictions.

These are **explainable heuristics, not semantic entailment or fact-checking
models** - RAGXRay tells you *what* looks wrong and *why* it flagged it, but
it can't verify real-world truth, and it can miss issues heuristics simply
aren't built to catch (e.g. subtle logical errors phrased without numbers or
negation words).

## What's intentionally NOT in v1

- No citation-level analysis
- No chunk-level analysis
- No LangChain/framework integrations
- No LLM-based evaluation
- No embeddings requiring model downloads

These may come in future versions - v1 is deliberately focused and offline.

## Development

```bash
pip install -e ".[dev]"
pytest
```

## License

MIT
