Metadata-Version: 2.4
Name: veralith
Version: 0.2.7
Summary: Hallucination diagnosis for RAG systems — Sufficiency, Faithfulness, Completeness verdicts plus rule-based remediation.
Author: Srijan Shekhar, Kaustav Dasgupta
License: MIT
Project-URL: Homepage, https://github.com/SrijanShekhar21/VeralithAI
Project-URL: Repository, https://github.com/SrijanShekhar21/VeralithAI
Project-URL: Issues, https://github.com/SrijanShekhar21/VeralithAI/issues
Keywords: rag,llm,evaluation,hallucination,openai,langchain,observability,agents,veralith
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openai>=1.40.0
Requires-Dist: pydantic>=2.6
Requires-Dist: python-dotenv>=1.0
Requires-Dist: tenacity>=8.2
Requires-Dist: tiktoken>=0.7.0
Requires-Dist: httpx>=0.27
Provides-Extra: langchain
Requires-Dist: langchain>=0.1.0; extra == "langchain"
Provides-Extra: sample
Requires-Dist: chromadb>=0.5.0; extra == "sample"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Dynamic: license-file

# Veralith

Hallucination diagnosis for RAG agents. For every `(query, context, response)`
trace, get a typed failure diagnosis and a concrete fix — evaluated on
Veralith's hosted API and streamed to your dashboard at
[app.veralithai.com](https://app.veralithai.com), or run fully offline.

Python 3.10+ · hosted or offline evaluation · non-blocking, fail-safe SDK.

## Installation

```bash
pip install veralith
```

## Quick start

```bash
export VERALITH_API_KEY=vk_live_...   # app.veralithai.com → project → API keys
```

```python
import veralith

def answer(query: str) -> str:
    chunks = my_retriever(query)
    response = my_generator(query, chunks)
    veralith.log(query=query, context=chunks, response=response)
    return response
```

`log()` enqueues the trace and returns immediately; evaluation runs on
Veralith's servers.

## API

### `veralith.log(query, context, response, latency_ms=None) -> None`

Enqueue a trace for server-side evaluation. Non-blocking.

| Param | Type | Notes |
|---|---|---|
| `query` | `str` | the user question |
| `context` | `str \| list[str] \| list[dict] \| list[ContextChunk]` | retrieved chunks — plain strings, `{"text": ...}` dicts, or `ContextChunk`s |
| `response` | `str` | the generated answer |
| `latency_ms` | `float \| None` | optional RAG response time, surfaced on the dashboard |

Requires `VERALITH_API_KEY`. If it is unset, `log()` is a **no-op** (warns once
per process) so the same code is safe to run in tests and local dev.

### `@veralith.trace`

Decorator alternative — capture `(response, context)` from the return value:

```python
import veralith

@veralith.trace
def rag(query: str):
    chunks = my_retriever(query)
    response = my_generator(query, chunks)
    return response, chunks          # (response, context)
```

Callers of `rag(query)` receive just `response`: the decorator strips the tuple,
logs the trace, and returns the response. The query is read from the first
positional argument or a `query=` keyword. `async` functions are supported.

When returning a bare tuple is awkward, return a `TraceReturn`:

```python
from veralith import trace, TraceReturn

@trace
def rag(user_question: str):
    ...
    return TraceReturn(response=answer, context=chunks)
```

### LangChain adapter

Auto-trace `RetrievalQA` chains with no call-site changes:

```python
import veralith.adapters.langchain as adapter
adapter.install()
# every RetrievalQA.invoke(...) now logs a trace
```

### `veralith.evaluate(query, context, response, persist=False) -> EvaluationResult`

Run the **full evaluation locally** — no account, no traffic to Veralith. Uses
your `OPENAI_API_KEY`. Intended for CI, prompt tuning, and air-gapped use.

```python
result = veralith.evaluate(
    query="What is a P/E ratio?",
    context=["Price-to-earnings ratio is share price / earnings per share."],
    response="A P/E ratio is share price divided by earnings per share.",
    persist=False,
)
print(result.diagnosis.failure_cell.value)   # 'complete_grounded'
```

Use `log()` in production and `evaluate()` in tests.

### `veralith.shutdown(wait=True) -> None`

Flush and stop the background worker. Registered via `atexit`, so long-running
apps rarely call it; use it in short scripts or tests to join pending traces
cleanly.

## Behavior

The SDK is designed to sit in a hot request path without risk:

- **Non-blocking** — `log()` / `@trace` enqueue onto an in-process
  `ThreadPoolExecutor` (`VERALITH_WORKER_CONCURRENCY`, default `4`) and return
  immediately.
- **Fail-safe** — a Veralith outage, network error, or malformed response is
  swallowed and warned, never raised into your call path.
- **No-op without a key** — with `VERALITH_API_KEY` unset the SDK does nothing
  (warns once). No feature flags, nothing to strip before prod.
- **Backpressure** — if the worker pool is saturated the trace is dropped
  (warned once) rather than blocking your app.
- **Clean shutdown** — an `atexit` hook flushes queued traces on exit.

Set `VERALITH_DEFAULT_SYNC=1` to send synchronously (blocking) instead — useful
in serverless runtimes where background threads may not flush before freeze.

## Failure cells

Each evaluated trace lands in one cell of Completeness × Faithfulness. Cell
names read `<completeness>_<faithfulness>`:

| | Grounded (claims supported) | Ungrounded (a claim invented) |
|---|---|---|
| **Complete** | `complete_grounded` | `complete_ungrounded` |
| **Incomplete** | `incomplete_grounded` | `incomplete_ungrounded` |
| **Extra** | `extra_grounded` | `extra_ungrounded` |

`complete_grounded` is healthy; `incomplete_ungrounded` (missed part of the
query **and** invented a claim) is the worst case. Each cell maps to a concrete
suggestion — lower temperature, raise retrieval-K, tighten the generator prompt,
fix a chunk boundary, etc. A per-trace sufficiency level (`HIGH`/`LOW`) is
calibrated per knowledge base from the distribution of healthy traces.

## Self-heal (via MCP, not this package)

Diagnosis is where this SDK stops — the fix loop lives on the platform. When
failing traces cluster into a recurring pattern, Veralith opens a *heal card*.
Point a coding agent (Claude Code, Codex, or Cursor) at Veralith's MCP server
and it reads the diagnosis plus your actual RAG code and opens a pull request:

```bash
claude mcp add --transport http veralith \
  https://api.veralithai.com/mcp/http \
  --header "Authorization: Bearer vk_live_..."
```

This runs through your agent over MCP; the `veralith` pip package itself only
handles instrumentation and evaluation. See
[docs.veralithai.com](https://docs.veralithai.com) for the full loop.

## The result object

`evaluate()` returns a typed `EvaluationResult` (all Pydantic models):

```python
class EvaluationResult:
    trace_id: int
    query: str
    sub_questions: list[SubQuestion]         # decomposed query
    claims: list[Claim]                      # decomposed response
    sufficiency: list[SufficiencyJudgment]   # per sub-question
    faithfulness: list[FaithfulnessJudgment] # per claim (+ grounding chunks)
    completeness: CompletenessJudgment | None
    diagnosis: Diagnosis | None              # failure_cell + sufficiency level + counts
    suggestion: Suggestion                   # title + body + steps
    latency_ms: dict[str, float]             # per-phase wall-clock timing
    errors: dict[str, str]                   # per-metric failures, if any
    created_at: datetime
```

## Configuration

| Variable | Default | Scope |
|---|---|---|
| `VERALITH_API_KEY` | — | **required** for `log()` / `@trace` |
| `VERALITH_API_URL` | `https://api.veralithai.com` | transport endpoint (override for self-host / staging) |
| `VERALITH_WORKER_CONCURRENCY` | `4` | background evaluation threads |
| `VERALITH_DEFAULT_SYNC` | `false` | send synchronously instead of in the background |
| `OPENAI_API_KEY` | — | offline `evaluate()` only |
| `VERALITH_JUDGE_MODEL` | `gpt-4o` | offline `evaluate()` judges |
| `VERALITH_DECOMPOSER_MODEL` | `gpt-4o-mini` | offline `evaluate()` decomposition |
| `VERALITH_EMBED_MODEL` | `text-embedding-3-small` | offline `evaluate()` embeddings |

Hosted `log()` evaluation runs on Veralith's own model keys and counts against
your project's monthly trace quota. Offline `evaluate()` runs on your own
`OPENAI_API_KEY`.

## Links

- Dashboard — https://app.veralithai.com
- Docs — https://docs.veralithai.com
- Source — https://github.com/SrijanShekhar21/VeralithAI
- Issues — https://github.com/SrijanShekhar21/VeralithAI/issues

## License

MIT — see [LICENSE](LICENSE).
