Metadata-Version: 2.5
Name: searchunify-evalsuite
Version: 0.1.0
Summary: Evaluate model calls against the material they were given
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: evaluation,faithfulness,groundedness,hallucination,langfuse,llm,observability,rag
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: langchain-anthropic>=1.0
Requires-Dist: langchain-aws>=1.0
Requires-Dist: langchain-google-genai>=4.0
Requires-Dist: langchain-openai>=1.0
Requires-Dist: langchain>=1.0
Requires-Dist: motifer>=1.0
Provides-Extra: all
Requires-Dist: langchain-anthropic>=1.0; extra == 'all'
Requires-Dist: langchain-aws>=1.0; extra == 'all'
Requires-Dist: langchain-google-genai>=4.0; extra == 'all'
Requires-Dist: langchain-openai>=1.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: langchain-anthropic>=1.0; extra == 'anthropic'
Provides-Extra: aws
Requires-Dist: langchain-aws>=1.0; extra == 'aws'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: google
Requires-Dist: langchain-google-genai>=4.0; extra == 'google'
Provides-Extra: openai
Requires-Dist: langchain-openai>=1.0; extra == 'openai'
Description-Content-Type: text/markdown

# searchunify-evalsuite

Evaluate model calls against the material they were given.

`evalkit` scores an LLM's output for **faithfulness** — the proportion of its
factual claims that the context actually supports. It reads traces from an
observability vendor, finds every model call inside them, judges each one with
a second model, and writes the score back.

The unit of evaluation is **one model call**, not one conversation. An agent
that retrieves, reasons, calls a tool and answers is four calls, scored
separately — which is what turns *"this run scored 0.7"* into *"the `agent`
node is ungrounded."*

---

## Install

```bash
pip install searchunify-evalsuite          # everything, every judge
pip install 'searchunify-evalsuite[dev]'   # + pytest, ruff
```

Installed as **`searchunify-evalsuite`**, imported as **`evalkit`**:

```python
import evalkit
```

The two names differ because `evalkit` on public PyPI is an unrelated project,
so the distribution cannot be published under it. The import name is the
original one and is not changing.

Python 3.11+. One install covers every judge: the core pulls `httpx`,
`motifer`, `langchain` and the LangChain package for each model family in
`MODEL_CONTEXT_TOKENS` — `langchain-openai`, `langchain-anthropic`,
`langchain-google-genai`, `langchain-aws`. Heavier than any one deployment
needs, and deliberately so: a provider package missing from the install is not
caught until `init_eval`, on someone else's machine.

The `openai`, `anthropic`, `google`, `aws` and `all` extras still exist so
older install commands keep working, but they add nothing the core does not
already have.

---

## Quick start

```python
import asyncio
from evalkit import init_eval

evaluator = init_eval({
    "provider":  {"name": "langfuse",
                  "credentials": {"host": "https://cloud.langfuse.com",
                                  "public_key": "pk-...",
                                  "secret_key": "sk-..."}},
    "judge":     {"model": "gpt-5-mini", "api_key": "sk-..."},
    "evaluator": "ragas:faithfulness",
})

async def main():
    result = await evaluator.evaluate(hours=24, limit=50)
    print(result.render())
    await evaluator.close()

asyncio.run(main())
```

**Nothing is read from the environment.** Every credential arrives as an
explicit argument, which makes it structurally impossible for the library to
pick up a key the caller did not hand it.

---

## Configuration

`init_eval` takes one dict. Every part is optional — each is checked by the
call that needs it, not at construction:

```python
init_eval({
    "provider":  {"name": "langfuse", "credentials": {...}},   # where traces come from
    "judge":     {"model": "gpt-5-mini", "api_key": "..."},    # who scores them
    "evaluator": "ragas:faithfulness",                         # or a list
    "sink":      {"name": "langfuse", "credentials": {...}},   # or a list, or omitted
    "run":       {"sample_rate": 0.2, "concurrency": 8, "max_traces": 200},
})
```

| Call | Needs |
| --- | --- |
| `score()` | judge + evaluator |
| `evaluate()` | judge + evaluator + provider |
| writing back | a sink, and `write_back=True` |

Any of the four parts may instead be an **already-built object**, which is how
a caller supplies their own implementation or reuses one across clients.

Nothing is contacted during `init_eval`. Credentials are not verified, so a bad
key surfaces on the first real request rather than at startup. What *is*
checked is shape — a missing part, an unknown name, arguments a constructor
will not accept — since those are wrong whatever the credentials are.

### Choosing a judge

LangChain resolves the model string to whoever owns it, so moving between
families is a configuration change and nothing else:

```python
{"judge": {"model": "gpt-5-mini", "api_key": "..."}}
{"judge": {"model": "claude-sonnet-4-5", "model_provider": "anthropic", "api_key": "..."}}
{"judge": {"model": "gemini-2.5-flash", "model_provider": "google_genai", "api_key": "..."}}
{"judge": {"model": "us.anthropic.claude-sonnet-4-5-20250929-v1:0",
           "model_provider": "bedrock_converse", "region_name": "us-east-1"}}
{"judge": {"model": "my-deployment", "model_provider": "azure_openai",
           "api_key": "...", "azure_endpoint": "https://....openai.azure.com",
           "api_version": "2024-10-21"}}
```

Anything else in the judge dict is passed through to the underlying chat model,
which is how region, endpoint and deployment reach the provider that needs them.

One option is worth knowing about. `structured_method` controls how the judge is
asked to return JSON, and defaults to `"json_schema"` — the strictest mechanism
where a provider offers it:

```python
{"judge": {"model": ..., "structured_method": "function_calling"}}
```

Support is uneven outside OpenAI. On Bedrock several models reject `json_schema`
outright, and a few accept it and return a schema-valid **empty** object — which
reads downstream as an answer asserting nothing rather than as a failure.
`function_calling` goes through tool use, is supported far more widely, and
never parses free text. Reach for it when a model returns no claims at all.

### Choosing what counts as context

The Langfuse provider takes `context_from`, and its default is the single most
consequential setting in the library:

```python
{"provider": {"name": "langfuse", "credentials": {...},
              "context_from": "auto"}}   # default
```

With `auto`, the context is wherever the retrieved material actually sits:

| The model call | Context |
|---|---|
| An agent — the prompt holds tool results | The **tool results** only |
| A chain that pasted documents into the prompt — no tool results | The **system and user** messages |

An earlier assistant turn is never context, in either case. It is the model's
own output — its summary of what a search returned, its working — and grounding
a claim in it is circular: one hallucination would stand as evidence for the
next, and a model that invents a fact and then repeats it would score as
perfectly faithful.

Every sample records which rule applied in `meta["context_source"]`
(`tool_results`, `prompt` or `input`). When an app pastes its chat history into
the user message as text, the role rule cannot separate the model's earlier
answers from the documents; such samples carry
`meta["context_includes_history"] = True` so they can be told apart.

`context_from="tool_results"` is the strict form: a call with no tool results is
skipped rather than judged against its prompt. `context_from="input"` treats the
whole prompt as groundable, earlier assistant turns included. It scores higher,
and means less.

---

## API

Every method is async. A caller that is not already async wraps one call in
`asyncio.run(...)`.

### `evaluate(**kwargs) -> RunResult`

Evaluate a window of traces.

```python
await evaluator.evaluate(
    hours=24,                 # or since=/until=
    tenant_id="acme",
    tags=["production"],
    limit=100,
    sample_rate=0.2,          # random, not "the first N"
    write_back=True,          # off by default
)
```

`write_back` is **off by default**, so a run never posts scores onto production
traces unless asked.

### `evaluate_trace(trace_id) -> RunResult`

One trace, every model call on it. No listing, no sampling — the path for
asking why a particular conversation scored the way it did.

### `score(context, answer) -> list[MetricResult]`

Score text directly. No provider involved, so a caller who already holds the
prompt and the answer — or has no observability vendor at all — can use the
same metric.

```python
results = await evaluator.score(
    context="The licence costs $50 per user per month.",
    answer="The licence costs $50 per user.",
)
```

### `close()`

Releases the pooled HTTP connections held by the provider, judge and sinks.

**Never called for you.** A client holds connections a long-lived process
should keep open across many runs, so the decision to drop them belongs to
whoever built the client — finishing a run closes nothing.

It closes every part it holds, including one you built yourself and passed in.
If you share a provider or a judge between clients, close it yourself rather
than closing a client that borrowed it.

---

## Results

```python
result = await evaluator.evaluate(hours=24)

result.summary["evaluated"]       # 48   model calls, not result rows
result.summary["skipped"]         # 12
result.summary["skip_classes"]    # {'by_design': 9, 'instrumentation_defect': 3}

faith = result.summary["metrics"]["ragas:faithfulness"]
faith["claim_level"]              # 0.7273  <- headline: supported / total claims
faith["mean_call"]                # 0.6333
faith["hallucination_rate"]       # 0.4167
faith["by_node"]                  # per-node scores: which step is ungrounded

result.results                    # list[MetricResult], one per model call per metric
result.skips                      # list[SkipRecord], with a reason each
print(result.render())            # formatted for a terminal
```

**Metrics are never pooled.** Every rate lives under the metric that produced
it, because averaging faithfulness with answer relevancy produces a number
describing neither, and a low relevancy score is not a hallucination. Run-level
figures — `evaluated`, `skipped`, `errors` — are counted in model calls, so one
call scored by two metrics is one call.

Each result carries its own identity — `trace_id`, `node_id`, `node_name` —
filled by the runner rather than the metric, so per-node reporting works for a
metric whose author never thought about it.

`score` is **`None`** whenever the metric is undefined — nothing was asserted,
a required field was missing, the judge failed. It is never `0` for those
cases: `0` means *"measured, and nothing was grounded"*, which is a far
stronger statement than *"could not measure"*.

Skips are classified, never collapsed into one number:

| Class | Meaning |
| --- | --- |
| `by_design` | Permanent and correct — the call invoked a tool, or asserted nothing |
| `instrumentation_defect` | Fixable — an answer was produced but nothing was recorded |
| `transient` | Retryable — a malformed payload, a judge timeout, a rate limit |

Only one of the three is a fixed ceiling. Collapsing them would make a
recording bug indistinguishable from the natural shape of the traffic.

---

## Logging

A run over a few hundred traces takes a while, so it reports progress as it
goes — traces collected, calls evaluated, and what reached the sink:

```
run 9c2f8b1e… starting
run 9c2f8b1e…: 300 trace(s) selected
run 9c2f8b1e…: collected 25/300 trace(s)
run 9c2f8b1e…: 412 model call(s) evaluable, 38 skipped at extraction
run 9c2f8b1e…: evaluated 50/412
run 9c2f8b1e…: 388 score(s) written to 1 sink(s), 12 declined, 0 failed
run 9c2f8b1e… completed: 412 scored, 38 skipped, 0 error(s)
```

Nothing is emitted until logging is configured. This library attaches no
handlers on import — the application owns that — so an app with its own
`logging.basicConfig` sees these lines with no further work.

To route this package's output through motifer instead:

```python
from evalkit.log import configure
configure(service="evalkit", level="info")
```

All loggers sit under the `evalkit` root, so `logging.getLogger("evalkit")`
is the one handle for level and handlers.

---

## What ships

```python
from evalkit import list_metrics, list_providers, list_sinks
```

| Role | Name | Notes |
| --- | --- | --- |
| Evaluator | `ragas:faithfulness` | Requires `context`, `answer` |
| Provider | `langfuse` | Langfuse traces over its public REST API |
| Sink | `langfuse` | Scores onto the originating trace **and** node |

Each is looked up by string name and imported lazily, so a caller never pays
the import cost of a part they do not use.

---

## How faithfulness is computed

1. **Decompose** — the answer is broken into atomic, pronoun-free claims.
2. **Verify** — each claim is checked against the context it was given.
3. **Divide** — `score = supported / total`, computed in Python.

The judge is never asked for a number. It makes per-claim yes/no judgements
with a reason, and the arithmetic happens here — so every score decomposes into
claims you can read.

### Context that does not fit

A request is `system prompt + context + claims + response`, and either the
context or the claim list can push it over the judge's window. Three tiers,
tried in order:

| | Situation | Response |
| --- | --- | --- |
| 1 | everything fits | one call, whole context, all claims |
| 2 | the claim list overflows | batch the claims, keep the context whole — **lossless** |
| 3 | the context overflows | split with 10% overlap, batch claims alongside — **lossy** |

Only the third loses anything: a claim drawing on two separate passages is
supported by neither half alone, so it can be reported unsupported. When that
happens the result records `detail["context_was_split"] = True`.

**Truncation is never an option.** In testing, a budget set below the model's
own window left the judge seeing only the head of a prompt — all system
instructions, no retrieved material — and a sound answer scored 0.0.

The budget comes from the judge's own model (`defaults.MODEL_CONTEXT_TOKENS`),
falling back to 120,000 tokens for a model that is not listed.

---

## Failure handling

Failures are isolated per model call. One unparseable trace or one judge
timeout reduces a run's coverage and is recorded as such — it never fails the
run, which would throw away every result already paid for.

- A trace that cannot be read becomes a `SkipRecord`, and the run continues.
- A judge failure becomes `MetricResult(status=ERROR, score=None)` for that one
  call.
- A sink that cannot be written to costs its own record, never the score — the
  result is appended to the run **before** any sink is tried.
- Permanent failures are not retried. A bad key or an unknown model fails
  identically every time, so it fails immediately rather than burning the
  backoff window to reach the same answer.

---

## Extending

Everything swappable is a `Protocol` — structural, so an implementation needs
the right methods and no base class.

```python
class MyScoreWriter:
    name = "mine"

    async def write(self, sample, result) -> bool:
        """Raise if a result that should have been written was not.

        Return False only when there was deliberately nothing to write.
        """

evaluator = init_eval({..., "sink": MyScoreWriter()})
```

| Protocol | Contract |
| --- | --- |
| `TraceProvider` | `list_traces`, `get_trace`, `to_samples` |
| `Judge` | `complete(system, user, schema, schema_name)` |
| `Evaluator` | `name`, `requires`, `evaluate(sample)` |
| `ScoreWriter` | `write(sample, result) -> bool`, raising on a failed write |

To add one that ships with the library: a directory under the relevant
`*_factory/`, and one entry in that factory's `constants.py`. Nothing else in
the library changes.

---

## Scope

Deliberately single-purpose. Scheduling, deduplication, alerting and retention
belong to whatever calls this — the library evaluates and returns, and writes
only where it is told to.

---

## Licence

[Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0). The full text ships
with the package as `LICENSE`.
