Metadata-Version: 2.4
Name: typed_evals
Version: 0.2.1
Summary: Evaluation for LLM and Agents using System One Models, with opt-in calibration.
Author: Aaryan Verma
License-Expression: MIT
Project-URL: Author, https://github.com/Aaryanverma
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typesafe-sdk<0.8,>=0.7.0
Requires-Dist: pydantic<3,>=2.12
Provides-Extra: openai
Requires-Dist: openai<4,>=3.26.0; extra == "openai"
Provides-Extra: calibration
Requires-Dist: scikit-learn<2,>=1.4; extra == "calibration"
Provides-Extra: langchain
Requires-Dist: langchain<2,>=1; extra == "langchain"
Provides-Extra: crewai
Requires-Dist: crewai<2,>=1.15.22; extra == "crewai"
Provides-Extra: agent-framework
Requires-Dist: agent-framework-core<2,>=1.19; extra == "agent-framework"
Provides-Extra: dev
Requires-Dist: pytest<10,>=8; extra == "dev"
Requires-Dist: pytest-asyncio<2,>=0.24; extra == "dev"
Requires-Dist: pytest-cov<8,>=5; extra == "dev"
Requires-Dist: ruff>=0.9; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Dynamic: license-file

<h1 align="center">Typed Evals</h1>

<p align="center">
  <strong>Evaluate responses. Guard actions.</strong><br>
  A Python toolkit for evaluating LLMs, RAG, and Agents using Decision Models with optional calibration against human labels
</p>

<p align="center">
  <a href="https://pypi.org/project/typed-evals/"><img src="https://img.shields.io/pypi/v/typed-evals?style=flat-square&amp;color=0f766e" alt="PyPI version"></a>
  <a href="https://pypi.org/project/typed-evals/"><img src="https://img.shields.io/badge/python-3.11%2B-2563eb?style=flat-square" alt="Python 3.11 and later"></a>
  <a href="https://github.com/TrustifAI/typed_evals/actions/workflows/test.yml"><img src="https://github.com/TrustifAI/typed_evals/actions/workflows/test.yml/badge.svg?branch=main" alt="Tests"></a>
  <a href="https://github.com/TrustifAI/typed_evals/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-64748b?style=flat-square" alt="MIT license"></a>
</p>

<p align="center">
  <a href="#quickstart">Quickstart</a> ·
  <a href="#evaluate-agent-execution">Agent evaluation</a> ·
  <a href="#guard-tool-calls">Tool guards</a> ·
  <a href="#calibration">Calibration</a> ·
  <a href="#examples-and-guides">Examples &amp; guides</a>
</p>

<p align="center">
  <img src="https://raw.githubusercontent.com/TrustifAI/typed_evals/main/docs/assets/readme-banner.svg" width="1200" alt="Know what passed. Decide what runs. Typed Evals takes your evidence through a metric panel and returns typed scores, thresholds, and status.">
</p>

Use **[Jev](https://typesafe.ai/)** by default, **[Microsoft Decision-1 on OpenRouter](https://openrouter.ai/microsoft/microsoft-decision-1)** through the same backend, or native **[OpenAI Decisions API](https://developers.openai.com/api/docs/guides/decisions)**, to judge generated responses and recorded agent executions. Check proposed tool calls before they run. Start with a preset, then bring your own metrics, thresholds, or judge backend as your application grows.

## Quickstart

Install with **Python 3.11+**. This quickstart uses the default Jev backend, which requires a [TypeSafe](https://typesafe.ai/) API key. For Microsoft Decision-1, follow [Use Microsoft Decision-1 on OpenRouter](#use-microsoft-decision-1-on-openrouter) with `OPENROUTER_API_KEY`. For OpenAI Decisions, set `OPENAI_API_KEY` and follow [Use OpenAI Decisions](#use-openai-decisions).

```bash
python -m pip install typed-evals
export TYPESAFE_API_KEY='your-key' # or export OPENAI_API_KEY='your_key'
```

Evaluate a response against the evidence used to generate it:

```python
from typed_evals import evaluate

result = evaluate(
    input="What is the refund period?",
    response="You can request a refund within 30 days.",
    contexts=["Refunds are allowed within 30 days of purchase."],
    preset="rag",
)

print("All checks passed:", result.passed)
for name, metric in result.metrics.items():
    print(f"{name}: score={metric.score}, passed={metric.passed}")
```

That checks **faithfulness**, **answer relevancy**, and **context relevance** in one judge request. The default backend uses `jev-1.13.0` through the official `typesafe-sdk` 0.7.x; live examples make API calls using your account.

**Want to explore without an API key?** Jump to the [offline demo](#try-it-offline).

### Use Microsoft Decision-1 on OpenRouter

Microsoft's [Decision-1 model](https://openrouter.ai/microsoft/microsoft-decision-1) works through `JevBackend` with [OpenRouter's TypeSafe-compatible System One endpoint](https://openrouter.ai/docs/guides/community/typesafe-sdk). Install `python-dotenv` to load a local `.env` file:

```bash
python -m pip install typed-evals python-dotenv
export OPENROUTER_API_KEY='your-key'
```

```python
import os

from dotenv import load_dotenv

from typed_evals import JevBackend, evaluate

load_dotenv()
backend = JevBackend(
    model="microsoft/microsoft-decision-1",
    base_url="https://openrouter.ai/api",
    api_key=os.environ["OPENROUTER_API_KEY"],
)
result = evaluate(
    input="What is the refund period?",
    response="You can request a refund within 30 days.",
    preset="response",
    backend=backend,
)
print(result.passed)
```

You can also put `OPENROUTER_API_KEY=your-key` in `.env`; call `load_dotenv()` before reading it. The SDK appends `/v1/systemone`, giving `https://openrouter.ai/api/v1/systemone`. The `"response"` preset checks answer relevancy; use `"rag"` or `"agent"` with their required evidence for those workflows. Run [examples/openrouter_decision1.py](examples/openrouter_decision1.py) from a checkout for scores and pass/fail results. This uses the core TypeSafe SDK dependency and needs no additional provider SDK.

### Use OpenAI Decisions

Install the optional official SDK and set its normal environment variable:

```bash
python -m pip install 'typed-evals[openai]'
export OPENAI_API_KEY='your-key'
```

```python
from typed_evals import OpenAIDecisionsBackend, evaluate

backend = OpenAIDecisionsBackend(model="gpt-6-luna", timeout=30.0, max_retries=2)
result = evaluate(
    input="What is the refund period?",
    response="You can request a refund within 30 days.",
    contexts=["Refunds are allowed within 30 days of purchase."],
    preset="rag",
    backend=backend,
)
print(result.passed)
```

This uses `POST /v1/decisions` through `AsyncOpenAI.decisions.create` with OpenAI SDK. All active metrics share one request. Samples support text, structured tool evidence, and images. Run [examples/openai_decisions.py](examples/openai_decisions.py) from a checkout for a complete text example.

Each batch owns and closes one async client, shared by its workers. With `OpenAIDecisionsBackend(client=your_async_client)`, you own the client lifecycle and its timeout/retry settings; evaluate asynchronously in the client's event loop. The SDK handles transport retries.

Decisions maps Noul to predicate, Choice to string-valued choices, and Score to ordered levels. Raw scores keep their existing meaning: positive-condition probability, summed probability of passing options, or normalized expected ordinal level. Choice/Score provider confidence is separate; predicates have no confidence field. Refusals are unavailable judgments: they raise by default, or become individual error results under `errors="record"`, preserving successful sibling metrics and leaving the sample's pass status unavailable. Runtime guards apply their configured error policy.

### Evaluate image evidence

Add images to a sample and select them in a custom metric's `required_fields`:

```python
from typed_evals import ImageInput, Metric, OpenAIDecisionsBackend, evaluate

visual_grounding = Metric(
    name="visual_grounding",
    kind="noul",
    instructions="Does the response accurately describe the supplied image?",
    pass_definition="Every material visual claim is supported by the image.",
    required_fields=("input", "response", "images"),
)
result = evaluate(
    input="Describe this receipt.",
    response="The receipt shows a total of $42.00.",
    images=[ImageInput.from_file("receipt.png", label="receipt")],
    metrics=[visual_grounding],
    backend=OpenAIDecisionsBackend(),
)
print(result.passed)
```

`ImageInput` accepts inline PNG, JPEG, WebP, and GIF data URLs, with `from_file` and `from_bytes` helpers. Images use native Decisions image parts alongside text; image order and optional labels identify the evidence. Built-in presets select text evidence, so attaching images alone does not include them in the judge request. Jev currently supports text; image-selecting metrics require a backend advertising image support. The sample and metric APIs are shared across providers, so future Jev image support can use the same code with `backend=JevBackend()`.

See the [image guide](docs/EVALUATION.md#image-evidence) and [runnable image example](examples/openai_decisions_images.py).

## What you can build

| Your workflow | What Typed Evals provides |
| --- | --- |
| **[Evaluate a RAG pipeline](#quickstart)** | Check whether answers address the query and stay grounded in the retrieved context. |
| **[Evaluate Agent runs](#evaluate-agent-execution)** | Judge task completion and claims about tool results against recorded execution evidence. |
| **[Guard tool calls](#guard-tool-calls)** | Evaluate proposed calls against your policy before executing the underlying function. |
| **[Define your own rubric](#write-a-custom-metric)** | Compose nine built-in metrics or write custom binary, categorical, and ordered-score checks. |
| **[Calibrate thresholds](#calibration)** | Fit per-metric curves to human pass/fail labels and inspect performance on held-out data. |

### Choose a preset

| Preset | Checks | Evidence to supply |
| --- | --- | --- |
| `"response"` · default | Answer relevancy | `input`, `response` |
| `"rag"` | Faithfulness, answer relevancy, context relevance | `input`, `response`, `contexts` |
| `"agent"` | Task completion, tool grounding | `input`, `response`, `trace`, `expected_outcome` |

Presets use a `0.5` threshold for each metric. **Supplying contexts alone does not enable RAG checks**—choose `preset="rag"` explicitly. Missing evidence and judge errors raise by default. Use custom metrics to choose different checks or thresholds.

## Evaluate agent execution

Use `preset="agent"` to check whether a completed run achieved its expected outcome and reported tool results accurately. Supply the final response, observed execution trace, and explicit success criteria:

```python
from typed_evals import evaluate

result = evaluate(
    preset="agent",
    input="Create a support ticket for the export failure.",
    response="I created ticket T-42 for your export failure.",
    expected_outcome="A new support ticket exists for the reported export failure.",
    trace=[
        {
            "name": "create_ticket",
            "arguments": {"title": "Export failure"},
            "status": "error",
            "output": {"created": False},
            "error": "Permission denied",
        }
    ],
)

for name, metric in result.metrics.items():
    print(f"{name}: score={metric.score}, passed={metric.passed}")
```

- **Task completion** checks whether execution evidence establishes `expected_outcome`. Plans, attempts, and claims of success are insufficient.
- **Tool grounding** checks whether the response's claims match the observed tool results, including failed or unverified executions.

Here, ticket creation failed while the agent claimed success. Both rubrics describe a failure; the returned scores depend on the selected judge's judgment.

Capture `trace` from actual executions in your integration. The preset covers completion and result reporting; step ordering, efficiency, and recovery quality need separate checks. See the [runnable example](https://github.com/TrustifAI/typed_evals/blob/main/examples/agent_evaluation.py) and the [runtime guide](https://github.com/TrustifAI/typed_evals/blob/main/docs/RUNTIME.md) for enforcing checks at agent and tool boundaries.

## Evaluate a dataset

Save your samples as `samples.jsonl`, with one JSON object per line:

```json
{"input":"Refund period?","response":"30 days.","contexts":["Refunds are allowed within 30 days of purchase."]}
```

```python
from typed_evals import evaluate

report = evaluate("samples.jsonl", preset="rag")
print(report.summary)
report.save("evaluation-report.json")
```

You can also pass a dictionary, an `EvaluationSample`, or a list/tuple of either. A single sample returns `SampleResult`; a sequence or file returns `EvaluationReport`, even for one row. JSON files contain an array of samples.

Each sample's metrics share a judge request. Batches validate inputs first, run up to **eight concurrent workers** by default, and preserve input order.

**In notebooks and async applications**, use `aevaluate` with the same dataset:

```python
from typed_evals import aevaluate

report = await aevaluate("samples.jsonl", preset="rag")
```

The synchronous API also works in notebooks, but blocks until evaluation finishes.

**From the terminal or CI:**

```bash
typed_evals evaluate samples.jsonl \
  --metrics faithfulness answer_relevancy context_relevance \
  --output evaluation-report.json \
  --fail-on-failure

# Native OpenAI Decisions; the default model is gpt-6-luna:
typed_evals evaluate samples.jsonl --backend openai-decisions \
  --metrics faithfulness answer_relevancy context_relevance \
  --output openai-report.json --fail-on-failure
```

The CLI returns a nonzero exit code for failed or unavailable evaluations when `--fail-on-failure` is set. See the [evaluation guide](https://github.com/TrustifAI/typed_evals/blob/main/docs/EVALUATION.md) for error policies, data formats, and report fields.

## Guard tool calls

Add a policy check to a LangChain tool before registering it:

```bash
python -m pip install 'typed-evals[langchain]'
```

```python
from langchain.tools import ToolRuntime, tool
from typed_evals.adapters.langchain import guard_tool


@tool
@guard_tool(
    policy="Only read tickets owned by the authenticated customer.",
    contexts=lambda runtime: runtime.context["authorization_evidence"],
)
def read_ticket(ticket_id: str, runtime: ToolRuntime) -> str:
    """Read the complete text of a support ticket."""
    return ticket_store.read_authorized(runtime.context["customer_id"], ticket_id)
```

Here, `ticket_store` is your application's ticket service. Supply the current customer ID and application-owned authorization evidence through the agent's runtime context; the [complete LangChain example](https://github.com/TrustifAI/typed_evals/blob/main/examples/langchain_guarded_tools.py) shows the wiring.

The adapter captures the latest human request, proposed arguments, tool name, and call ID. By default, it checks `ToolSafety` at a `0.9` threshold and raises `GuardrailViolation` on a failed or unavailable judgment **before the function runs**. Successful calls preserve the tool's native return value. Keep deterministic authorization inside the tool, as shown above.

### Use your framework

| Integration | Install | Import `guard_tool` from |
| --- | --- | --- |
| Plain Python | `pip install typed-evals` | `typed_evals` |
| LangChain | `pip install 'typed-evals[langchain]'` | `typed_evals.adapters.langchain` |
| CrewAI | `pip install 'typed-evals[crewai]'` | `typed_evals.adapters.crewai` |
| Microsoft Agent Framework | `pip install 'typed-evals[agent-framework]'` | `typed_evals.adapters.agent_framework` |

The plain Python helper takes explicit `input` and `contexts`, as values or callbacks. The CrewAI adapter creates a native tool; the Microsoft adapter wraps a function under `@agent_framework.tool`. See the [adapter guide](https://github.com/TrustifAI/typed_evals/blob/main/docs/ADAPTERS.md) for each framework's registration and evidence requirements, and the [runtime guide](https://github.com/TrustifAI/typed_evals/blob/main/docs/RUNTIME.md) for agent decorators and additional checkpoints.

The currently supported CrewAI release requires OpenAI SDK `<3`, while Decisions requires `>=3.26.0`. Use separate environments for the CrewAI extra and native Decisions until those dependency bounds are compatible. The plain Python, LangChain, and Microsoft guard integrations accept a Decisions backend.

## Make the checks yours

Choose **custom metrics instead of a preset** to control the panel and thresholds:

```python
from typed_evals import AnswerRelevancy, Faithfulness, evaluate

result = evaluate(
    input="Refund period?",
    response="30 days.",
    contexts=["Refunds are allowed within 30 days."],
    metrics=[Faithfulness(threshold=0.8), AnswerRelevancy(threshold=0.8)],
)
```

For reusable configuration, use `Evaluator(preset="rag")` or `Evaluator(metrics=[...])`.

<details>
<summary><strong>Explore the nine built-in metrics</strong></summary>

| Focus | Metrics |
| --- | --- |
| Responses and retrieval | `answer_correctness`, `answer_relevancy`, `context_relevance`, `faithfulness` |
| Recorded agent execution | `task_completion`, `tool_grounding` |
| Proposed tool calls | `tool_accuracy`, `tool_safety` |
| Content policy | `policy_compliance` |

```python
from typed_evals import list_metrics

print(list_metrics())  # No API credentials needed.
```

`PolicyCompliance` and `ToolSafety` require `policy=...` when constructed. See the [metric definitions](https://github.com/TrustifAI/typed_evals/blob/main/docs/EVALUATION.md#built-in-metrics) for required evidence and score meanings.

</details>

<a id="write-a-custom-metric"></a>

<details>
<summary><strong>Write a custom metric</strong></summary>

Define an ordered rubric from least to most desirable. Higher scores pass the threshold, so the last level should describe your best outcome:

```python
from typed_evals import Metric, evaluate

completeness = Metric(
    name="completeness",
    kind="score",
    instructions="How completely does `response` cover the details requested in `input`?",
    criteria=(
        "None of the requested details are supplied.",
        "Some details are supplied, but essential requested details are missing.",
        "All essential requested details are supplied.",
    ),
    pass_definition="All essential requested details are supplied.",
    required_fields=("input", "response"),
    threshold=0.8,
)

result = evaluate(
    input="What is the refund period, and how do I request one?",
    response="You have 30 days. Contact support with your order number.",
    metrics=[completeness],
)
```

An ordered `score` rubric produces a normalized expected level between `0` and `1`; it is not automatically a probability of human acceptance. For binary and categorical rubrics, use `kind="noul"` and `kind="choice"`. The [custom metrics example](https://github.com/TrustifAI/typed_evals/blob/main/examples/custom_metrics.py) shows all three formats.

</details>

<details>
<summary><strong>Choose another model or implement a backend</strong></summary>

Pass `backend=JevBackend(model="...")` to choose a different Jev model, or `backend=OpenAIDecisionsBackend()` for native Decisions. Custom providers implement the public `Backend` and `JudgeSession` protocols and can be passed as `backend=` to evaluation and tool guards.

For a model hosted at a TypeSafe-compatible endpoint, pass its API root and key:

```python
import os

from typed_evals import JevBackend, evaluate

result = evaluate(
    input="What is the refund period?",
    response="Refunds are allowed within 30 days.",
    backend=JevBackend(
        model="my-judge-model",
        base_url="https://judge.example.com",
        api_key=os.environ["MY_JUDGE_API_KEY"],
    ),
)
```

The SDK appends `/v1/systemone` to `base_url`; supply the API root, without that endpoint path. If omitted, `base_url` and `api_key` use `TYPESAFE_BASE_URL` and `TYPESAFE_API_KEY`, respectively. These environment variables also work with the CLI. The server must accept TypeSafe's `state`, `questions`, and `model` request and return its typed answers; an OpenAI-compatible chat endpoint needs a custom backend adapter. See [backend configuration](docs/EVALUATION.md#hosted-typesafe-compatible-models) for client injection.

[Microsoft Decision-1 on OpenRouter](#use-microsoft-decision-1-on-openrouter) is a concrete example: use `model="microsoft/microsoft-decision-1"`, `base_url="https://openrouter.ai/api"`, and your OpenRouter API key with `JevBackend`.

An adapter supplies a model ID, an async `session()` context manager, and `async judge(state, questions)` returning a `JudgeResponse`. It translates the TypeSafe SDK's Noul, Choice, and Score formats into provider requests and matching answers. Image-capable adapters also expose `supported_modalities={"text", "image"}` and translate selected `ImageInput` evidence into their provider's format; existing backends without that attribute remain text-capable. The `typesafe-sdk` dependency remains required for metric construction. OpenAI loads only when used; Jev and custom backends work without the `openai` extra. Both CLI commands accept `--backend jev` (default) or `--backend openai-decisions`; omitted `--model` uses that backend's default.

See the [backend contract](https://github.com/TrustifAI/typed_evals/blob/main/docs/ARCHITECTURE.md#backend-interface) and [synthetic backend example](https://github.com/TrustifAI/typed_evals/blob/main/examples/offline_demo.py).

</details>

## Calibration

**Make thresholds reflect your own acceptance criteria.** A raw judge score of `0.8` does not necessarily mean humans would pass 80% of similar examples.

With `typed-evals[calibration]`, use `EvaluationPipeline` to fit per-metric isotonic curves against representative human pass/fail labels. Inspect the held-out diagnostics, save the fitted artifact, and load it for later evaluations. Calibration is opt-in; improvements are measured on held-out data rather than assumed.

Read the [calibration walkthrough](https://github.com/TrustifAI/typed_evals/blob/main/docs/CALIBRATION.md) for fitting, validation, and saved artifacts. The included labeled datasets are synthetic examples; replace them with human-reviewed labels for your application.

For Decisions, install `typed-evals[openai,calibration]` and fit a separate artifact with `backend=OpenAIDecisionsBackend()` or `typed_evals calibrate ... --backend openai-decisions`. Calibration uses raw metric scores, never provider confidence. New v2 artifacts record backend/compiler provenance and ordered request configuration; provider or compiler changes require a refit, while threshold changes do not. V1 artifacts remain readable for the established Jev compiler and custom backends without provenance; they cannot establish Decisions compatibility and require a fresh Decisions fit. See [artifact compatibility](docs/CALIBRATION.md#artifact-versions-and-compatibility).

`metric.score` uses the raw score unless calibration is applied. `metric.passed` compares it with the threshold and is `None` for unavailable evaluations. Judge results can be wrong; individual checks do not establish a single probability that an entire answer is true.

## Benchmark: TRIVIA+

On **645 TRIVIA+ test answers**, calibration reduced Jev's expected calibration error (ECE) by **68.1%**, from 0.0982 to 0.0313, with no additional judge calls. With cutoffs chosen on validation data, hallucination detection remained similar: F1 moved from 0.5833 to 0.5877, while accuracy stayed around 72%.

The measured gain was in how closely probabilities matched human judgments. Read the [full benchmark](docs/BENCHMARK.md) for the dataset, methodology, results, and limitations.

## Try it offline

Clone the repository to get the example scripts and datasets; they are not bundled in the installed package:

```bash
git clone https://github.com/TrustifAI/typed_evals.git
cd typed_evals
python -m pip install -e '.[calibration]'
python examples/offline_demo.py
```

This demo uses a deterministic synthetic judge: **no API key and no network calls**. It shows how a raw score, a calibrated probability, and a pass/fail decision relate.

From the same checkout, explore runtime enforcement:

```bash
python examples/runtime_guardrails.py
python examples/decorated_agent.py
```

The [CrewAI](https://github.com/TrustifAI/typed_evals/blob/main/examples/crewai_guarded_tools.py) and [Microsoft Agent Framework](https://github.com/TrustifAI/typed_evals/blob/main/examples/agent_framework_guarded_tools.py) demos also use synthetic judges; install the corresponding framework extra first.

## Examples and guides

| Start here | What you'll learn |
| --- | --- |
| [Evaluation](https://github.com/TrustifAI/typed_evals/blob/main/docs/EVALUATION.md) | Metrics, datasets, reports, custom checks, and CLI usage. |
| [Microsoft Decision-1 on OpenRouter](examples/openrouter_decision1.py) | Evaluate a response through `JevBackend` with a custom base URL and OpenRouter API key. |
| [Runtime guards](https://github.com/TrustifAI/typed_evals/blob/main/docs/RUNTIME.md) | Tool guards, agent decorators, and failure behavior. |
| [Framework adapters](https://github.com/TrustifAI/typed_evals/blob/main/docs/ADAPTERS.md) | LangChain, CrewAI, and Microsoft Agent Framework integration. |
| [Calibration](https://github.com/TrustifAI/typed_evals/blob/main/docs/CALIBRATION.md) | Human labels, held-out validation, and reusable artifacts. |
| [Architecture](https://github.com/TrustifAI/typed_evals/blob/main/docs/ARCHITECTURE.md) | Package structure and the backend extension contract. |
| [Runnable examples](https://github.com/TrustifAI/typed_evals/tree/main/examples) | RAG evaluation, agent traces, calibration, and guarded tools. |
| [Introductory notebook](https://github.com/TrustifAI/typed_evals/blob/main/notebooks/sample_notebook.ipynb) | An interactive introduction to evaluation. |
| [Advanced notebook](https://github.com/TrustifAI/typed_evals/blob/main/notebooks/advanced_usage.ipynb) | Custom metrics, calibration, and agent integrations. |

For notebooks, install the repository into the kernel's environment and run from the repository root. They use `python-dotenv` and make live requests with `TYPESAFE_API_KEY` by default. The introductory notebook also includes Microsoft Decision-1 through OpenRouter using `OPENROUTER_API_KEY`. Calibration needs the `calibration` extra; the advanced agent sections also need `agent-framework`, `langchain`, `langchain-google-genai`, and Gemini credentials.

## Contribute

Found a confusing result, need an integration, or have a useful evaluation example? [Open an issue](https://github.com/TrustifAI/typed_evals/issues) or send a pull request. Include a minimal reproduction for bugs and tests for behavior changes.

From a repository checkout:

```bash
python -m pip install -e '.[openai,calibration,dev,langchain,agent-framework]'
pytest -m "not live" --cov=typed_evals --cov-report=term-missing
ruff check .
ruff format --check .
python -m build
```

CI installs the OpenAI extra and runs both providers' contract tests against mocked HTTP, along with framework integrations using fake judges. CrewAI tests run in a separate environment with `.[calibration,dev,crewai]` because its SDK bounds currently conflict with Decisions. Normal tests need no credentials or paid calls. Tests for optional frameworks are skipped when those extras are absent. The separate Jev live smoke test requires both `TYPED_EVALS_LIVE=1` and `TYPESAFE_API_KEY` and makes a billable request. Decisions' three-question text smoke test is disabled by default; to opt into its billable request after setting `OPENAI_API_KEY`, run:

```bash
TYPED_EVALS_OPENAI_LIVE=1 pytest tests/test_live.py::test_live_openai_decisions_smoke -m live
```

It defaults to `gpt-6-luna`; `TYPED_EVALS_OPENAI_MODEL` optionally selects another model ID. Live smoke tests check the service contract rather than metric accuracy. Maintainers can follow the [publishing guide](https://github.com/TrustifAI/typed_evals/blob/main/docs/PUBLISHING.md) to cut a release.

---

<p align="center">
  Built by <a href="https://github.com/Aaryanverma">Aaryan Verma</a> ·
  <a href="https://github.com/TrustifAI/typed_evals/blob/main/LICENSE">MIT licensed</a><br>
  <strong>Useful for your next evaluation? Star the repo to keep it close.</strong>
</p>
