# Pareta SDKs — full documentation

> Pareta is one OpenAI-compatible endpoint with one model id: `"auto"`. Each request is planned, routed to benchmark-proven open specialists, verified, and falls back to a frontier model when that's the right call — one request, one bill. Its SDKs also let you benchmark `"auto"` against frontier models on your own data, read your auto traffic metrics, and find the grading contract that scores your eval data (`tasks.match`). Authenticate with a `pareta_sk_` key from the dashboard or the `PARETA_API_KEY` environment variable.

Pareta ships one SDK per language, all sharing these docs and the same `/v1` HTTP API: Python (`pip install pareta`, `from pareta import Pareta`); TypeScript/JavaScript (`npm install pareta`, `import { Pareta } from "pareta"`).

This file concatenates the entire Pareta SDK documentation (guide + examples + reference, every language) for single-read agent consumption. Source: sdk/docs/ in the repo; browsable at https://docs.pareta.ai.



---

<!-- guide/installation.md -->

# Installation & authentication

The `pareta` package is the official client for [Pareta](https://pareta.ai), available for **Python** (`pip install pareta`) and **TypeScript/JavaScript** (`npm install pareta`). It runs metered OpenAI-compatible inference against `model="auto"` (Pareta's routing brain — nothing to deploy, no model to pick), benchmarks auto against frontier models on your own data, and browses the task catalog — all from code. This page gets you installed, authenticated, and making a first call.

A few platform truths to know up front, because they shape the whole API:

- **GPUs are hidden.** You never pass a hardware knob. The serving stack behind `model="auto"` is Pareta's job, resolved per request.
- **There is one model id.** `models.list()` returns exactly one entry — `"auto"`. Which model serves a request is Pareta's decision, made per request.
- **Inference and evals are metered against your org balance.** A successful call debits credit; an empty balance raises `InsufficientCreditsError`. Top-up is browser-only — the SDK never touches billing.
- **Inference is OpenAI-compatible.** The `/v1/chat/completions` endpoint speaks the OpenAI wire format, so you can use this SDK or the stock `openai` client interchangeably.

## Install

`pareta` requires Python 3.10+ and depends only on `httpx`. Install it with whichever tool you already use:

```bash
pip install pareta
```

```bash
uv add pareta
```

```bash
poetry add pareta
```

The package ships type hints (`py.typed`), so editors and `mypy` get full autocomplete on every method and response model.

### Optional extras: CLI and MCP server

Two more interfaces ship as optional extras on the same Python package:

```bash
pip install "pareta[cli]"     # the `pareta` shell command
pip install "pareta[mcp]"     # the `pareta-mcp` Model Context Protocol server
```

The [CLI](cli.md) gives you the same surface as shell commands (`pareta chat`, `pareta evals run`, …); the [MCP server](mcp.md) exposes it to an AI agent (Claude Desktop, Cursor) as tools. Both authenticate from the same `PARETA_API_KEY`. Each installs a console script, so an isolated install with [`pipx`](https://pipx.pypa.io) (`pipx install "pareta[cli]"`) keeps it off your project's dependency tree while still putting the command on your PATH.

## Authenticate

Every request is authenticated with a `pareta_sk_` secret key sent as a Bearer token. You mint keys in the [dashboard](https://pareta.ai) — key management is browser-only, and the SDK only ever *consumes* a key. It never creates, lists, or revokes them.

### Recommended: `from_env()`

The cleanest path is to put your key in the environment and let the client read it. `from_env()` reads `PARETA_API_KEY` and the optional `PARETA_BASE_URL`:

```bash
export PARETA_API_KEY="pareta_sk_..."
```

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()                       # reads PARETA_API_KEY (+ PARETA_BASE_URL)

# List the model catalog — exactly one entry: "auto".
for model in pa.models.list():
    print(model.id, model.owned_by)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();                  // reads PARETA_API_KEY (+ PARETA_BASE_URL)

// List the model catalog — exactly one entry: "auto".
for (const model of await pa.models.list()) {
  console.log(model.id, model.ownedBy);
}
```

Keeping the key out of source is the point — `from_env()` means your code carries no secret.

### Explicit key

You can also pass the key directly. The constructor is keyword-only:

**Python**

```python
from pareta import Pareta

pa = Pareta(api_key="pareta_sk_...")
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = new Pareta({ apiKey: "pareta_sk_..." });
```

If `api_key` is falsy and `PARETA_API_KEY` is unset, the client raises `ParetaError` at construction time with a message pointing you to mint a key:

**Python**

```python
import pareta

try:
    pa = pareta.Pareta(api_key=None)         # and PARETA_API_KEY unset
except pareta.ParetaError as e:
    print(e)  # missing API key. Pass api_key=… or set PARETA_API_KEY (mint a pareta_sk_ key in the dashboard).
```

**TypeScript**

```typescript
import { Pareta, ParetaError } from "pareta";

try {
  const pa = new Pareta({ apiKey: undefined }); // and PARETA_API_KEY unset
} catch (e) {
  if (e instanceof ParetaError) {
    console.log(e.message); // missing API key. Pass apiKey: … or use Pareta.fromEnv() with PARETA_API_KEY (mint a pareta_sk_ key in the dashboard).
  }
}
```

## Constructor options

**Python**

```python
Pareta(
    api_key: str | None = None,              # pareta_sk_ key; falls back to nothing (from_env reads the env)
    base_url: str | None = None,             # defaults to "https://api.pareta.ai"
    timeout=None,                            # defaults to httpx.Timeout(60.0, connect=10.0)
    max_retries: int = 2,                    # retries on 408/409/429/500/502/503/504
    http_client: httpx.Client | None = None, # bring your own httpx.Client
)
```

**TypeScript**

```typescript
new Pareta({
  apiKey?: string,        // pareta_sk_ key; falls back to nothing (fromEnv reads the env)
  baseURL?: string,       // defaults to "https://api.pareta.ai"
  timeout?: number,       // milliseconds; defaults to 60_000
  maxRetries?: number,    // default 2; retries on 408/409/429/500/502/503/504
  fetch?: typeof fetch,   // inject your own fetch implementation
});
```

- **`base_url`** defaults to the production API, `https://api.pareta.ai`, and is normalized (trailing slash stripped). Override it only to point at a non-prod environment; set `PARETA_BASE_URL` to do the same via `from_env()`.
- **`max_retries`** (default 2) retries idempotent failures and rate limits with exponential backoff that honors a `Retry-After` header. See [Errors & retries](errors-and-retries.md).
- **`http_client`** lets you supply a pre-configured `httpx.Client` (custom proxies, connection limits, transport). When you pass one, the SDK does not own it and `close()` will not shut it down.

## Manage the connection

The client holds a pooled HTTP connection. Use it as a context manager so the pool is released cleanly:

**Python**

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    resp = pa.chat.completions.create(
        model="auto",                        # the one model id — Pareta routes the request
        messages=[{"role": "user", "content": "Extract the total from this invoice: ..."}],
    )
    print(resp.choices[0].message.content)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const resp = await pa.chat.completions.create({
  model: "auto",                          // the one model id — Pareta routes the request
  messages: [{ role: "user", content: "Extract the total from this invoice: ..." }],
});
console.log(resp.choices[0].message.content);
```

Outside a `with` block, call `pa.close()` when you are done. (`close()` is a no-op when you supplied your own `http_client`.) The TypeScript client holds no owned connection — it uses `fetch` per request, so there is nothing to close.

## Async client

`AsyncPareta` mirrors `Pareta` exactly — same constructor, same `from_env()`, same resource namespaces — with awaitable methods and `aclose()` / `async with`:

**Python**

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        resp = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Summarize this contract clause: ..."}],
        )
        print(resp.choices[0].message.content)

asyncio.run(main())
```

**TypeScript**

There is no `AsyncPareta` in TypeScript — there is one `Pareta` client and it is already async. Every I/O method returns a `Promise` you `await` (and streaming methods return an `AsyncIterable` you drive with `for await`). The same client works in sync-looking and concurrent code; there is no separate sync/async split to choose between.

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Summarize this contract clause: ..." }],
});
console.log(resp.choices[0].message.content);
```

## Your first metered call

Inference debits your org balance on success. If the balance is empty, the call raises `InsufficientCreditsError` (402) — top up in the dashboard, which is the only place billing lives:

**Python**

```python
from pareta import Pareta, InsufficientCreditsError

pa = Pareta.from_env()

try:
    resp = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "What is the invoice number?"}],
        temperature=0,                       # extra OpenAI params pass straight through
    )
    print(resp.choices[0].message.content)
    print(resp.usage.total_tokens, "tokens")
except InsufficientCreditsError:
    print("Org out of credit — top up in the dashboard.")
```

**TypeScript**

```typescript
import { Pareta, InsufficientCreditsError } from "pareta";

const pa = Pareta.fromEnv();

try {
  const resp = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "What is the invoice number?" }],
    temperature: 0,                        // extra OpenAI params pass straight through
  });
  console.log(resp.choices[0].message.content);
  console.log(resp.usage.totalTokens, "tokens");
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    console.log("Org out of credit — top up in the dashboard.");
  } else {
    throw e;
  }
}
```

The `model` is `"auto"` — the only model id there is. One request, one debit, however many internal model calls Pareta's plan makes. See [Inference](./inference.md) for streaming and the full chat-completions surface.

## Zero-install alternative for inference

You do not need this SDK to run inference at all. Because it's OpenAI-compatible, you can point the stock `openai` client at Pareta's `base_url` with the same `pareta_sk_` key and call `model="auto"` — there is nothing to set up first:

**Python**

```python
from openai import OpenAI

client = OpenAI(api_key="pareta_sk_...", base_url="https://api.pareta.ai/v1")

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What is the invoice number?"}],
)
print(resp.choices[0].message.content)
```

**TypeScript**

```typescript
import OpenAI from "openai";

const client = new OpenAI({ apiKey: "pareta_sk_...", baseURL: "https://api.pareta.ai/v1" });

const resp = await client.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "What is the invoice number?" }],
});
console.log(resp.choices[0].message.content);
```

This is handy for inference-only workloads or dropping Pareta into an existing OpenAI codebase. The `pareta` SDK's distinct value is everything around that call that the OpenAI client cannot reach: matching intent to tasks, benchmarking `"auto"` against frontier models on your own data, and reading auto's metrics.

## Next steps

- [Inference](./inference.md) — chat completions, streaming, and metering.
- [Core concepts](./core-concepts.md) — tasks, `model="auto"`, and how requests are planned, routed, and billed.
- [Evals](evaluation.md) — build eval sets and benchmark `"auto"` against frontier baselines.
- [Errors & retries](errors-and-retries.md) — the typed exception hierarchy and retry policy.
- [The `pareta` CLI](cli.md) — the same surface from your shell (`pip install "pareta[cli]"`).
- [MCP server](mcp.md) — drive Pareta from an AI agent (Claude Code, Codex, Claude Desktop, Cursor) over MCP (`pip install "pareta[mcp]"`).
- [The `/pareta` skill](skill.md) — a slash-command `SKILL.md` for Claude Code and Codex that drives the CLI.



---

<!-- guide/quickstart.md -->

# Quickstart

Pareta is one endpoint. Send any request with `model="auto"` and Pareta plans
it, routes each part to the cheapest model that holds frontier-grade quality,
verifies, and answers — billed as one request, with a frontier model as the
built-in quality floor. Inference is OpenAI-compatible and metered against
your org's balance.

## The 30-second version

```python
from pareta import Pareta

client = Pareta.from_env()          # reads PARETA_API_KEY
completion = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize this contract: …"}],
)
print(completion.choices[0].message.content)
```

That is the product. Everything below — benchmarking auto against frontier
models on your own data and monitoring spend + projected savings — exists to
prove and operate that one call.

- **Prove it**: `client.evals` with `"auto"` among the candidates (see
  [Evaluation](evaluation.md)) — per-contender quality + cost on YOUR data.
- **Watch it**: `client.auto.metrics()` — requests, success rate, spend, and
  the projected savings vs frontier.
- **Compare it**: `client.auto.compare_frontier(...)` — one prompt against a
  frontier vendor, metered, for a side-by-side.

## Install

```bash
pip install pareta        # or: uv add pareta / poetry add pareta
```

## Authenticate

Mint a `pareta_sk_` key in the dashboard (key management is browser-only) and
export it. `Pareta.from_env()` reads `PARETA_API_KEY` (and an optional
`PARETA_BASE_URL`).

```bash
export PARETA_API_KEY="pareta_sk_..."
```

The SDK only ever consumes a key. It never creates, lists, or revokes them, and
it never exposes your balance or payment methods. Topping up credit is
browser-only.

## Find the grading contract for your data

There is no model to pick and nothing to deploy — send any generation job
straight to `model="auto"`. The one lookup you'll ever do is for
benchmarking: `tasks.match` maps a plain-English description of your dataset
to the grading contract an eval scores it with:

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()                                  # reads PARETA_API_KEY

m = pa.tasks.match("extract key fields from contracts")
print(m.type)                    # "task" — a benchmarked task covers this
if m.chosen:
    print(m.chosen.task_id)      # e.g. "contract-key-fields"
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();                              // reads PARETA_API_KEY

const m = await pa.tasks.match("extract key fields from contracts");
console.log(m.matched);          // true — a benchmarked task covers this
console.log(m.chosen?.taskId);   // e.g. "contract-key-fields"
```

`m.type` is one of four verdicts: `"task"` (a benchmarked task fits),
`"capability"` (a general lane — chat, coding, vision, … — covers it),
`"unsupported"` (Pareta does not cover this; a correct answer, not an error),
or `"none"` (the router was unavailable and the lexical fallback found nothing
confident). Whatever the verdict names, running the job is always the same
call: `chat.completions.create(model="auto", ...)`.

Browse the whole catalog behind the router with `pa.tasks.list()` and
`pa.tasks.retrieve(task_id)` — see [tasks](../reference/tasks.md).

## Stream the response

Pass `stream=True` to get an iterator of `ChatCompletionChunk`. The incremental
text lives on `chunk.choices[0].delta.content` (it can be `None` on the first
and last chunks, so guard it).

**Python**

```python
for chunk in pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Write a haiku about invoices."}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="", flush=True)
print()
```

**TypeScript**

```typescript
for await (const chunk of pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Write a haiku about invoices." }],
  stream: true,
})) {
  process.stdout.write(chunk.choices[0].delta.content || "");
}
console.log();
```

Extra OpenAI parameters (`temperature`, `max_tokens`, `top_p`, and so on) pass
straight through as keyword arguments.

## Cost and credit

Every successful completion debits your org's balance — one debit per request,
no matter how many internal model calls auto's plan makes. If the balance is
empty, the call raises `InsufficientCreditsError` (HTTP 402). Top-up is
browser-only.

**Python**

```python
from pareta import InsufficientCreditsError

try:
    resp = pa.chat.completions.create(model="auto", messages=[
        {"role": "user", "content": "ping"},
    ])
except InsufficientCreditsError:
    print("Out of credit — top up in the dashboard.")
```

**TypeScript**

```typescript
import { InsufficientCreditsError } from "pareta";

try {
  const resp = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "ping" }],
  });
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    console.log("Out of credit — top up in the dashboard.");
  } else {
    throw e;
  }
}
```

Evaluation runs are metered the same way (auto plus frontier compute). An
`EvalRun` reports its billed total on `run.cost`, a `Decimal` in dollars floored
to whole cents (so a sub-cent run reads `Decimal("0.00")`); the raw value is on
`run.cost_micro_usd`. See [Evals](evaluation.md).

## Clean up

There is nothing running on your account to stop — auto's serving fleet is
Pareta's to operate. Cleanup is just closing the client (or using it as a
context manager).

**Python**

```python
pa.close()
```

**TypeScript**

```typescript
// No close() in TS: the client owns no connection (it uses the native fetch),
// so there is nothing to release and no context-manager form to wrap it in.
```

**Python**

```python
# Context-manager form closes the HTTP client for you.
with Pareta.from_env() as pa:
    resp = pa.chat.completions.create(model="auto", messages=[
        {"role": "user", "content": "hi"},
    ])
```

**TypeScript**

```typescript
// No context manager in TS — just construct and use it; nothing to close.
const pa = Pareta.fromEnv();
const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "hi" }],
});
```

## List what you can call

`models.list()` returns the OpenAI-compatible model list. It has exactly one
entry — `"auto"` — which is the point: the id you pass to
`chat.completions.create(model=...)` is never a decision.

**Python**

```python
for m in pa.models.list():
    print(m.id, m.owned_by)
```

**TypeScript**

```typescript
for (const m of await pa.models.list()) {
  console.log(m.id, m.ownedBy);
}
```

## Async

`AsyncPareta` mirrors the sync client; resource methods are `async def` and
streams are async iterators.

**Python**

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        resp = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Say hello."}],
        )
        print(resp.choices[0].message.content)

asyncio.run(main())
```

**TypeScript**

```typescript
// There is no AsyncPareta in TypeScript — the single `Pareta` client is already
// async. Every I/O method returns a Promise (await it), and streams are async
// iterables (`for await`).
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Say hello." }],
});
console.log(resp.choices[0].message.content);
```

## Already using the OpenAI SDK?

You do not need this SDK just to run inference. Point the `openai` client at
your `base_url` plus your `pareta_sk_` key:

**Python**

```python
from openai import OpenAI

client = OpenAI(api_key="pareta_sk_...", base_url="https://api.pareta.ai/v1")
resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "hi"}],
)
```

**TypeScript**

```typescript
import OpenAI from "openai";

const client = new OpenAI({ apiKey: "pareta_sk_...", baseURL: "https://api.pareta.ai/v1" });
const resp = await client.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "hi" }],
});
```

This SDK's unique value is everything around that call: benchmark `"auto"` on
your own data (`evals`), match intent to coverage (`tasks.match`), and watch
traffic + savings (`auto.metrics()`) — from code.

## Next steps

- [Core concepts](core-concepts.md) — tasks and capabilities, the routing
  brain, metering, and the match → eval → production funnel.
- [Evals](evaluation.md) — benchmark `"auto"` against frontier baselines on
  your own data.
- [Errors](errors-and-retries.md) — the `ParetaError` hierarchy and retry behavior.



---

<!-- guide/core-concepts.md -->

# Core concepts

Pareta is one OpenAI-compatible endpoint with one model id: **`"auto"`**. This
page covers the handful of ideas the rest of the SDK assumes you understand:
the **routing brain** behind `model="auto"`, **one interface per data
shape**, **tasks** (the grading contracts evals score against), **open vs
frontier** models, why **models** and **hardware**
are hidden, how **metering** works, and the **funnel** that ties them together
(prove `"auto"` on your data, ship it, watch the metrics).

Every code block below is runnable as written. They all start from a client:

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (+ optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();   // reads PARETA_API_KEY (+ optional PARETA_BASE_URL)
```

`from_env()` is the path you want in almost every case. The explicit form is
`Pareta(api_key="pareta_sk_...", base_url="https://api.pareta.ai")`; arguments
are keyword-only. See [Authentication](installation.md) for key minting
(browser-only) and [The client](../reference/client.md) for timeouts, retries, and the
async `AsyncPareta` mirror.

## The routing brain: `model="auto"`

Every request you send with `model="auto"` is **planned**, its parts **routed**
to benchmark-proven open specialists, the output **verified**, with a
**fallback** to a frontier model when that is the right call. One request, one
bill, a frontier model as the built-in quality floor.

There is nothing to deploy and no model to pick — "which model?" is the
question Pareta answers for you, per request. `models.list()` reflects that:
it returns exactly one entry.

**Python**

```python
for m in pa.models.list():
    print(m.id)          # exactly one entry: "auto"
```

**TypeScript**

```typescript
for (const m of await pa.models.list()) {
  console.log(m.id);     // exactly one entry: "auto"
}
```

Calling the brain is plain chat — see
[Inference is OpenAI-compatible](#inference-is-openai-compatible) below. The
surfaces *around* that call live on `pa.auto`:

- `auto.metrics()` — your org's `"auto"` traffic, rolled up: requests +
  success rate (30d), spend, hourly p50/p95/error buckets (7d), daily success
  cells (30d), and the projected savings vs frontier.
- `auto.compare_frontier(model=..., messages=...)` (TypeScript
  `auto.compareFrontier({ model, messages })`) — one prompt against a frontier
  vendor for a side-by-side with `"auto"`. Metered at the vendor's actual
  token cost; a failed vendor call bills $0. Allowed models: `gpt-5.5`,
  `gemini-3-5-flash`, `gemini-3-1-pro`, `claude-sonnet-4-6`.

## One interface per data shape

You never choose a model anywhere on Pareta — you choose the **data shape**,
and each shape has exactly one interface:

| Your data | Interface | Behind it |
|---|---|---|
| messages in, text out | `chat.completions` with `model="auto"` | the routing brain |
| a query + documents to rank | [`rerank`](../reference/rerank.md) | a purpose-trained reranker |
| text to turn into vectors | [`embeddings`](../reference/embeddings.md) | an open embedder that beats the frontier's |
| audio in / audio out | [`audio`](../reference/audio.md) | the speech lanes |

Within every shape, routing, model choice, and escalation are Pareta's job.
The separate routes exist because vectors, ranked lists, and audio bytes
don't fit the chat message contract — not because there is anything to
navigate.

## Tasks: grading contracts for evals

Internally, `"auto"`'s quality guarantees come from a catalog of benchmarked
jobs — Pareta has measured open and frontier models against each on real
data. You don't navigate that catalog to use Pareta. You meet it in exactly
one place: **benchmarking on your own data**, where a **task** is the
grading contract — the row shape your dataset must follow and the scorer
that grades outputs against your labels.

Every task has a stable `id` (e.g. `"contract-key-fields"`), a
`default_scorer` (the function that grades a model's output — field-F1,
nDCG@10, WER, judge panel), and a `has_blob_input` flag (true when the rows
carry documents or images, not just text).

**Python**

```python
for task in pa.tasks.list():
    print(task.id, task.default_scorer, "blob" if task.has_blob_input else "text")

# Fetch one contract, optionally with sample rows to see its input shape
t = pa.tasks.retrieve("contract-key-fields", examples_n=3)
print(t.id, t.default_scorer, t.has_blob_input)
```

**TypeScript**

```typescript
for (const task of await pa.tasks.list()) {
  console.log(task.id, task.defaultScorer, task.hasBlobInput ? "blob" : "text");
}

// Fetch one contract, optionally with sample rows to see its input shape
const t = await pa.tasks.retrieve("contract-key-fields", { examplesN: 3 });
console.log(t.id, t.defaultScorer, t.hasBlobInput);
```

Rather than reading the scorer list, describe your dataset in plain English
and `tasks.match` names the contract that grades it:

**Python**

```python
m = pa.tasks.match("vendor invoices with labeled line items and totals")
if m.matched:
    print("grade with:", m.chosen.task_id)   # -> evals.runs.create(task=...)
```

**TypeScript**

```typescript
const m = await pa.tasks.match("vendor invoices with labeled line items and totals");
if (m.matched && m.chosen) {
  console.log("grade with:", m.chosen.taskId);
}
```

`match()` raises `ValueError` on an empty query. A no-match answer is a
statement about *scoring* — no benchmarked contract fits that description —
not about serving: generation work always goes to `model="auto"`. See
[Tasks](../reference/tasks.md) for the full matcher surface.


## Open vs frontier models

Two kinds of model stand behind every task:

- **Open** models are the open-weights specialists `"auto"` routes to. Pareta
  benchmarks them, serves them, and picks between them — you never call one
  directly or learn its identity.
- **Frontier** models are hosted vendor models (OpenAI, Google, Anthropic, and
  so on). They play two roles: the built-in quality floor `"auto"` falls back
  to when no specialist holds the bar, and the **baseline** you measure
  `"auto"` against in evals. The whole point of Pareta is showing that
  `"auto"` matches or beats the frontier on *your* task at a fraction of the
  cost.

Frontier (vendor) ids appear in the clear — those are public products — in
exactly two places: eval baselines and `auto.compare_frontier()`. To enumerate
the frontier roster you can evaluate against, annotated for a given task, use
`evals.frontier_models`:

**Python**

```python
for fm in pa.evals.frontier_models(task="contract-key-fields"):
    print(fm.id, fm.vendor, "vision" if fm.vision else "text",
          "(benchmarked)" if fm.benchmarked else "")
```

**TypeScript**

```typescript
for (const fm of await pa.evals.frontierModels("contract-key-fields")) {
  console.log(fm.id, fm.vendor, fm.vision ? "vision" : "text",
    fm.benchmarked ? "(benchmarked)" : "");
}
```

Passing `task=` annotates each model's `benchmarked` flag (measured on that
task) and filters the roster by capability (for example, only vision-capable
models are returned for document tasks). Feed the `id` values into an eval
run's `frontier=` list.

## Models are hidden

You never pick a model, and open-weights identities never cross the API. The
only model id you send is `"auto"`; the only model ids you read back are
`"auto"` and frontier vendor ids in eval and comparison results
(`result.model_id`).

This is a feature, not an omission. There are no open-model ids to look up,
hard-code, or keep current — when Pareta promotes a better specialist behind a
task, your requests get it on the next call, with no code change and no
migration.

## Hardware is hidden

You never choose a GPU, tensor-parallel degree, quantization scheme, or
serving mode. The specialists behind `"auto"` run on serving classes Pareta
resolves from its registry, and capacity — warm pools, autoscaling, cold
starts — is Pareta's problem. The one place serving infrastructure surfaces in
the SDK is `EndpointNotReadyError` (503): a serving backend behind auto is
warming or briefly unavailable. The SDK retries 503s automatically, so you
rarely see it.

## Inference is OpenAI-compatible

Call the brain through `chat.completions.create` with `model="auto"`. The
request and response match the OpenAI chat schema, so the official `openai`
client works against the same base URL and key.

**Python**

```python
resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Extract the contract effective date."}],
    temperature=0,                          # extra OpenAI params pass straight through
)
print(resp.choices[0].message.content)
print(resp.usage.total_tokens)
```

**TypeScript**

```typescript
const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Extract the contract effective date." }],
  temperature: 0,                           // extra OpenAI params pass straight through
});
console.log(resp.choices[0].message.content);
console.log(resp.usage.totalTokens);
```

Streaming yields `ChatCompletionChunk` objects; the incremental text is on
`chunk.choices[0].delta.content`:

**Python**

```python
for chunk in pa.chat.completions.create(model="auto", messages=[...], stream=True):
    print(chunk.choices[0].delta.content or "", end="")
```

**TypeScript**

```typescript
for await (const chunk of pa.chat.completions.create({ model: "auto", messages: [...], stream: true })) {
  process.stdout.write(chunk.choices[0].delta.content || "");
}
```

`create()` raises `ValueError` up front if `model` or `messages` is empty. See
[Running inference](./inference.md) for streaming details and the async
iterator form.

## Metering and billing

Both inference and evals are **metered against your organization's balance**.

- **Inference:** a successful `chat.completions.create()` debits the org
  balance **once per request** — no matter how many internal model calls
  auto's plan makes (planning, specialists, verification, fallback).
  Orchestration overhead is Pareta's cost, not yours.
- **Speech:** the `pa.audio` namespace (`pa.audio.transcriptions(...)`,
  `pa.audio.speech(...)`, Python-only) is billed **per minute** of audio — see
  [Audio](../reference/audio.md).
- **Evals:** `evals.runs.create()` debits for the compute it spends: `"auto"`
  and any frontier baselines you include. A FAILED run is not charged.
- **Frontier comparisons:** `auto.compare_frontier()` is metered at the
  vendor's actual token cost; a failed vendor call bills $0.
- **Empty balance:** every path raises `InsufficientCreditsError` (HTTP 402).

**Python**

```python
from pareta import InsufficientCreditsError

try:
    resp = pa.chat.completions.create(model="auto", messages=[{"role": "user", "content": "hi"}])
except InsufficientCreditsError:
    print("Top up the org balance in the dashboard, then retry.")
```

**TypeScript**

```typescript
import { InsufficientCreditsError } from "pareta";

try {
  const resp = await pa.chat.completions.create({ model: "auto", messages: [{ role: "user", content: "hi" }] });
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    console.log("Top up the org balance in the dashboard, then retry.");
  } else {
    throw e;
  }
}
```

Topping up is **browser-only**. The SDK never exposes the balance, payment
methods, or top-up. It only consumes credit and surfaces the 402 when there is
none.

### Reading cost off an eval run

An eval run reports what it cost. The SDK follows one money convention
(`SDK_PLAN` §6): the **billed total is floored to whole cents** so the SDK never
overstates a charge, while sub-cent precision stays available in micro-USD.

- `run.cost` is a `Decimal` in dollars, floored to cents. A 5 µUSD run reads
  `Decimal("0.00")`.
- `run.cost_micro_usd` is the raw integer (`1_000_000` = `$1.00`).
- Per-item unit rates such as `result.mean_cost_micro_usd` stay in
  **micro-USD**. Flooring them to cents would erase the auto-vs-frontier
  comparison that is the whole point.

**Python**

```python
print(run.cost)               # Decimal("0.42"): billed dollars, floored to cents
print(run.cost_micro_usd)     # 420715: raw micro-USD
```

**TypeScript**

```typescript
console.log(run.cost);          // "0.42": billed dollars (string), floored to cents
console.log(run.costMicroUsd);  // 420715: raw micro-USD
```

## The proof funnel

The pieces above compose into one path from "I have a job" to "auto is running
it in production, cheaper." This is the recommended flow:

```
match  ->  eval on YOUR data  ->  model="auto" in production  ->  watch the metrics
```

1. **Match** your dataset to its grading contract (`tasks.match`).
2. **Eval** `"auto"` against frontier baselines on *your own* data. Public
   benchmarks are a starting point; your rows are the deciding vote.
3. **Ship** `model="auto"` — the same call, now carrying production traffic.
4. **Watch** `auto.metrics()` — requests, success rate, spend, projected
   savings vs frontier.

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()

# 1. Match your dataset to its grading contract
match = pa.tasks.match("extract key fields from contracts")
task = match.chosen.task_id

# 2. Evaluate "auto" against frontier baselines on YOUR rows.
#    Pass items + intent to create the eval set inline (task is optional —
#    here we pin the contract we just matched), or use an existing set id.
run = pa.evals.runs.create(
    intent="extract the effective date from each contract",
    task=task,
    items=[
        {"input": "...your contract text...", "expected": {"effective_date": "2026-01-01"}},
        # ...more rows...
    ],
    models=["auto"],
    frontier="benchmarked",       # the frontier baselines measured on this task
    wait=True,                    # block until the run is terminal
)

# 3. Read results (quality + cost), then ship the same call to production
for r in sorted(run.results, key=lambda r: (r.quality_mean or 0), reverse=True):
    print(r.model_id, r.kind, r.quality_mean, r.mean_cost_micro_usd, f"n={r.n_succeeded}")

print("eval cost:", run.cost)     # Decimal dollars, floored to cents

resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "...your contract text..."}],
)

# 4. Watch it in production
m = pa.auto.metrics()
print(m["requests_30d"], m["success_rate_30d"], m["savings_vs_frontier_micro_usd_30d"])
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

// 1. Match free-text intent to a task
const match = await pa.tasks.match("extract key fields from contracts");
const task = match.chosen!.taskId;

// 2. Evaluate "auto" against frontier baselines on YOUR rows.
//    Pass items + intent to create the eval set inline (task is optional —
//    here we pin the contract we just matched), or use an existing set id.
const run = await pa.evals.runs.create({
  intent: "extract the effective date from each contract",
  task,
  items: [
    { input: "...your contract text...", expected: { effective_date: "2026-01-01" } },
    // ...more rows...
  ],
  models: ["auto"],
  frontier: "benchmarked",      // the frontier baselines measured on this task
  wait: true,                   // block until the run is terminal
});

// 3. Read results (quality + cost), then ship the same call to production
for (const r of [...run.results].sort((a, b) => (b.qualityMean ?? 0) - (a.qualityMean ?? 0))) {
  console.log(r.modelId, r.kind, r.qualityMean, r.meanCostMicroUsd, `n=${r.nSucceeded}`);
}

console.log("eval cost:", run.cost);   // dollar string, floored to cents

const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "...your contract text..." }],
});

// 4. Watch it in production
const m = await pa.auto.metrics();
console.log(m.requests_30d, m.success_rate_30d, m.savings_vs_frontier_micro_usd_30d);
```

A few notes on the eval call:

- Provide **either** `eval_set=<id>` (an existing set) **or** `items=... +
  intent=...` to create one inline (`task=` is optional — pass it to pin a
  specific contract). With neither, `create()` raises `ValueError`.
- `frontier=` accepts `None`/`"none"` (no baselines), an explicit list of
  frontier ids, `"all"` (every frontier model for the task), or `"benchmarked"`
  (only the frontier models measured on this task, vision-filtered for
  document tasks). Keyword resolution needs to know the task; with
  `eval_set=`, the SDK looks the task up for you.
- `wait=True` polls until the run reaches `"completed"` or `"failed"`
  (`run.is_terminal`), then returns the final `EvalRun`. For document tasks,
  attach binaries with `evals.sets.upload_document(...)` before running.

For the full eval API (building sets, attaching documents, inline vs. existing
sets, and polling semantics) see [Evaluating models](evaluation.md). For the
catalog and matcher surface in depth, see [Tasks](../reference/tasks.md).

## Errors at a glance

Every SDK error subclasses `ParetaError`. The status-mapped subclasses let you
branch on what went wrong without inspecting status codes:

| Exception | Status | When |
|---|---|---|
| `AuthenticationError` | 401 | bad or missing key |
| `InsufficientCreditsError` | 402 | org out of credit (top up in the dashboard) |
| `PermissionDeniedError` | 403 | the user lacks permission |
| `NotFoundError` | 404 | unknown task or run |
| `ConflictError` | 409 | transient contention (auto-retried) |
| `RateLimitError` | 429 | throttled (auto-retried) |
| `EndpointNotReadyError` | 503 | a serving backend behind auto is warming or briefly unavailable (auto-retried) |
| `BadRequestError` | 400/422 | malformed request |
| `APIConnectionError` / `APITimeoutError` | n/a | transport failure (auto-retried) |

**Python**

```python
import pareta

try:
    resp = pa.chat.completions.create(model="auto", messages=[{"role": "user", "content": "hi"}])
except pareta.EndpointNotReadyError:
    print("A backend is warming; retries are exhausted — try again shortly.")
except pareta.InsufficientCreditsError:
    print("Out of credit. Top up in the dashboard.")
except pareta.ParetaError as e:
    print("request failed:", e)
```

**TypeScript**

```typescript
import { EndpointNotReadyError, InsufficientCreditsError, ParetaError } from "pareta";

try {
  const resp = await pa.chat.completions.create({ model: "auto", messages: [{ role: "user", content: "hi" }] });
} catch (e) {
  if (e instanceof EndpointNotReadyError) {
    console.log("A backend is warming; retries are exhausted — try again shortly.");
  } else if (e instanceof InsufficientCreditsError) {
    console.log("Out of credit. Top up in the dashboard.");
  } else if (e instanceof ParetaError) {
    console.log("request failed:", e);
  } else {
    throw e;
  }
}
```

See [Error handling](errors-and-retries.md) for the full hierarchy, the `request_id`
attribute for support, and the retry policy.



---

<!-- guide/inference.md -->

# Running inference

You run inference through `chat.completions.create`, which has the same shape as the OpenAI chat completions API. Pass `model="auto"`, a list of messages, and you get a `ChatCompletion` back. Set `stream=True` and you get an iterator of token deltas instead.

Pareta is OpenAI-compatible on the wire, so you can run inference with this SDK, with the `openai` package, or with raw HTTP, whichever fits your stack. This SDK's extra value is the control plane (evals, task match, auto metrics); for plain inference the two are interchangeable.

A few platform truths that shape this page:

- **There is no model to pick.** `model` is the literal string `"auto"`; Pareta routes each request behind it. Real open-weights model ids never reach you; the backend resolves them. You never pick a GPU.
- **Inference is metered against your org balance.** A successful completion debits your balance — one debit per request, no matter how many internal model calls auto's plan makes. If the balance is empty, the call raises `InsufficientCreditsError` (402). Top-up is browser-only; the SDK has no balance or payment surface.

## `model="auto"` — the routing brain

The model id for every request is the literal string `"auto"`.
Pareta decomposes the request, routes each part to the cheapest model that
holds frontier-grade quality, verifies checkable outputs (escalating to a
frontier model on a failed check), and synthesizes one answer. One request,
one debit; a request that errors out bills $0. Streaming works the same way —
the answer streams token by token, and the SSE stream carries
`: pareta-progress <stage>` comments (`planning` / `executing` / `answering`)
you can surface as status.

```python
completion = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "…"}],
)
```

Everything below — setup, streaming, async, errors — is that one call in
different shapes.

## Setup

Mint a `pareta_sk_` key in the dashboard, export it, and build the client from the environment:

```bash
export PARETA_API_KEY=pareta_sk_...
```

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (+ optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();   // reads PARETA_API_KEY (+ optional PARETA_BASE_URL)
```

`from_env()` is the recommended path. You can also pass the key explicitly: `Pareta(api_key="pareta_sk_...")`. The client is a context manager, so `with Pareta.from_env() as pa:` cleans up the HTTP connection for you.

## A basic completion

Pass `model="auto"` and a non-empty `messages` list in OpenAI format. You get back a `ChatCompletion`.

**Python**

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    resp = pa.chat.completions.create(
        model="auto",   # the routing brain — the only model id
        messages=[
            {"role": "system", "content": "You extract structured fields from documents."},
            {"role": "user", "content": "What is the invoice total?\n\nINVOICE\nTotal due: $4,210.00"},
        ],
    )

    print(resp.choices[0].message.content)
    print(resp.usage.total_tokens, "tokens")
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const resp = await pa.chat.completions.create({
  model: "auto",   // the routing brain — the only model id
  messages: [
    { role: "system", content: "You extract structured fields from documents." },
    { role: "user", content: "What is the invoice total?\n\nINVOICE\nTotal due: $4,210.00" },
  ],
});

console.log(resp.choices[0].message.content);
console.log(resp.usage.totalTokens, "tokens");
```

`model` and `messages` are both required. The SDK raises `ValueError` before sending if `model` is falsy or `messages` is empty, so a malformed call fails fast without burning a request.

## The ChatCompletion shape

`create()` returns a `ChatCompletion`. The fields mirror OpenAI:

**Python**

```python
resp.id                              # str | None
resp.model                           # str | None: echoes "auto"
resp.created                         # int | None: Unix timestamp
resp.choices                         # list[Choice]
resp.choices[0].index                # int | None
resp.choices[0].finish_reason        # "stop", "length", ...
resp.choices[0].message.role         # "assistant"
resp.choices[0].message.content      # str | None: the generated text
resp.usage.prompt_tokens             # int | None
resp.usage.completion_tokens         # int | None
resp.usage.total_tokens              # int | None
```

**TypeScript**

```typescript
resp.id                              // string | null
resp.model                           // string | null: echoes "auto"
resp.created                         // number | null: Unix timestamp
resp.choices                         // Choice[]
resp.choices[0].index                // number | null
resp.choices[0].finishReason         // "stop", "length", ...
resp.choices[0].message.role         // "assistant"
resp.choices[0].message.content      // string | null: the generated text
resp.usage.promptTokens              // number | null
resp.usage.completionTokens          // number | null
resp.usage.totalTokens               // number | null
```

Every response object keeps the raw server JSON. If a field isn't surfaced as a typed property, reach it with `resp.to_dict()` or `resp["..."]`. Nothing the API returns is lost behind the typed layer.

## Passthrough parameters

Any extra keyword you pass goes straight into the request body, so the full OpenAI parameter set is available without the SDK enumerating it:

**Python**

```python
resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize this contract clause: ..."}],
    temperature=0.2,
    max_tokens=512,
    top_p=0.9,
)
```

**TypeScript**

```typescript
const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Summarize this contract clause: ..." }],
  temperature: 0.2,
  max_tokens: 512,
  top_p: 0.9,
});
```

`temperature`, `max_tokens`, `top_p`, `stop`, `seed`, and friends all pass through unchanged.

## Streaming

Set `stream=True` and `create()` returns an iterator of `ChatCompletionChunk` objects instead of a single `ChatCompletion`. Each chunk carries a `delta` (not a `message`); the incremental text is at `chunk.choices[0].delta.content`.

**Python**

```python
with Pareta.from_env() as pa:
    stream = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Draft a one-paragraph status update."}],
        stream=True,
    )
    for chunk in stream:
        print(chunk.choices[0].delta.content or "", end="", flush=True)
    print()
```

**TypeScript**

```typescript
const pa = Pareta.fromEnv();
const stream = pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Draft a one-paragraph status update." }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0].delta.content || "");
}
console.log();
```

`ChatCompletionChunk` has the same schema as `ChatCompletion`; it exists as a distinct type only for hinting. Guard `delta.content` with `or ""`: the first and last chunks of a stream often carry role or finish metadata with no text.

The stream is data-only SSE and always terminates on a `[DONE]` sentinel, which the SDK consumes for you, so the iterator simply ends. Note that retries only cover the initial handshake. Once tokens are flowing, a mid-stream drop raises immediately rather than silently resuming.

## Async

`AsyncPareta` mirrors the sync client. Methods are `async def`; for streaming you `await` the call once, then `async for` over the chunks.

**Python**

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        # Non-streaming
        resp = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "What is the invoice total?"}],
        )
        print(resp.choices[0].message.content)

        # Streaming
        stream = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Stream me a haiku about ledgers."}],
            stream=True,
        )
        async for chunk in stream:
            print(chunk.choices[0].delta.content or "", end="", flush=True)
        print()

asyncio.run(main())
```

**TypeScript**

```typescript
// There is no AsyncPareta in TypeScript — the one Pareta client is already
// Promise-only. Every I/O method returns a Promise you `await`; streaming
// returns an AsyncIterable you drive with `for await`.
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

// Non-streaming
const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "What is the invoice total?" }],
});
console.log(resp.choices[0].message.content);

// Streaming
const stream = pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Stream me a haiku about ledgers." }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0].delta.content || "");
}
console.log();
```

## Reading the cost of a request

Every completion tells you what it cost without any SDK: the
`X-Pareta-Billed` response header is the debit in micro-USD, and
`X-Pareta-Frontier-Would-Have-Cost` is what a single list-priced frontier
call on the same prompt would have cost — each response carries its own
savings receipt. Streamed responses deliver the same two numbers as SSE
comment lines just before `[DONE]`. See the
[HTTP API reference](../reference/http-api.md) for details.

## Handling metering and not-ready errors

Two error cases are specific to running inference. Both subclass `ParetaError`, so a single `except ParetaError` is a fine catch-all; the specific classes let you branch.

**Python**

```python
from pareta import (
    Pareta,
    InsufficientCreditsError,   # 402: org balance empty
    EndpointNotReadyError,      # 503: a serving backend is warming / briefly unavailable
)

with Pareta.from_env() as pa:
    try:
        resp = pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Hello"}],
        )
        print(resp.choices[0].message.content)
    except InsufficientCreditsError:
        # Balance hit zero. Top up in the dashboard (billing is browser-only);
        # the SDK exposes no balance or payment surface.
        print("Out of credit. Top up in the dashboard, then retry.")
    except EndpointNotReadyError:
        # A serving backend behind auto is warming. The SDK already retried
        # the 503 with backoff; wait briefly and retry the call.
        print("Backend warming — retry shortly.")
```

**TypeScript**

```typescript
import {
  Pareta,
  InsufficientCreditsError,   // 402: org balance empty
  EndpointNotReadyError,      // 503: a serving backend is warming / briefly unavailable
} from "pareta";

const pa = Pareta.fromEnv();
try {
  const resp = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "Hello" }],
  });
  console.log(resp.choices[0].message.content);
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    // Balance hit zero. Top up in the dashboard (billing is browser-only);
    // the SDK exposes no balance or payment surface.
    console.log("Out of credit. Top up in the dashboard, then retry.");
  } else if (e instanceof EndpointNotReadyError) {
    // A serving backend behind auto is warming. The SDK already retried
    // the 503 with backoff; wait briefly and retry the call.
    console.log("Backend warming — retry shortly.");
  } else {
    throw e;
  }
}
```

Transient failures (429 rate limits, 5xx, connection timeouts) are retried automatically with exponential backoff, `max_retries` times (default 2). You only see `RateLimitError` or `APITimeoutError` after retries are exhausted. See [Errors](errors-and-retries.md) for the full hierarchy.

## Using the OpenAI SDK instead

Because Pareta is one OpenAI-compatible endpoint, you don't need this SDK to *call* it. Point the `openai` client at Pareta's base URL with your `pareta_sk_` key. Note the `/v1` suffix the OpenAI client expects:

**Python**

```python
from openai import OpenAI

client = OpenAI(api_key="pareta_sk_...", base_url="https://api.pareta.ai/v1")

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What is the invoice total?"}],
)
print(resp.choices[0].message.content)
```

**TypeScript**

```typescript
import OpenAI from "openai";

const client = new OpenAI({ apiKey: "pareta_sk_...", baseURL: "https://api.pareta.ai/v1" });

const resp = await client.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "What is the invoice total?" }],
});
console.log(resp.choices[0].message.content);
```

Tooling that discovers model ids by listing keeps working too: `models.list()` (`GET /v1/models`) returns exactly one entry, `"auto"` — there is only one model id to call. Field details in the [models reference](../reference/models.md).

Streaming, `temperature`, `max_tokens`, and the rest work exactly as they do against OpenAI. Metering still applies: a zero balance returns a 402, which the `openai` client surfaces as its own status error. Reach for the Pareta SDK when you want typed errors and the control plane: matching intent to the task catalog ([core concepts](core-concepts.md)) and [running evals](evaluation.md).



---

<!-- guide/evaluation.md -->

# Evaluating on your own data

Benchmarks tell you which model wins on someone else's data. This page is about the only number that matters: how `model="auto"` scores on *your* rows.

Upload your data, say what the model should do with each item, and Pareta identifies the **grading contract** — how your data will be scored — and shows it to you before anything runs. Then it runs `"auto"` and the frontier baselines you name on the same items and returns per-contender quality with confidence intervals and cost. No GPUs to size, no scorer to wire up, no judge to host.

An eval set is **data + intent**: the same rows can mean different jobs, and only you know which, so `intent` — one sentence on what the model should do with each item — is **required**. From your intent and the data's shape, the binder resolves the grading contract; you never have to know the contract's name.

The shape is always the same:

1. State your **intent** + upload your rows → the binder picks the **grading contract** (preview it with `evals.propose_contract`), or turn them into an **eval set** directly with `evals.sets.create`.
2. Kick off an **eval run** with `"auto"` as the candidate and frontier baselines to beat (`evals.runs.create`), optionally waiting for it to finish.
3. Read `run.results` to compare quality and cost; read `run.cost` for the bill.

## Benchmark Pareta itself: `"auto"` as a contender

The candidate is the literal string `"auto"`, and the eval runs Pareta's
routing brain against every item — the same planning, routing, and
verification that serves your production traffic, scored by the same scorer
as the baselines. The per-contender result rows (quality mean + CI, mean cost
per item) are the product's core claim, measured on your data:

```python
run = client.evals.runs.create(
    eval_set=my_set,
    models=["auto"],          # the contender: Pareta's routing brain
    frontier=["gpt-5.5"],     # the frontier baseline to beat
)
```

A completed run's rows let you read the verdict directly: overlapping
quality CIs at lower cost = frontier-grade; higher mean without overlap =
ahead. Auto's failures count as errors (not skips) — availability is part of
what a benchmark should measure.


## A complete run, top to bottom

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()  # reads PARETA_API_KEY (and optional PARETA_BASE_URL)

run = pa.evals.runs.create(
    intent="extract the effective and termination dates from each contract",
    items=[
        {"input": "Effective as of January 1, 2026, ...", "expected": {"effective_date": "2026-01-01"}},
        {"input": "This Agreement terminates on 2027-12-31 ...", "expected": {"termination_date": "2027-12-31"}},
    ],
    models=["auto"],                 # the contender
    frontier="benchmarked",          # baselines already benchmarked on this task
    wait=True,                       # block until the run is terminal
)

print(run.status)          # "completed"
print(f"billed ${run.cost}")  # Decimal dollars, floored to cents

for r in run.results:
    print(f"{r.model_id:16} {(r.kind or ''):8} q={r.quality_mean:.3f} "
          f"[{r.quality_ci_low:.3f}, {r.quality_ci_high:.3f}]  "
          f"~{r.mean_cost_micro_usd} uUSD/item  "
          f"({r.n_succeeded} ok, {r.error_count} err)")
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv(); // reads PARETA_API_KEY (and optional PARETA_BASE_URL)

const run = await pa.evals.runs.create({
  intent: "extract the effective and termination dates from each contract",
  items: [
    { input: "Effective as of January 1, 2026, ...", expected: { effective_date: "2026-01-01" } },
    { input: "This Agreement terminates on 2027-12-31 ...", expected: { termination_date: "2027-12-31" } },
  ],
  models: ["auto"],               // the contender
  frontier: "benchmarked",        // baselines already benchmarked on this task
  wait: true,                     // block until the run is terminal
});

console.log(run.status);          // "completed"
console.log(`billed $${run.cost}`); // dollar string, floored to cents

for (const r of run.results) {
  console.log(
    `${r.modelId} ${r.kind ?? ""} q=${r.qualityMean} ` +
      `[${r.qualityCiLow}, ${r.qualityCiHigh}]  ` +
      `~${r.meanCostMicroUsd} uUSD/item  ` +
      `(${r.nSucceeded} ok, ${r.errorCount} err)`,
  );
}
```

That single call created the eval set inline, started the run, polled it to completion, and returned aggregates per contender. Everything below unpacks the pieces so you can vary them.

`models=` is always `["auto"]` — individual open-weights models are not part of the eval surface; they stay behind auto's routing. Frontier (vendor) ids are in the clear, and `frontier=` chooses which of them get scored alongside.

## Step 1: build an eval set

An eval set is your rows + your intent, bound to a grading contract. Create one explicitly when you want to reuse it across several runs. Pass `intent` (required) — the binder resolves the grading contract from it and your data's shape, and auto-binds a clean single match:

**Python**

```python
eval_set = pa.evals.sets.create(
    intent="extract the effective and termination dates from each contract",
    items=[
        {"input": "Effective as of January 1, 2026, ...", "expected": {"effective_date": "2026-01-01"}},
        {"input": "This Agreement terminates on 2027-12-31 ...", "expected": {"termination_date": "2027-12-31"}},
    ],
    name="Q2 contracts sample",   # optional; defaults to "sdk eval set (N items)"
)

print(eval_set.id)               # pass this to runs.create(eval_set=...)
print(eval_set.task_id)          # the contract the binder chose, e.g. "contract-key-fields"
print(eval_set.intent)           # your intent, stored on the set
print(eval_set.item_count)       # 2
print(eval_set.scoring_strategy) # e.g. "contract" — how this contract scores
```

**TypeScript**

```typescript
const evalSet = await pa.evals.sets.create({
  intent: "extract the effective and termination dates from each contract",
  items: [
    { input: "Effective as of January 1, 2026, ...", expected: { effective_date: "2026-01-01" } },
    { input: "This Agreement terminates on 2027-12-31 ...", expected: { termination_date: "2027-12-31" } },
  ],
  name: "Q2 contracts sample", // optional; defaults to "sdk eval set (N items)"
});

console.log(evalSet.id);              // pass this to runs.create({ evalSet: ... })
console.log(evalSet.taskId);          // the contract the binder chose, e.g. "contract-key-fields"
console.log(evalSet.intent);          // your intent, stored on the set
console.log(evalSet.itemCount);       // 2
console.log(evalSet.scoringStrategy); // e.g. "contract" — how this contract scores
```

`intent` and `items` are both required (the SDK raises if either is missing or empty). `task` is optional — omit it and the binder picks the contract; pass `task="..."` to pin one explicitly. When the binder can't safely choose (your intent and the data disagree, the set looks mixed, or nothing specific fits), `create` raises with the proposals so you decide — it never binds the wrong contract silently.

### Preview the binding first: `propose_contract`

To see which contract Pareta will use — and exactly how it scores — before persisting anything, call `propose_contract`. It's stateless (nothing is created):

**Python**

```python
proposal = pa.evals.propose_contract(
    intent="extract the effective and termination dates from each contract",
    items=[{"input": "...", "expected": {"effective_date": "2026-01-01"}}],
)
print(proposal.bound_task)   # the contract a task-less create would bind, or None if you must choose
for p in proposal.proposals:
    print(p.task_id, p.confidence, p.evidence.get("validated_n"), "/", p.evidence.get("total_n"))
```

**TypeScript**

```typescript
const proposal = await pa.evals.proposeContract({
  intent: "extract the effective and termination dates from each contract",
  items: [{ input: "...", expected: { effective_date: "2026-01-01" } }],
});
console.log(proposal.boundTask); // the contract a task-less create would bind, or null if you must choose
```

When no specific contract fits your data, the binder offers the **custom-eval** universal floor — graded by a judge panel on your stated intent (win rate vs the frontier anchor), so no dataset dead-ends. It's a floor you opt into (`task="custom-eval"`), never an auto-bind; a matched contract grades more precisely.

Each item is a row in the contract's input schema; the rows go up as JSONL on the wire. To inspect a contract's schema and pull sample items, use `tasks.retrieve(task_id, examples_n=...)` — see the [tasks reference](../reference/tasks.md).

Manage sets like any other resource:

**Python**

```python
pa.evals.sets.list()                  # -> list[EvalSet]
pa.evals.sets.retrieve(eval_set.id)   # -> EvalSet
pa.evals.sets.delete(eval_set.id)     # -> None
```

**TypeScript**

```typescript
await pa.evals.sets.list();             // -> EvalSet[]
await pa.evals.sets.retrieve(evalSet.id); // -> EvalSet
await pa.evals.sets.delete(evalSet.id);   // -> void
```

### Document and image tasks

Some tasks score over documents (PDFs, scanned invoices, images) rather than plain text. A task tells you this via `task.has_blob_input == True`. For those, each row references a binary that you attach after creating the set, one field at a time:

**Python**

```python
eval_set = pa.evals.sets.create(
    intent="extract the total and vendor from each invoice",
    task="invoice-extraction",
    items=[
        {"expected": {"total": "1240.00", "vendor": "Katana ML"}},   # the doc is attached next
        {"expected": {"total": "89.50", "vendor": "Acme"}},
    ],
)

# Attach the PDF for row 0's `document` field.
pa.evals.sets.upload_document(
    eval_set.id,
    "invoices/katana-0001.pdf",   # path, raw bytes, or a binary file-like object
    idx=0,                        # 0-based row index
    field_name="document",        # the blob input field on this task
)

pa.evals.sets.upload_document(eval_set.id, "invoices/acme-0002.pdf", idx=1, field_name="document")
```

**TypeScript**

```typescript
const evalSet = await pa.evals.sets.create({
  intent: "extract the total and vendor from each invoice",
  task: "invoice-extraction",
  items: [
    { expected: { total: "1240.00", vendor: "Katana ML" } }, // the doc is attached next
    { expected: { total: "89.50", vendor: "Acme" } },
  ],
});

// Attach the PDF for row 0's `document` field.
await pa.evals.sets.uploadDocument(
  evalSet.id,
  "invoices/katana-0001.pdf", // path, Blob, or bytes
  {
    idx: 0,                 // 0-based row index
    fieldName: "document",  // the blob input field on this task
  },
);

await pa.evals.sets.uploadDocument(evalSet.id, "invoices/acme-0002.pdf", { idx: 1, fieldName: "document" });
```

`upload_document` collapses the whole upload dance into one call. Files under 5 MiB go up inline; larger files get a signed URL and stream straight to storage. It accepts a path (`str`/`Path`), raw `bytes`, or any object with `.read()`; anything else raises `TypeError`. The MIME type is guessed from the filename and can be overridden with `mime=`:

**Python**

```python
with open("invoices/scan.tiff", "rb") as f:
    pa.evals.sets.upload_document(eval_set.id, f, idx=2, field_name="document", mime="image/tiff")
```

**TypeScript**

```typescript
import { readFile } from "node:fs/promises";

const bytes = await readFile("invoices/scan.tiff");
await pa.evals.sets.uploadDocument(evalSet.id, bytes, { idx: 2, fieldName: "document", mime: "image/tiff" });
```

Frontier baselines on document tasks are automatically vision-filtered — you never accidentally score a contract scan against a text-only model.

## Step 2: run the eval

`evals.runs.create` is the workhorse. You can drive an existing set, or create one inline in the same call.

**Python**

```python
# Against an existing set
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier="benchmarked", wait=True)

# Inline: create the set and run it in one shot
run = pa.evals.runs.create(
    intent="extract the key fields from each contract",
    items=[{"input": "...", "expected": {...}}],
    models=["auto"],
    frontier="benchmarked",
    wait=True,
)
```

**TypeScript**

```typescript
// Against an existing set
let run = await pa.evals.runs.create({ evalSet: evalSet.id, models: ["auto"], frontier: "benchmarked", wait: true });

// Inline: create the set and run it in one shot
run = await pa.evals.runs.create({
  intent: "extract the key fields from each contract",
  items: [{ input: "...", expected: {} }],
  models: ["auto"],
  frontier: "benchmarked",
  wait: true,
});
```

You must pass **either** `eval_set=<id>` **or** `items=… + intent=…` (with `task=` optional); the SDK raises `ValueError` if you give neither. `models` is required — pass `["auto"]`; `frontier=` names the baselines it is measured against. Each run is **metered**: the org balance is debited for the compute across auto and the frontier baselines. If the balance is empty, `create` raises `InsufficientCreditsError` (402). Top-up is browser-only — the SDK never exposes balance or payment methods. See [Errors and metering](errors-and-retries.md).

### Choosing frontier baselines

`frontier=` controls which vendor models get scored alongside `"auto"`, so the report shows you exactly how much quality (and cost) you're trading. It accepts a keyword or an explicit list, resolved SDK-side:

| `frontier=` | Baselines scored |
| --- | --- |
| `None` or `"none"` (default `None`) | none — `"auto"` alone |
| `"all"` | every frontier model available for the task |
| `"benchmarked"` | frontier models Pareta has already benchmarked on the task (vision-filtered for document tasks) |
| `["gpt-5.5", "claude-sonnet-4-6"]` | exactly these frontier model ids |

**Python**

```python
# Just auto, no baseline
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier="none", wait=True)

# Everything in the frontier pool for the task
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier="all", wait=True)

# A hand-picked baseline
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier=["gpt-5.5"], wait=True)
```

**TypeScript**

```typescript
// Just auto, no baseline
let run = await pa.evals.runs.create({ evalSet: evalSet.id, models: ["auto"], frontier: "none", wait: true });

// Everything in the frontier pool for the task
run = await pa.evals.runs.create({ evalSet: evalSet.id, models: ["auto"], frontier: "all", wait: true });

// A hand-picked baseline
run = await pa.evals.runs.create({ evalSet: evalSet.id, models: ["auto"], frontier: ["gpt-5.5"], wait: true });
```

The `"all"` and `"benchmarked"` keywords need to know the contract. When you pin one (`task=…`) the SDK already has it; otherwise it reads the bound contract off the set — the one you passed as `eval_set=…`, or the one the binder just chose for an inline `items=… + intent=…` create. If it still can't resolve a contract it raises `ValueError`, and an unrecognized keyword (anything other than `"all"`/`"benchmarked"`/`"none"`) raises `ValueError` too.

To see and pin the roster yourself, list it first:

**Python**

```python
roster = pa.evals.frontier_models(task="contract-key-fields")
for m in roster:
    print(m.id, m.vendor, "vision" if m.vision else "text", "benchmarked" if m.benchmarked else "-")

# Pin two of them explicitly
ids = [m.id for m in roster if m.benchmarked][:2]
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier=ids, wait=True)
```

**TypeScript**

```typescript
const roster = await pa.evals.frontierModels("contract-key-fields");
for (const m of roster) {
  console.log(m.id, m.vendor, m.vision ? "vision" : "text", m.benchmarked ? "benchmarked" : "-");
}

// Pin two of them explicitly
const ids = roster.filter((m) => m.benchmarked).map((m) => m.id).slice(0, 2);
const run = await pa.evals.runs.create({ evalSet: evalSet.id, models: ["auto"], frontier: ids, wait: true });
```

`frontier_models()` annotates `benchmarked` and applies the capability filter only when you pass `task=`. Without a task it returns the full roster, unannotated.

### Waiting, or not

By default `create` returns as soon as the run is queued, so you can poll on your own schedule:

**Python**

```python
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"])
print(run.status)   # "running" (or queued)

run = pa.evals.runs.retrieve(run.id)   # refetch full state
while not run.is_terminal:
    run = pa.evals.runs.retrieve(run.id)
```

**TypeScript**

```typescript
let run = await pa.evals.runs.create({ evalSet: evalSet.id, models: ["auto"] });
console.log(run.status); // "running" (or queued)

run = await pa.evals.runs.retrieve(run.id); // refetch full state
while (!run.isTerminal) {
  run = await pa.evals.runs.retrieve(run.id);
}
```

Or let the SDK block for you. `wait=True` polls `runs.retrieve` every `poll_interval` seconds (default 3.0) until the run is terminal, up to `timeout` seconds (default 900.0), then returns the final `EvalRun`. If the deadline passes first it raises `ParetaError`. You can also poll an already-started run with the same semantics:

**Python**

```python
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"])
run = pa.evals.runs.wait(run.id, poll_interval=5.0, timeout=1800.0)
```

**TypeScript**

```typescript
let run = await pa.evals.runs.create({ evalSet: evalSet.id, models: ["auto"] });
run = await pa.evals.runs.wait(run.id, { pollInterval: 5.0, timeout: 1800.0 });
```

`is_terminal` is true when `status` is `"completed"` or `"failed"`. On failure, read `run.error_detail` for the message.

## Step 3: read the results

A terminal `EvalRun` carries one `EvalResult` per contender in `run.results` — `"auto"` plus each frontier baseline — and the bill.

**Python**

```python
run = pa.evals.runs.retrieve(run_id)

if run.status == "failed":
    print("run failed:", run.error_detail)
else:
    auto = next(r for r in run.results if r.model_id == "auto")
    print(f"auto: q={auto.quality_mean:.3f} @ ~{auto.mean_cost_micro_usd} uUSD/item")

    for r in run.results:
        print(r.model_id, r.kind, r.quality_mean,
              r.quality_ci_low, r.quality_ci_high,
              r.mean_cost_micro_usd, r.n_succeeded, r.error_count)
```

**TypeScript**

```typescript
const run = await pa.evals.runs.retrieve(runId);

if (run.status === "failed") {
  console.log("run failed:", run.errorDetail);
} else {
  const auto = run.results.find((r) => r.modelId === "auto")!;
  console.log(`auto: q=${auto.qualityMean} @ ~${auto.meanCostMicroUsd} uUSD/item`);

  for (const r of run.results) {
    console.log(
      r.modelId, r.kind, r.qualityMean,
      r.qualityCiLow, r.qualityCiHigh,
      r.meanCostMicroUsd, r.nSucceeded, r.errorCount,
    );
  }
}
```

Each `EvalResult` has:

- `model_id` — `"auto"` for Pareta's row; the vendor id for each frontier baseline.
- `kind` — `"frontier"` on the baseline rows. Filter on it to separate the contender from what it is measured against.
- `quality_mean`, `quality_ci_low`, `quality_ci_high` — mean score in `[0, 1]` with a 95% confidence interval. Use the interval: two contenders whose CIs overlap are not meaningfully different on this sample, so add rows before declaring a winner.
- `mean_cost_micro_usd` — average cost per item in **micro-USD** (1,000,000 = $1.00). This stays in micro-USD on purpose: flooring sub-cent unit rates to whole cents would erase the auto-vs-frontier cost gap that the whole exercise is about.
- `n_succeeded`, `error_count` — how many items scored vs. errored for that contender.
- `per_item` — the per-item rows, each an `EvalItemResult` with `idx`, `score`, `error`, and `prediction` (the model's raw output, truncated). Reach for `prediction` when a `score` is surprisingly low — it's the actual answer, so you can see *why* it lost points without re-running the eval.

### What the run cost

The run total comes back two ways. `run.cost` is what you're billed — a `Decimal` in dollars, **floored to whole cents** (the SDK never rounds a charge up). `run.cost_micro_usd` is the raw integer for precise accounting.

**Python**

```python
print(run.cost)             # Decimal('0.07')  -> dollars, floored to cents
print(run.cost_micro_usd)   # 74211            -> raw micro-USD
```

**TypeScript**

```typescript
console.log(run.cost);         // "0.07"  -> dollar string, floored to cents
console.log(run.costMicroUsd); // 74211   -> raw micro-USD
```

A run that costs less than a cent reads `Decimal("0.00")` on `run.cost` while still carrying its true micro-USD value on `run.cost_micro_usd`. The same money convention applies everywhere in the SDK; see [Errors and metering](errors-and-retries.md) for the full picture.

Every response object also keeps the raw server JSON: `run.to_dict()`, `result.to_dict()`, and `eval_set.to_dict()` give you lossless access to anything not yet surfaced as a typed field.

## Async

Every method has an async twin on `AsyncPareta` with the same signatures. Run evals concurrently and await the results:

**Python**

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        run = await pa.evals.runs.create(
            intent="extract the key fields from each contract",
            items=[{"input": "...", "expected": {...}}],
            models=["auto"],
            frontier="benchmarked",
            wait=True,
        )
        for r in run.results:
            print(r.model_id, r.kind, r.quality_mean)
        print("billed", run.cost)

asyncio.run(main())
```

**TypeScript**

In TypeScript there is no separate async client — `Pareta` is already Promise-only, so every I/O method is just `await`ed. Concurrency is `Promise.all`:

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

const run = await pa.evals.runs.create({
  intent: "extract the key fields from each contract",
  items: [{ input: "...", expected: {} }],
  models: ["auto"],
  frontier: "benchmarked",
  wait: true,
});

for (const r of run.results) {
  console.log(r.modelId, r.kind, r.qualityMean);
}
console.log("billed", run.cost);

// Fan several runs out concurrently with Promise.all:
const [a, b] = await Promise.all([
  pa.evals.runs.create({ evalSet: setA, models: ["auto"], frontier: "benchmarked", wait: true }),
  pa.evals.runs.create({ evalSet: setB, models: ["auto"], frontier: "benchmarked", wait: true }),
]);
```

`await pa.evals.runs.wait(runId)` and `await pa.evals.frontierModels(task)` work the same way. Document uploads are async too: `await pa.evals.sets.uploadDocument(...)`.

## From eval to production

There is no deploy step. The routing that just won your eval is the same routing that serves `model="auto"` in production — keep sending it your traffic:

**Python**

```python
resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Extract the effective date from: ..."}],
)
print(resp.choices[0].message.content)
```

**TypeScript**

```typescript
const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Extract the effective date from: ..." }],
});
console.log(resp.choices[0].message.content);
```

Inference is OpenAI-compatible and metered the same way evals are — one debit per request, no matter how many internal model calls auto's plan makes. To watch the production side of the story — requests, success rate, spend, and the projected savings vs frontier — poll `auto.metrics()`. See [Running inference](./inference.md) and [Cost & quality monitoring](../examples/cost-and-metrics.md).

## See also

- [The tasks reference](../reference/tasks.md) — find the right task id, inspect its schema, pull example rows.
- [Running inference](./inference.md) — the OpenAI-compatible `model="auto"` chat surface.
- [Cost & quality monitoring](../examples/cost-and-metrics.md) — read run costs and watch live auto traffic with `auto.metrics()`.
- [Errors and metering](errors-and-retries.md) — `InsufficientCreditsError`, the money convention, and the exception hierarchy.



---

<!-- guide/errors-and-retries.md -->

# Errors, retries & timeouts

Every failure the SDK can raise is a subclass of `ParetaError`, so one `except`
clause catches everything, and a more specific clause catches exactly the case
you care about. The client also retries transient failures for you (network
blips, 429s, 5xx) with exponential backoff before giving up. This page is the
map: which exception means what, what is retried automatically, and how to tune
the timeout and retry budget.

Import the exceptions straight from the package:

**Python**

```python
from pareta import (
    Pareta,
    ParetaError,                # base class for everything below
    APIConnectionError,         # never reached the server (DNS/TCP/TLS)
    APITimeoutError,            # subclass of APIConnectionError
    APIStatusError,             # any non-2xx from the server
    BadRequestError,            # 400, 422
    AuthenticationError,        # 401
    PermissionDeniedError,      # 403
    InsufficientCreditsError,   # 402 — org out of balance
    NotFoundError,              # 404
    ConflictError,              # 409
    RateLimitError,             # 429
    EndpointNotReadyError,      # 503 — a backend behind auto warming/briefly down
)
```

**TypeScript**

```typescript
import {
  Pareta,
  ParetaError,                // base class for everything below
  APIConnectionError,         // never reached the server (DNS/TCP/TLS)
  APITimeoutError,            // subclass of APIConnectionError
  APIStatusError,             // any non-2xx from the server
  BadRequestError,            // 400, 422
  AuthenticationError,        // 401
  PermissionDeniedError,      // 403
  InsufficientCreditsError,   // 402 — org out of balance
  NotFoundError,              // 404
  ConflictError,              // 409
  RateLimitError,             // 429
  EndpointNotReadyError,      // 503 — a backend behind auto warming/briefly down
} from "pareta";
```

## The hierarchy

```
ParetaError
├── APIConnectionError          request never reached the server
│   └── APITimeoutError         timed out before any response
└── APIStatusError              server returned a non-2xx status
    ├── BadRequestError         400, 422
    ├── AuthenticationError     401
    ├── InsufficientCreditsError 402
    ├── PermissionDeniedError   403
    ├── NotFoundError           404
    ├── ConflictError           409
    ├── RateLimitError          429
    └── EndpointNotReadyError   503
```

`ParetaError` is also raised directly (not as an `APIStatusError`) in two
non-HTTP cases: constructing a client with no API key, and an `evals.runs.wait()`
poll loop that exceeds its `timeout`. See [Timeouts](#timeouts) below.

## Status code to exception

The server is FastAPI, so error bodies are `{"detail": "<message>"}` with an HTTP
status. The SDK maps the status to the most specific subclass so you catch by
meaning, not by sniffing integers.

| Status | Exception | What it means |
|--------|-----------|---------------|
| 400, 422 | `BadRequestError` | Request validation failed (bad params, malformed body) |
| 401 | `AuthenticationError` | API key missing or invalid |
| 402 | `InsufficientCreditsError` | Org is out of balance; top up in the dashboard |
| 403 | `PermissionDeniedError` | Authenticated, but not allowed to do this |
| 404 | `NotFoundError` | Task / eval set / run id does not exist |
| 409 | `ConflictError` | Conflict (transient lock/contention) |
| 429 | `RateLimitError` | Rate limited; honor `Retry-After` |
| 503 | `EndpointNotReadyError` | A serving backend behind `auto` is warming or briefly unavailable |
| other 5xx | `APIStatusError` | Generic server error |

## Reading an `APIStatusError`

Every `APIStatusError` carries the fields you need to log and debug. `request_id`
comes from the `x-request-id` response header and is the fastest thing to quote
in a support thread.

**Python**

```python
from pareta import Pareta, APIStatusError

with Pareta.from_env() as pa:
    try:
        pa.tasks.retrieve("nonexistent-task")
    except APIStatusError as e:
        print(e.status_code)   # 404
        print(e.detail)        # server's `detail` string (or raw body)
        print(e.request_id)    # "req_…" — quote this in bug reports
        print(e.response)      # the underlying httpx.Response, for advanced use
```

**TypeScript**

```typescript
import { Pareta, APIStatusError } from "pareta";

const pa = Pareta.fromEnv();
try {
  await pa.tasks.retrieve("nonexistent-task");
} catch (e) {
  if (e instanceof APIStatusError) {
    console.log(e.status);     // 404
    console.log(e.detail);     // server's `detail` string (or raw body)
    console.log(e.requestId);  // "req_…" — quote this in bug reports
    console.log(e.response);   // the underlying fetch Response, for advanced use
  }
}
```

`str(e)` is the server's `detail` message when present, otherwise `HTTP <code>`.

## The errors worth catching

Most code only needs to handle a handful of these explicitly. The rest are fine
to let bubble up to a top-level `except ParetaError`.

### `InsufficientCreditsError` (402) — out of balance

Both inference and evals are metered against your org's balance. A successful
[`chat.completions.create()`](./inference.md) debits the balance; an
[`evals.runs.create()`](evaluation.md) debits for the auto and frontier compute it
runs. When the balance can't cover the call, you get a 402. Top-up is
browser-only — the SDK exposes no balance or payment surface — so the right move
is to surface a clear message pointing at the dashboard.

**Python**

```python
from pareta import Pareta, InsufficientCreditsError

with Pareta.from_env() as pa:
    try:
        resp = pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Extract the parties."}],
        )
    except InsufficientCreditsError:
        raise SystemExit("Org balance is empty. Top up at https://pareta.ai dashboard.")
```

**TypeScript**

```typescript
import { Pareta, InsufficientCreditsError } from "pareta";

const pa = Pareta.fromEnv();
try {
  const resp = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "Extract the parties." }],
  });
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    throw new Error("Org balance is empty. Top up at https://pareta.ai dashboard.");
  }
  throw e;
}
```

### `NotFoundError` (404) — wrong id

A stale or mistyped task id, eval set id, or run id.

**Python**

```python
from pareta import Pareta, NotFoundError

with Pareta.from_env() as pa:
    try:
        task = pa.tasks.retrieve("nonexistent-task")
    except NotFoundError:
        match = pa.tasks.match("extract key fields from contracts")  # recover the real id
        if match.chosen:
            task = pa.tasks.retrieve(match.chosen.task_id)
```

**TypeScript**

```typescript
import { Pareta, NotFoundError } from "pareta";

const pa = Pareta.fromEnv();
let task;
try {
  task = await pa.tasks.retrieve("nonexistent-task");
} catch (e) {
  if (e instanceof NotFoundError) {
    const match = await pa.tasks.match("extract key fields from contracts");  // recover the real id
    if (match.chosen?.taskId) task = await pa.tasks.retrieve(match.chosen.taskId);
  } else {
    throw e;
  }
}
```

### `EndpointNotReadyError` (503) — a backend is still warming

`auto` routes each request across serving backends that Pareta manages.
Occasionally the one your request needs is warming up (a cold start) or briefly
unavailable, and the request surfaces a 503. The SDK already retries 503 a
couple of times (see [Automatic retries](#automatic-retries)), which absorbs
most warm-ups; if it still surfaces, there is nothing to start or fix on your
side — wait briefly and re-issue the request.

**Python**

```python
import time

from pareta import Pareta, EndpointNotReadyError

with Pareta.from_env() as pa:
    try:
        resp = pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "ping"}],
        )
    except EndpointNotReadyError:
        time.sleep(10)                       # still warming after the SDK's own retries
        resp = pa.chat.completions.create(   # same request, second pass
            model="auto",
            messages=[{"role": "user", "content": "ping"}],
        )
```

**TypeScript**

```typescript
import { Pareta, EndpointNotReadyError } from "pareta";

const pa = Pareta.fromEnv();
const request = () =>
  pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "ping" }],
  });
let resp;
try {
  resp = await request();
} catch (e) {
  if (e instanceof EndpointNotReadyError) {
    await new Promise((r) => setTimeout(r, 10_000));  // still warming after the SDK's own retries
    resp = await request();                           // same request, second pass
  } else {
    throw e;
  }
}
```

### `RateLimitError` (429) — slow down

Already retried automatically, honoring the server's `Retry-After`. You only see
it after the retry budget is exhausted. Back off and try again later.

**Python**

```python
from pareta import Pareta, RateLimitError

with Pareta.from_env() as pa:
    try:
        pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "hi"}],
        )
    except RateLimitError as e:
        print(f"Still rate limited after retries (request {e.request_id}); back off.")
```

**TypeScript**

```typescript
import { Pareta, RateLimitError } from "pareta";

const pa = Pareta.fromEnv();
try {
  await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "hi" }],
  });
} catch (e) {
  if (e instanceof RateLimitError) {
    console.log(`Still rate limited after retries (request ${e.requestId}); back off.`);
  } else {
    throw e;
  }
}
```

### `AuthenticationError` (401) vs missing key

A 401 means the key reached the server and was rejected (wrong or revoked). That
is distinct from constructing a client with *no* key at all, which fails fast
client-side with a plain `ParetaError` before any request goes out:

**Python**

```python
import pareta

try:
    pa = pareta.Pareta(api_key="")   # or PARETA_API_KEY unset with from_env()
except pareta.ParetaError as e:
    print(e)  # "missing API key. Pass api_key=… or set PARETA_API_KEY …"
```

**TypeScript**

```typescript
import { Pareta, ParetaError } from "pareta";

try {
  const pa = new Pareta({ apiKey: "" });   // or PARETA_API_KEY unset with Pareta.fromEnv()
} catch (e) {
  if (e instanceof ParetaError) {
    console.log(e.message);  // "missing API key. Pass apiKey: … or use Pareta.fromEnv() …"
  }
}
```

## Pre-flight `ValueError` / `TypeError`

Some mistakes never become an HTTP call. The SDK validates the obvious ones up
front and raises the standard Python exception — not a `ParetaError` — because
they are programming errors, not server responses:

- [`chat.completions.create()`](./inference.md) raises `ValueError` if `model`
  or `messages` is empty.
- [`tasks.match()`](../reference/tasks.md) raises `ValueError` if `query` is empty.
- [`evals.sets.create()`](evaluation.md) raises `ValueError` if `items` or
  `intent` is empty.
- [`evals.runs.create()`](evaluation.md) raises `ValueError` if neither
  `eval_set=` nor `items=` (with `intent=`) is supplied, and
  `ValueError`/`TypeError` if `frontier=` is an unparseable keyword or a
  frontier keyword can't be resolved to a task.
- [`evals.sets.upload_document()`](evaluation.md) raises `TypeError` if `file` is
  not a path, bytes, or a binary file-like object.

These are fine to let crash in development; they signal a bug in the call, not a
runtime condition to recover from.

## Automatic retries

The client retries transient failures for you before raising. You usually do not
need a retry loop of your own.

**What is retried:** status codes `408, 409, 429, 500, 502, 503, 504`, plus
connection-level errors that happen *between* attempts. The default budget is
`max_retries=2` (so up to three attempts total).

**Backoff:** if the server sent a `Retry-After` header, the SDK waits that many
seconds (capped at 30s). Otherwise it uses exponential backoff with jitter:
`min(0.5 * 2**attempt, 8.0) + random(0, 0.25)` seconds, so roughly 0.5s, then
1s, capped at 8s.

**What is not retried:** stable 4xx (400, 401, 402, 403, 404, 422) raise
immediately — retrying a bad request or an empty balance won't help. Connection
errors on the very first attempt are surfaced as `APIConnectionError` /
`APITimeoutError` once the budget is exhausted.

Tune the budget per client. Set `max_retries=0` to disable retries entirely:

**Python**

```python
from pareta import Pareta

# More aggressive: up to 6 attempts on transient failures.
pa = Pareta.from_env(max_retries=5)

# No retries — fail fast and handle it yourself.
strict = Pareta.from_env(max_retries=0)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

// More aggressive: up to 6 attempts on transient failures.
const pa = Pareta.fromEnv({ maxRetries: 5 });

// No retries — fail fast and handle it yourself.
const strict = Pareta.fromEnv({ maxRetries: 0 });
```

### Streaming and retries

Retries apply only to the initial handshake (connect and status line). Once SSE
bytes are flowing — token chunks from a streamed
[chat completion](./inference.md) — a mid-stream drop raises immediately, because
the stream cannot be safely resumed. Catch it and restart the request from the
top if you need to.

**Python**

```python
from pareta import Pareta, APIConnectionError

with Pareta.from_env() as pa:
    try:
        for chunk in pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Summarize the contract."}],
            stream=True,
        ):
            piece = chunk.choices[0].delta.content
            if piece:
                print(piece, end="", flush=True)
    except APIConnectionError:
        print("\n[stream dropped — re-issue the request to retry]")
```

**TypeScript**

```typescript
import { Pareta, APIConnectionError } from "pareta";

const pa = Pareta.fromEnv();
try {
  const stream = pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "Summarize the contract." }],
    stream: true,
  });
  for await (const chunk of stream) {
    const piece = chunk.choices[0].delta.content;
    if (piece) process.stdout.write(piece);
  }
} catch (e) {
  if (e instanceof APIConnectionError) {
    console.log("\n[stream dropped — re-issue the request to retry]");
  } else {
    throw e;
  }
}
```

## Timeouts

The default per-request timeout is `httpx.Timeout(60.0, connect=10.0)`: 60s
overall, 10s to establish the connection. A request that exceeds it raises
`APITimeoutError` (a subclass of `APIConnectionError`) after the retry budget is
spent. Override it with any `httpx.Timeout` (or a bare float):

**Python**

```python
import httpx
from pareta import Pareta, APITimeoutError

# 120s overall, 5s to connect — handy for long generations.
pa = Pareta.from_env(timeout=httpx.Timeout(120.0, connect=5.0))

with pa:
    try:
        pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Write a long summary."}],
            max_tokens=4096,
        )
    except APITimeoutError:
        print("Request timed out; consider streaming or a larger timeout.")
```

**TypeScript**

```typescript
import { Pareta, APITimeoutError } from "pareta";

// 120s overall (one budget — there's no separate connect timeout in TS).
const pa = Pareta.fromEnv({ timeout: 120_000 });

try {
  await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "Write a long summary." }],
    max_tokens: 4096,
  });
} catch (e) {
  if (e instanceof APITimeoutError) {
    console.log("Request timed out; consider streaming or a larger timeout.");
  } else {
    throw e;
  }
}
```

### Eval-run wait timeout

[`evals.runs.create(wait=True)`](evaluation.md) and `evals.runs.wait()` are
different: they poll the run to completion. The `timeout` parameter there bounds
the *whole poll loop* (default 900s), not a single HTTP request. If the run
hasn't reached a terminal status (`completed` or `failed`) by the deadline, the
poll helper raises a plain `ParetaError` — the run keeps going server-side, so
you can re-`retrieve()` it later by id.

**Python**

```python
from pareta import Pareta, ParetaError

with Pareta.from_env() as pa:
    try:
        run = pa.evals.runs.create(
            intent="extract the key fields from each contract",
            items=[{"input": "...", "expected": "..."}],
            models=["auto"],
            frontier="benchmarked",
            wait=True,
            timeout=600.0,      # give up waiting after 10 minutes
            poll_interval=5.0,
        )
        print(run.status, run.cost)        # e.g. "completed" Decimal("0.42")
    except ParetaError as e:
        print(e)  # "eval run … did not finish within 600s" — poll later with runs.retrieve(id)
```

**TypeScript**

```typescript
import { Pareta, ParetaError } from "pareta";

const pa = Pareta.fromEnv();
try {
  const run = await pa.evals.runs.create({
    intent: "extract the key fields from each contract",
    items: [{ input: "...", expected: "..." }],
    models: ["auto"],
    frontier: "benchmarked",
    wait: true,
    timeout: 600,        // give up waiting after 10 minutes
    pollInterval: 5,
  });
  console.log(run.status, run.cost);   // e.g. "completed" "0.42"
} catch (e) {
  if (e instanceof ParetaError) {
    console.log(e.message);  // "eval run … did not finish within 600s" — poll later with runs.retrieve(id)
  } else {
    throw e;
  }
}
```

Note that a run finishing with `status == "failed"` is *not* an exception — it's
a terminal state you read off the returned `EvalRun` (`run.is_terminal` is True,
`run.error_detail` carries the message). Only the wait *timeout* raises.

## Async

`AsyncPareta` raises the exact same exception classes; wrap `await` calls in the
same `try`/`except`. Retries, backoff, and timeouts behave identically — backoff
just uses `asyncio.sleep` under the hood.

**Python**

```python
import asyncio
from pareta import AsyncPareta, InsufficientCreditsError, EndpointNotReadyError

async def main():
    async with AsyncPareta.from_env() as pa:
        try:
            resp = await pa.chat.completions.create(
                model="auto",
                messages=[{"role": "user", "content": "Extract the parties."}],
            )
            print(resp.choices[0].message.content)
        except InsufficientCreditsError:
            print("Top up your org balance in the dashboard.")
        except EndpointNotReadyError:
            print("A backend behind auto is still warming — retry shortly.")

asyncio.run(main())
```

**TypeScript**

```typescript
// There is no AsyncPareta in TypeScript — the single `Pareta` client is already
// async: every I/O method returns a Promise, so you just `await` it. The same
// exception classes, retries, backoff, and timeouts apply unchanged.
import { Pareta, InsufficientCreditsError, EndpointNotReadyError } from "pareta";

async function main() {
  const pa = Pareta.fromEnv();
  try {
    const resp = await pa.chat.completions.create({
      model: "auto",
      messages: [{ role: "user", content: "Extract the parties." }],
    });
    console.log(resp.choices[0].message.content);
  } catch (e) {
    if (e instanceof InsufficientCreditsError) {
      console.log("Top up your org balance in the dashboard.");
    } else if (e instanceof EndpointNotReadyError) {
      console.log("A backend behind auto is still warming — retry shortly.");
    } else {
      throw e;
    }
  }
}

main();
```

## A layered handler

A practical pattern: catch the few cases you can act on, then fall back to the
base class so nothing escapes unhandled.

**Python**

```python
from pareta import (
    Pareta,
    InsufficientCreditsError,
    EndpointNotReadyError,
    RateLimitError,
    APITimeoutError,
    ParetaError,
)

with Pareta.from_env() as pa:
    try:
        resp = pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Extract the parties."}],
        )
        print(resp.choices[0].message.content)
    except InsufficientCreditsError:
        print("Out of balance — top up in the dashboard.")
    except EndpointNotReadyError:
        print("A backend is still warming — wait briefly, then retry.")
    except RateLimitError:
        print("Rate limited after retries — back off and try again.")
    except APITimeoutError:
        print("Timed out — raise the timeout or stream the response.")
    except ParetaError as e:
        print(f"Unexpected SDK error: {e}")  # request_id is on APIStatusError subclasses
```

**TypeScript**

```typescript
import {
  Pareta,
  InsufficientCreditsError,
  EndpointNotReadyError,
  RateLimitError,
  APITimeoutError,
  ParetaError,
} from "pareta";

const pa = Pareta.fromEnv();
try {
  const resp = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "Extract the parties." }],
  });
  console.log(resp.choices[0].message.content);
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    console.log("Out of balance — top up in the dashboard.");
  } else if (e instanceof EndpointNotReadyError) {
    console.log("A backend is still warming — wait briefly, then retry.");
  } else if (e instanceof RateLimitError) {
    console.log("Rate limited after retries — back off and try again.");
  } else if (e instanceof APITimeoutError) {
    console.log("Timed out — raise the timeout or stream the response.");
  } else if (e instanceof ParetaError) {
    console.log(`Unexpected SDK error: ${e.message}`);  // requestId is on APIStatusError subclasses
  } else {
    throw e;
  }
}
```

## See also

- [Inference](./inference.md) — OpenAI-compatible chat completions and streaming
- [Evals](evaluation.md) — eval sets, runs, `wait`, and `run.cost`
- [Tasks](../reference/tasks.md) — the benchmark catalog and `match()`



---

<!-- guide/async.md -->

# Async usage

`AsyncPareta` is the asyncio-native client. It mirrors the synchronous [`Pareta`](./quickstart.md) client method-for-method: same constructor, same resource namespaces (`chat`, `models`, `tasks`, `evals`, `auto`, `audio`), same return types. The difference is that request methods are coroutines you `await`, streams are async iterators you drive with `async for`, and many independent calls can run concurrently under one event loop instead of blocking one after another.

Reach for it when you are inside an async app (FastAPI, an aiohttp worker, a Discord bot) or when you want to fan out work: run a batch of prompts against `model="auto"` without waiting on each round trip, or kick off several eval runs at once.

## The client

Build it from the environment, exactly like the sync client. `from_env()` reads `PARETA_API_KEY` and the optional `PARETA_BASE_URL`.

**Python**

```python
import asyncio
from pareta import AsyncPareta


async def main():
    pa = AsyncPareta.from_env()  # reads PARETA_API_KEY
    try:
        models = await pa.models.list()
        for m in models:
            print(m.id, m.owned_by)
    finally:
        await pa.aclose()


asyncio.run(main())
```

**TypeScript**

In TypeScript there is one client and it is already async — every I/O method returns a Promise you `await`. There is no `AsyncPareta`, no event loop to manage, and no `aclose()`: the client holds no owned connection.

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv(); // reads PARETA_API_KEY

const models = await pa.models.list();
for (const m of models) {
  console.log(m.id, m.ownedBy);
}
```

`models.list()` returns the same `ModelList` as the sync path: exactly one entry, `"auto"` — the only model id you pass to `chat.completions.create(model=...)`.

### Lifecycle: prefer `async with`

The client owns an `httpx.AsyncClient` and you must release it. Use `async with` and cleanup is automatic; otherwise call `await pa.aclose()` in a `finally`.

**Python**

```python
import asyncio
from pareta import AsyncPareta


async def main():
    async with AsyncPareta.from_env() as pa:
        models = await pa.models.list()
        print([m.id for m in models])  # ['auto']
    # the underlying HTTP client is closed here


asyncio.run(main())
```

**TypeScript**

The TypeScript client owns no connection, so there is nothing to release — no `async with`, no `aclose()`. Build it once and use it; native `fetch` manages its own pooling. If you need a custom transport (tests, a polyfill), pass `fetch:`.

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const models = await pa.models.list();
console.log([...models].map((m) => m.id)); // ["auto"]
// nothing to close
```

The async lifecycle methods are `await pa.aclose()`, `async with` (which calls `__aenter__` / `__aexit__`). There is no sync `close()` on the async client. If you pass your own `http_client=httpx.AsyncClient(...)`, the SDK will not close it for you; that one is yours to manage.

You can also pass `api_key=`, `base_url=`, `timeout=`, and `max_retries=` directly, same as the sync client:

**Python**

```python
from pareta import AsyncPareta

pa = AsyncPareta(api_key="pareta_sk_...", max_retries=4)
```

**TypeScript**

The constructor takes a single options object with camelCase keys. Note `timeout` is in **milliseconds** here (Python's httpx is seconds).

```typescript
import { Pareta } from "pareta";

const pa = new Pareta({ apiKey: "pareta_sk_...", maxRetries: 4 });
```

## Await every request method

Every resource method that hits the API is a coroutine. Await it.

**Python**

```python
import asyncio
from pareta import AsyncPareta


async def main():
    async with AsyncPareta.from_env() as pa:
        # inference (OpenAI-compatible)
        completion = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Extract the total due."}],
            temperature=0,
        )
        print(completion.choices[0].message.content)
        print(completion.usage.total_tokens, "tokens")

        # catalog discovery
        match = await pa.tasks.match("pull key fields out of contracts")
        if match.matched:
            print("task:", match.chosen.task_id, match.chosen.confidence)

        # auto rollup (requests, success, spend, projected savings)
        metrics = await pa.auto.metrics()
        print(metrics["requests_30d"], "requests in 30d")

        # eval roster
        frontier = await pa.evals.frontier_models(task="contract-key-fields")
        print([f.id for f in frontier])


asyncio.run(main())
```

**TypeScript**

Every method already returns a Promise, so `await` is all you need — no coroutine wrapper, no event loop. `chat.completions.create` takes an options object; extra OpenAI params (`temperature`) pass through verbatim.

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

// inference (OpenAI-compatible)
const completion = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Extract the total due." }],
  temperature: 0,
});
console.log(completion.choices[0].message.content);
console.log(completion.usage.totalTokens, "tokens");

// catalog discovery
const match = await pa.tasks.match("pull key fields out of contracts");
if (match.matched) {
  console.log("task:", match.chosen.taskId, match.chosen.confidence);
}

// auto rollup (requests, success, spend, projected savings)
const metrics = await pa.auto.metrics();
console.log(metrics.requests_30d, "requests in 30d");

// eval roster
const frontier = await pa.evals.frontierModels("contract-key-fields");
console.log(frontier.map((f) => f.id));
```

`chat.completions.create()` is metered: a successful completion debits your org balance — one debit per request, no matter how many internal model calls auto's plan makes. If the balance is empty it raises `InsufficientCreditsError` (402). Top-up is browser-only; the SDK does not expose balance or payment. See [Errors](errors-and-retries.md) and [Billing](core-concepts.md).

## Streaming with `async for`

Streaming chat works in two steps. First `await` the `create(stream=True)` call to get the async iterator, then drive it with `async for`. Each chunk is a `ChatCompletionChunk`; the incremental text is `chunk.choices[0].delta.content` (which can be `None` on non-content frames, so guard it).

**Python**

```python
import asyncio
from pareta import AsyncPareta


async def main():
    async with AsyncPareta.from_env() as pa:
        stream = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Summarize this clause."}],
            stream=True,
        )
        async for chunk in stream:
            delta = chunk.choices[0].delta.content
            if delta:
                print(delta, end="", flush=True)
        print()


asyncio.run(main())
```

**TypeScript**

With `stream: true` the call returns an `AsyncIterable<ChatCompletionChunk>` directly — no separate await for the handshake. Drive it with `for await`. The incremental text is `chunk.choices[0].delta.content`, which can be `null` on non-content frames, so guard it.

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

const stream = pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Summarize this clause." }],
  stream: true,
});
for await (const chunk of stream) {
  const delta = chunk.choices[0].delta.content;
  if (delta) process.stdout.write(delta);
}
console.log();
```

The stream ends on the wire's `[DONE]` sentinel; the async iterator simply stops. Retries apply only to the initial handshake. Once bytes are flowing, a mid-stream drop raises immediately rather than silently resuming.

## Running many calls concurrently

This is the reason to go async. Independent calls can run at the same time under one event loop with `asyncio.gather`, instead of serializing on each network round trip. Reuse one client across all of them so they share the connection pool.

### Fan out inference over a batch

**Python**

```python
import asyncio
from pareta import AsyncPareta

PROMPTS = [
    "Extract the invoice total.",
    "Extract the due date.",
    "Extract the vendor name.",
    "Extract the PO number.",
]


async def classify(pa: AsyncPareta, prompt: str) -> str:
    completion = await pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return completion.choices[0].message.content


async def main():
    async with AsyncPareta.from_env() as pa:
        results = await asyncio.gather(
            *(classify(pa, p) for p in PROMPTS)
        )
        for prompt, answer in zip(PROMPTS, results):
            print(prompt, "->", answer)


asyncio.run(main())
```

**TypeScript**

In TypeScript every call is already a Promise, so concurrency is just `Promise.all` over the calls you kick off — no `asyncio.gather`, no separate async API. Reuse one client so they share the connection pool.

```typescript
import { Pareta } from "pareta";

const PROMPTS = [
  "Extract the invoice total.",
  "Extract the due date.",
  "Extract the vendor name.",
  "Extract the PO number.",
];

const pa = Pareta.fromEnv();

async function classify(prompt: string): Promise<string | null> {
  const completion = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: prompt }],
    temperature: 0,
  });
  return completion.choices[0].message.content;
}

const results = await Promise.all(PROMPTS.map((p) => classify(p)));
PROMPTS.forEach((prompt, i) => console.log(prompt, "->", results[i]));
```

Each of those `create()` calls is metered independently and debits the org balance on success. If your balance runs out mid-batch, the in-flight calls that have not yet been billed raise `InsufficientCreditsError`. With `gather`, the first exception propagates and cancels the rest; pass `return_exceptions=True` if you would rather collect partial results and inspect failures per item.

**Python**

```python
results = await asyncio.gather(
    *(classify(pa, p) for p in PROMPTS),
    return_exceptions=True,
)
for prompt, result in zip(PROMPTS, results):
    if isinstance(result, Exception):
        print(prompt, "FAILED:", result)
    else:
        print(prompt, "->", result)
```

**TypeScript**

`Promise.all` rejects on the first failure, just like `gather`. The equivalent of `return_exceptions=True` is `Promise.allSettled`, which collects a `{ status, value | reason }` per item.

```typescript
const settled = await Promise.allSettled(PROMPTS.map((p) => classify(p)));
PROMPTS.forEach((prompt, i) => {
  const result = settled[i];
  if (result.status === "rejected") {
    console.log(prompt, "FAILED:", result.reason);
  } else {
    console.log(prompt, "->", result.value);
  }
});
```

### Mix resources in one gather

`gather` does not care that the coroutines hit different routes. Kick off an inference call, a catalog match, and your org's auto rollup in one shot:

**Python**

```python
import asyncio
from pareta import AsyncPareta


async def main():
    async with AsyncPareta.from_env() as pa:
        completion, match, metrics = await asyncio.gather(
            pa.chat.completions.create(
                model="auto",
                messages=[{"role": "user", "content": "Extract the parties."}],
            ),
            pa.tasks.match("pull key fields out of contracts"),
            pa.auto.metrics(),
        )
        print(completion.choices[0].message.content)
        if match.matched:
            print("task:", match.chosen.task_id)
        print("requests (30d):", metrics["requests_30d"])


asyncio.run(main())
```

**TypeScript**

`Promise.all` is heterogeneous too — the tuple keeps each result's type.

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

const [completion, match, metrics] = await Promise.all([
  pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "Extract the parties." }],
  }),
  pa.tasks.match("pull key fields out of contracts"),
  pa.auto.metrics(),
]);
console.log(completion.choices[0].message.content);
if (match.matched) {
  console.log("task:", match.chosen?.taskId);
}
console.log("requests (30d):", metrics.requests_30d);
```

### Run several eval runs in parallel

`evals.runs.create(..., wait=True)` polls `runs.retrieve()` until the run is terminal using `asyncio.sleep`, so it never blocks the loop. That makes benchmarking `"auto"` on several of your datasets — one run per eval set — a natural `gather`. Passing `intent=` + `items=` creates the eval set and the run in one call.

**Python**

```python
import asyncio
from pareta import AsyncPareta

JOBS = {
    "extract the payment amount from each contract": [
        {"input": "Acme Corp agrees to pay $5,000 net 30.", "expected": {"amount": "5000"}},
        {"input": "Total due: $1,200 by 2026-07-01.", "expected": {"amount": "1200"}},
    ],
    "extract the total from each invoice": [
        {"input": "INVOICE #4471 ... TOTAL $1,240.00 ...", "expected": {"total": "1240.00"}},
    ],
}


async def main():
    async with AsyncPareta.from_env() as pa:
        # one run per dataset: "auto" against the task's frontier baselines
        runs = await asyncio.gather(
            *(
                pa.evals.runs.create(
                    intent=intent, items=items,
                    models=["auto"], frontier="benchmarked", wait=True,
                )
                for intent, items in JOBS.items()
            )
        )
        for run in runs:
            print(run.id, run.status, "cost", run.cost)  # run.cost is a Decimal in dollars
            for r in run.results:
                print(" ", r.model_id, r.kind, r.quality_mean)


asyncio.run(main())
```

**TypeScript**

`runs.create({ ..., wait: true })` polls `runs.retrieve()` until terminal, so the fan-out is a natural `Promise.all`. Note `run.cost` is a fixed-2dp dollar **string** here (`run.costMicroUsd` is the raw integer).

```typescript
import { Pareta } from "pareta";

const JOBS: Record<string, Array<Record<string, unknown>>> = {
  "extract the payment amount from each contract": [
    { input: "Acme Corp agrees to pay $5,000 net 30.", expected: { amount: "5000" } },
    { input: "Total due: $1,200 by 2026-07-01.", expected: { amount: "1200" } },
  ],
  "extract the total from each invoice": [
    { input: "INVOICE #4471 ... TOTAL $1,240.00 ...", expected: { total: "1240.00" } },
  ],
};

const pa = Pareta.fromEnv();

// one run per dataset: "auto" against the task's frontier baselines
const runs = await Promise.all(
  Object.entries(JOBS).map(([intent, items]) =>
    pa.evals.runs.create({ intent, items, models: ["auto"], frontier: "benchmarked", wait: true }),
  ),
);
for (const run of runs) {
  console.log(run.id, run.status, "cost", run.cost); // run.cost is a dollar string
  for (const r of run.results) {
    console.log(" ", r.modelId, r.kind, r.qualityMean);
  }
}
```

Eval runs are metered against the org balance for the compute used (`"auto"` plus any frontier baselines), and raise `InsufficientCreditsError` on an empty balance. `run.cost` is a `Decimal` in dollars, floored to whole cents (so a sub-cent run reads `Decimal("0.00")`); `run.cost_micro_usd` is the raw integer micro-USD if you need the exact figure. See [Evals](evaluation.md) and [Billing](core-concepts.md).

The `frontier=` keywords pick the vendor baselines. In the async client, `"all"` and `"benchmarked"` resolve the roster by awaiting `evals.frontier_models()` SDK-side, so they need a contract to resolve against (a pinned `task=`, or the contract bound to the eval set — including the one the binder chose for an inline `items=… + intent=…` create):

**Python**

```python
run = await pa.evals.runs.create(
    eval_set=eval_set.id,     # an existing set from evals.sets.create(...)
    models=["auto"],
    frontier="benchmarked",   # or "all", or an explicit list of frontier ids, or None
    wait=True,
)
```

**TypeScript**

`"all"` and `"benchmarked"` resolve the roster SDK-side via `evals.frontierModels()`, so they need a task to resolve against (from `task` or looked up from the eval set).

```typescript
const run = await pa.evals.runs.create({
  evalSet: evalSet.id, // an existing set from evals.sets.create(...)
  models: ["auto"],
  frontier: "benchmarked", // or "all", an explicit list of frontier ids, or null
  wait: true,
});
```

### Polling a run yourself

If you started a run with `wait=False`, await `runs.wait()` later, or poll `runs.retrieve()` on your own schedule. `wait()` accepts `poll_interval` (default 3.0s) and `timeout` (default 900s), and raises `ParetaError` if the run does not reach a terminal status in time.

**Python**

```python
run = await pa.evals.runs.create(eval_set=eval_set.id, models=["auto"])
print("queued:", run.id, run.status)
# ... do other work ...
final = await pa.evals.runs.wait(run.id, poll_interval=5.0, timeout=600.0)
print(final.status, final.is_terminal, final.cost)
```

**TypeScript**

`runs.wait(id, { pollInterval, timeout })` takes seconds (defaults 3 / 900) and throws `ParetaError` if the run does not finish in time. The run id is positional; the schedule is an options object.

```typescript
const run = await pa.evals.runs.create({ evalSet: evalSet.id, models: ["auto"] });
console.log("queued:", run.id, run.status);
// ... do other work ...
const final = await pa.evals.runs.wait(run.id, { pollInterval: 5, timeout: 600 });
console.log(final.status, final.isTerminal, final.cost);
```

## Bounding concurrency

`gather` launches everything at once. For large batches, cap the in-flight count with an `asyncio.Semaphore` so you do not trip rate limits (which surface as `RateLimitError`, 429; the client already retries those with backoff up to `max_retries`).

**Python**

```python
import asyncio
from pareta import AsyncPareta


async def main():
    sem = asyncio.Semaphore(5)  # at most 5 concurrent requests

    async with AsyncPareta.from_env() as pa:
        async def one(prompt: str) -> str:
            async with sem:
                completion = await pa.chat.completions.create(
                    model="auto",
                    messages=[{"role": "user", "content": prompt}],
                )
                return completion.choices[0].message.content

        prompts = [f"Extract field {i}." for i in range(100)]
        answers = await asyncio.gather(*(one(p) for p in prompts))
        print(len(answers), "done")


asyncio.run(main())
```

**TypeScript**

`Promise.all` launches everything at once too. There is no built-in semaphore, so cap concurrency by draining a shared work queue from a fixed pool of workers — at most `LIMIT` requests are in flight at any time.

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const LIMIT = 5; // at most 5 concurrent requests

const prompts = Array.from({ length: 100 }, (_, i) => `Extract field ${i}.`);

async function one(prompt: string): Promise<string | null> {
  const completion = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: prompt }],
  });
  return completion.choices[0].message.content;
}

const answers: (string | null)[] = new Array(prompts.length);
let next = 0;
async function worker() {
  while (next < prompts.length) {
    const i = next++;
    answers[i] = await one(prompts[i]);
  }
}
await Promise.all(Array.from({ length: LIMIT }, () => worker()));
console.log(answers.length, "done");
```

## Errors

The async client raises the exact same exception hierarchy as the sync client; the only difference is that errors surface out of an awaited call or an `async for`. Catch them the usual way.

**Python**

```python
from pareta import (
    AsyncPareta,
    InsufficientCreditsError,
    EndpointNotReadyError,
    RateLimitError,
    ParetaError,
)


async def safe_call(pa: AsyncPareta):
    try:
        return await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "hi"}],
        )
    except InsufficientCreditsError:
        print("org balance is empty; top up in the dashboard")
    except EndpointNotReadyError:
        print("a backend behind auto is briefly unavailable; retry in a moment")
    except RateLimitError:
        print("rate limited even after retries")
    except ParetaError as e:
        print("pareta error:", e)
```

**TypeScript**

Same exception hierarchy, surfaced out of an awaited call (or a `for await`). JavaScript has one `catch` clause, so branch on `instanceof` — most specific first, since the subclasses all extend `ParetaError`.

```typescript
import {
  Pareta,
  InsufficientCreditsError,
  EndpointNotReadyError,
  RateLimitError,
  ParetaError,
} from "pareta";

async function safeCall(pa: Pareta) {
  try {
    return await pa.chat.completions.create({
      model: "auto",
      messages: [{ role: "user", content: "hi" }],
    });
  } catch (e) {
    if (e instanceof InsufficientCreditsError) {
      console.log("org balance is empty; top up in the dashboard");
    } else if (e instanceof EndpointNotReadyError) {
      console.log("a backend behind auto is briefly unavailable; retry in a moment");
    } else if (e instanceof RateLimitError) {
      console.log("rate limited even after retries");
    } else if (e instanceof ParetaError) {
      console.log("pareta error:", e);
    } else {
      throw e;
    }
  }
}
```

Pre-flight validation (empty `model`/`messages`, empty `items`, an unparseable `frontier`) raises `ValueError`/`TypeError` when you `await` the call — the check runs at the top of the coroutine, before any network I/O (not when the coroutine object is first created). See [Errors](errors-and-retries.md) for the full mapping.

## Sync and async, side by side

| Concern | `Pareta` (sync) | `AsyncPareta` (async) |
|---|---|---|
| Build | `Pareta.from_env()` | `AsyncPareta.from_env()` |
| Cleanup | `pa.close()` / `with pa:` | `await pa.aclose()` / `async with pa:` |
| Request method | `pa.models.list()` | `await pa.models.list()` |
| Streaming chat | `for chunk in pa.chat.completions.create(stream=True)` | `stream = await pa...create(stream=True)` then `async for chunk in stream` |
| Wait on a run | `pa.evals.runs.wait(run_id)` | `await pa.evals.runs.wait(run_id)` |
| Auto metrics | `pa.auto.metrics()` | `await pa.auto.metrics()` |
| Concurrency | thread pool / one at a time | `asyncio.gather`, one event loop |

Same metering, same OpenAI-compatible inference, same hidden models and GPUs. Once you have the sync flow in [Quickstart](./quickstart.md), the async version is the same calls with `await` in front and `async for` over the streams.



---

<!-- guide/configuration.md -->

# Configuration

Every Pareta call goes through one client object. Configuration is just how you build that client: which API key it sends, which environment it points at, how patient it is on slow or flaky requests, and (optionally) what HTTP stack it rides on. This page covers all of it for both `Pareta` (sync) and `AsyncPareta` (async).

The short version: set `PARETA_API_KEY` and use `Pareta.from_env()`. Everything below is for when the defaults are not enough.

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()
print(pa.models.list())
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
console.log(await pa.models.list());
```

## The fast path: `from_env()`

`from_env()` reads two environment variables and builds the client for you:

- `PARETA_API_KEY` — your `pareta_sk_…` key (required)
- `PARETA_BASE_URL` — optional environment override (defaults to production)

```bash
export PARETA_API_KEY="pareta_sk_live_…"
```

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
```

`from_env()` forwards any extra keyword arguments straight to the constructor, so you can keep the key in the environment while overriding everything else in code:

**Python**

```python
pa = Pareta.from_env(max_retries=5, timeout=120.0)
```

**TypeScript**

```typescript
// timeout is milliseconds in TS (120 seconds → 120_000).
const pa = Pareta.fromEnv({ maxRetries: 5, timeout: 120_000 });
```

There is no separate async client in TypeScript — there is one `Pareta` class and every I/O method returns a `Promise` you `await`:

**Python**

```python
from pareta import AsyncPareta

pa = AsyncPareta.from_env()
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

// Same client; await the calls (e.g. await pa.models.list()).
const pa = Pareta.fromEnv();
```

Prefer `from_env()` over hardcoding keys. It keeps `pareta_sk_…` secrets out of source control and lets the same code run against staging or production by flipping one environment variable.

## Constructor parameters

Both clients take the same arguments:

**Python**

```python
from pareta import Pareta

pa = Pareta(
    api_key="pareta_sk_live_…",
    base_url="https://api.pareta.ai",
    timeout=60.0,
    max_retries=2,
    http_client=None,
)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = new Pareta({
  apiKey: "pareta_sk_live_…",
  baseURL: "https://api.pareta.ai",
  timeout: 60_000, // milliseconds
  maxRetries: 2,
  // fetch: customFetch, // bring your own fetch instead of http_client
});
```

| Parameter | Type | Default | What it does |
|-----------|------|---------|--------------|
| `api_key` | `str \| None` | `None` | Your `pareta_sk_…` key. Sent as a Bearer token. Required. |
| `base_url` | `str \| None` | `"https://api.pareta.ai"` | API root. Use the staging URL to point at the staging environment. |
| `timeout` | `httpx.Timeout \| float \| None` | `httpx.Timeout(60.0, connect=10.0)` | Per-request timeout. |
| `max_retries` | `int` | `2` | Automatic retries on transient failures. |
| `http_client` | `httpx.Client \| httpx.AsyncClient \| None` | `None` | Bring your own httpx client (proxies, custom transports, connection pools). |

`AsyncPareta` is identical except `http_client` takes an `httpx.AsyncClient`.

## `api_key`

The key is the one required piece of configuration. Pass it explicitly or via `PARETA_API_KEY`; the SDK sends it as `Authorization: Bearer <key>` on every request.

**Python**

```python
from pareta import Pareta

pa = Pareta(api_key="pareta_sk_live_…")
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = new Pareta({ apiKey: "pareta_sk_live_…" });
```

If the key is missing or empty (and the env var is unset when using `from_env()`), the constructor raises `ParetaError` before any network call:

**Python**

```python
from pareta import Pareta, ParetaError

try:
    pa = Pareta(api_key="")
except ParetaError as e:
    print(e)
    # missing API key. Pass api_key=… or set PARETA_API_KEY
    # (mint a pareta_sk_ key in the dashboard).
```

**TypeScript**

```typescript
import { Pareta, ParetaError } from "pareta";

try {
  const pa = new Pareta({ apiKey: "" });
} catch (e) {
  if (e instanceof ParetaError) {
    console.log(e.message);
    // missing API key. Pass apiKey: … or use Pareta.fromEnv() with PARETA_API_KEY
    // (mint a pareta_sk_ key in the dashboard).
  }
}
```

Mint keys in the dashboard. If the key is present but rejected by the server, you get a `401` as `AuthenticationError` on the first request, not at construction time. See [Errors](errors-and-retries.md) for the full exception hierarchy.

## `base_url` (production vs staging)

`base_url` selects the environment. It defaults to production and is normalized with a trailing-slash strip, so `https://api.pareta.ai/` and `https://api.pareta.ai` behave identically.

| Environment | `base_url` |
|-------------|------------|
| Production (default) | `https://api.pareta.ai` |
| Staging | `https://api-staging.pareta.ai` |

**Python**

```python
from pareta import Pareta

# Production — base_url omitted, defaults applied.
prod = Pareta(api_key="pareta_sk_live_…")

# Staging — pass it explicitly, or set PARETA_BASE_URL.
staging = Pareta(
    api_key="pareta_sk_test_…",
    base_url="https://api-staging.pareta.ai",
)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

// Production — baseURL omitted, defaults applied.
const prod = new Pareta({ apiKey: "pareta_sk_live_…" });

// Staging — pass it explicitly, or set PARETA_BASE_URL.
const staging = new Pareta({
  apiKey: "pareta_sk_test_…",
  baseURL: "https://api-staging.pareta.ai",
});
```

Via the environment, no code change needed:

```bash
export PARETA_API_KEY="pareta_sk_test_…"
export PARETA_BASE_URL="https://api-staging.pareta.ai"
```

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()   # now talks to staging
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv(); // now talks to staging
```

Keys are environment-scoped: a production key will not authenticate against staging and vice versa. Pair each `base_url` with a key minted for that environment.

## `timeout`

`timeout` caps how long a single request may take. The default is `httpx.Timeout(60.0, connect=10.0)`: up to 10 seconds to establish the connection and 60 seconds overall. A bare float sets the overall timeout for read, write, and connect alike.

**Python**

```python
import httpx
from pareta import Pareta

# Simple: one number for everything.
pa = Pareta(api_key="pareta_sk_live_…", timeout=120.0)

# Granular: long read budget for big generations, short connect budget.
pa = Pareta(
    api_key="pareta_sk_live_…",
    timeout=httpx.Timeout(120.0, connect=10.0),
)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

// One overall budget, in milliseconds (no separate connect budget in TS).
const pa = new Pareta({ apiKey: "pareta_sk_live_…", timeout: 120_000 });
```

When to raise it:

- **Long completions.** A 4096-token generation can run well past 60 seconds. Either raise `timeout` or stream the response so tokens arrive incrementally (see [Inference](inference.md)).
- **Long eval runs.** `evals.runs.create(..., wait=True)` does its own polling and has a separate `timeout` argument (default `900.0` seconds) that governs the wait loop, independent of the per-request HTTP timeout. See [Evals](evaluation.md).

A request that exceeds the timeout raises `APITimeoutError` (a subclass of `APIConnectionError`) after retries are exhausted.

## `max_retries`

The SDK automatically retries transient failures. The default is `2` (so up to 3 attempts total). Values below zero are clamped to `0`.

Retries fire only on these status codes:

```
408  Request Timeout
409  Conflict (transient lock/contention)
429  Too Many Requests
500  Internal Server Error
502  Bad Gateway
503  Service Unavailable
504  Gateway Timeout
```

Backoff is exponential with jitter, capped at 8 seconds: `min(0.5 * 2 ** attempt, 8.0)` plus a small random jitter. When the server sends a `Retry-After` header, the SDK honors it (capped at 30 seconds) instead of computing its own delay.

**Python**

```python
from pareta import Pareta

# More patient: handy for batch jobs against a busy environment.
pa = Pareta(api_key="pareta_sk_live_…", max_retries=5)

# Disable retries entirely: every failure surfaces immediately.
pa = Pareta(api_key="pareta_sk_live_…", max_retries=0)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

// More patient: handy for batch jobs against a busy environment.
const patient = new Pareta({ apiKey: "pareta_sk_live_…", maxRetries: 5 });

// Disable retries entirely: every failure surfaces immediately.
const failFast = new Pareta({ apiKey: "pareta_sk_live_…", maxRetries: 0 });
```

What is *not* retried:

- **4xx errors other than 408/409/429** — these are your request, not a transient blip. A `402 InsufficientCreditsError`, `401 AuthenticationError`, or `404 NotFoundError` raises on the first attempt.
- **Connection errors on initial connect** (DNS, TCP, TLS refusal) — raised after the retry budget for the handshake is spent.
- **Mid-stream drops.** Streaming calls (`chat.completions.create(stream=True)`) retry only the initial handshake. Once SSE bytes are flowing, a drop raises immediately, because a partial stream cannot be safely resumed.

A `409` is worth a note: it is in the retry set because some backends use it for transient lock contention. A stable `409` from Pareta simply exhausts the retries and then raises `ConflictError`, so you see the right error either way. See [Errors](errors-and-retries.md).

## `http_client` (bring your own httpx)

By default the client constructs its own httpx client, configured with your `timeout`, and closes it for you. Pass `http_client=` when you need control over the transport layer: an outbound proxy, a custom transport, mTLS, shared connection pools, or test doubles.

**Python**

```python
import httpx
from pareta import Pareta

# Route through a corporate proxy with a tuned connection pool.
my_client = httpx.Client(
    proxy="http://proxy.internal:8080",
    limits=httpx.Limits(max_connections=50, max_keepalive_connections=10),
    timeout=httpx.Timeout(120.0, connect=10.0),
)

pa = Pareta(api_key="pareta_sk_live_…", http_client=my_client)
```

The TS SDK has no owned HTTP client — it uses the global `fetch`. To control the transport (proxy, custom pool, mTLS, test doubles), inject your own `fetch` implementation via `fetch:`. In Node, a proxy/pool is configured on an undici `Agent` and threaded through a wrapper fetch:

**TypeScript**

```typescript
import { Pareta } from "pareta";
import { ProxyAgent } from "undici";

// Route through a corporate proxy with a tuned connection pool.
const agent = new ProxyAgent({
  uri: "http://proxy.internal:8080",
  connections: 50, // pool size
});

const myFetch: typeof fetch = (input, init) =>
  fetch(input, { ...init, dispatcher: agent } as RequestInit);

const pa = new Pareta({ apiKey: "pareta_sk_live_…", fetch: myFetch });
```

There is a single client, so there is no separate async variant to configure — the same injected `fetch` serves every awaited call.

**Ownership matters.** When you inject a client, you own its lifecycle. `pa.close()` (or `await pa.aclose()`) will *not* close a client you passed in. Close it yourself:

**Python**

```python
my_client.close()           # you opened it, you close it
```

**TypeScript**

```typescript
await agent.close(); // you opened it, you close it
```

The TS SDK owns no connection pool of its own, so there is nothing on the client to close — only your injected transport (if any). The Python context-manager forms below rely on the SDK-owned client.

One caveat: an injected client carries its own timeout configuration. The constructor's `timeout` argument is applied to the SDK-owned client only, so set the timeout on your own client when you bring one.

## Lifecycle and cleanup

Each client owns an HTTP connection pool. Release it when you are done.

### Sync

Use the context manager so cleanup happens automatically:

**Python**

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    completion = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Extract the parties."}],
    )
    print(completion.choices[0].message.content)
# HTTP client closed on exit
```

The TS client owns no connection pool, so there is no context manager and nothing to close — just construct it and `await` your calls:

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const completion = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Extract the parties." }],
});
console.log(completion.choices[0].message.content);
```

Or close it explicitly:

**Python**

```python
pa = Pareta.from_env()
try:
    pa.models.list()
finally:
    pa.close()
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

// No close() — the client holds no pool of its own.
const pa = Pareta.fromEnv();
await pa.models.list();
```

### Async

**Python**

```python
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        models = await pa.models.list()
        print(models)
    # HTTP client closed on exit
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

// One client; every call is already async. No async variant, no aclose().
const pa = Pareta.fromEnv();
const models = await pa.models.list();
console.log(models);
```

Or close it explicitly with `await pa.aclose()`.

Remember the ownership rule: if you passed `http_client=`, neither `close()` nor exiting the context manager touches it. Close your own client.

## Platform truths worth knowing

These hold no matter how you configure the client. They are why there is no GPU knob, no balance API, and no model catalog to wire up.

- **GPUs are hidden.** You configure a key, a URL, timeouts, and retries — never hardware. The serving stack behind `model="auto"` (GPUs, tensor-parallelism, quantization) is Pareta's job, resolved per request. There is no hardware parameter anywhere in the SDK.
- **There is one model id.** `models.list()` returns exactly one entry — `"auto"`. "Which model?" is the question Pareta answers for you, per request. Frontier vendor ids (`gpt-5.5`, …) appear only in eval and comparison contexts, as baselines. See [Core concepts](core-concepts.md).
- **Inference and evals are metered against your org balance.** A successful `chat.completions.create()` debits your balance once per request — however many internal model calls auto's plan makes, orchestration overhead is Pareta's cost, not yours. `evals.runs.create()` debits for the run's compute (auto and frontier alike). `run.cost` reports the billed total as a `Decimal` in dollars (floored to whole cents), and `run.cost_micro_usd` the raw micro-USD. When the balance hits zero, both paths raise `InsufficientCreditsError` (402). Top-up is browser-only — the SDK exposes neither balance nor payment methods, by design. See [Evals](evaluation.md).
- **Inference is OpenAI-compatible.** `base_url` plus your `pareta_sk_…` key is a drop-in OpenAI endpoint — point the `openai` SDK at the same `base_url` and call `model="auto"`. Pareta's SDK adds what `openai` cannot do: the task catalog, intent matching, evals, and auto's metrics. See [Inference](inference.md).

## Configuration cookbook

A few complete, runnable setups.

**Production, defaults, env-driven** — the recommended baseline:

**Python**

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    print(pa.models.list())
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
console.log(await pa.models.list());
```

**Staging, patient retries, long timeout** — for a batch job against a busy environment:

**Python**

```python
import httpx
from pareta import Pareta

pa = Pareta(
    api_key="pareta_sk_test_…",
    base_url="https://api-staging.pareta.ai",
    timeout=httpx.Timeout(180.0, connect=10.0),
    max_retries=5,
)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = new Pareta({
  apiKey: "pareta_sk_test_…",
  baseURL: "https://api-staging.pareta.ai",
  timeout: 180_000, // milliseconds
  maxRetries: 5,
});
```

**Fail fast** — no retries, surface every error on the first attempt (good for tests):

**Python**

```python
from pareta import Pareta

pa = Pareta(api_key="pareta_sk_test_…", max_retries=0)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = new Pareta({ apiKey: "pareta_sk_test_…", maxRetries: 0 });
```

**Async, custom transport** — own the httpx client, own the cleanup:

**Python**

```python
import asyncio
import httpx
from pareta import AsyncPareta

async def main():
    client = httpx.AsyncClient(proxy="http://proxy.internal:8080")
    pa = AsyncPareta.from_env(http_client=client)
    try:
        print(await pa.models.list())
    finally:
        await client.aclose()   # you opened it, you close it

asyncio.run(main())
```

**TypeScript**

```typescript
import { Pareta } from "pareta";
import { ProxyAgent } from "undici";

const agent = new ProxyAgent({ uri: "http://proxy.internal:8080" });
const fetchViaProxy: typeof fetch = (input, init) =>
  fetch(input, { ...init, dispatcher: agent } as RequestInit);

const pa = Pareta.fromEnv({ fetch: fetchViaProxy });
try {
  console.log(await pa.models.list());
} finally {
  await agent.close(); // you opened it, you close it
}
```

## See also

- [Inference](inference.md) — OpenAI-compatible chat completions, streaming, and metering.
- [Core concepts](core-concepts.md) — tasks, `model="auto"`, and how requests are planned and routed.
- [Evals](evaluation.md) — benchmark `"auto"` against frontier baselines on your own data, including `run.cost`.
- [Errors](errors-and-retries.md) — the full exception hierarchy and how retries interact with it.



---

<!-- guide/cli.md -->

# The `pareta` CLI

The `pareta` command is the SDK in your shell: call `model="auto"`, match a task, run an eval on your own data, and read auto's metrics — each as one command, rendered as a table or, with `--json`, as machine-readable output for scripts. It ships with the Python package (`pip install "pareta[cli]"`) and, once installed, works from any shell regardless of your project's language.

## Install

The CLI is an optional extra on the Python package — it adds `typer` + `rich`:

```bash
pip install "pareta[cli]"
```

Because the install puts a `pareta` console script on your PATH, an isolated install with [`pipx`](https://pipx.pypa.io) is often cleaner — it keeps the CLI and its dependencies out of your project's environment while still putting the command on your PATH:

```bash
pipx install "pareta[cli]"
```

Either way you get the same command. Confirm it:

```bash
pareta --version
```

## Authenticate

The CLI reads the same environment as the SDK — `PARETA_API_KEY` (required) and the optional `PARETA_BASE_URL`. Mint a `pareta_sk_` key in the [dashboard](https://pareta.ai) and export it:

```bash
export PARETA_API_KEY="pareta_sk_…"
```

There is no `login` command and no config file: auth *is* the environment, so the same export works in your shell, a Makefile, or CI. A missing or bad key prints a one-line error to stderr and exits non-zero (`2` for an auth/config problem you can fix locally, `1` for a genuine API error) — never a traceback.

## Output: tables or `--json`

Every command prints a human-readable table by default. Add the global `--json` (or `-j`) flag — **before** the subcommand — to get the raw JSON the SDK returns instead, for piping into `jq` or a script:

```bash
pareta --json models list | jq '.[].id'
```

Data goes to stdout and diagnostics to stderr, so piped output stays clean.

## Command tree

`pareta --help` (or `pareta <group> --help`) documents the whole tree. The groups mirror the SDK's resource namespaces.

### `chat` — one-shot inference

```bash
pareta chat "Summarize this contract clause: …"           # prompt as an argument
echo "Summarize this clause: …" | pareta chat             # or piped on stdin
pareta chat "Tell me a story" --stream                    # stream tokens as they arrive
```

Every chat goes to `model="auto"`: Pareta plans the request, routes it to benchmark-proven open specialists, verifies, and falls back to a frontier model when that's the right call — one request, one debit. Inference is metered.

### `tasks` — browse the catalog + match intent

```bash
pareta tasks match "pull the key fields out of these contracts"   # intent → task / capability / unsupported
pareta tasks list                                                 # every benchmarked task auto routes across
pareta tasks show contract-key-fields                             # one task's schema + default scorer
```

`match` resolves a free-text description of your dataset to the grading contract an eval scores it with (or a general lane, or no match — a statement about scoring, not serving). `--top-k` (default 5) controls how many candidate tasks it considers.

### `models` — the model catalog

```bash
pareta models list            # exactly one entry: "auto"
```

There is one model id. Everything behind it — planning, routing, verification, frontier fallback — is Pareta's job, per request.

### `evals` — benchmark on your own data

```bash
# Build a set on the fly from a JSONL file and benchmark "auto" against the frontier baselines:
pareta evals run --intent "extract the key fields from each contract" --file rows.jsonl \
  --models auto --frontier --wait

# Or run an existing eval set:
pareta evals run --eval-set es_abc --models auto --wait

# Preview the grading contract your rows will bind to (nothing is created):
pareta evals propose --file rows.jsonl --intent "extract the key fields from each contract"

# Manage eval sets (your data rows):
pareta evals sets create --intent "extract the key fields from each contract" --file rows.jsonl
pareta evals sets list
pareta evals sets show es_abc
pareta evals sets delete es_abc --yes
```

`--models` is required — pass `auto` to benchmark Pareta's routing itself. When you build a set from `--file`, `--intent` (one sentence on what the model should do with each item) is required — the binder resolves the grading contract from it and your data's shape, so `--task` is optional (pass it only to pin a specific contract). Running an existing set with `--eval-set` needs neither. `--frontier` adds the contract's benchmarked vendor models as baselines, which is the comparison that matters. Each item in `--file` is one JSON object per line; eval runs are metered against your org balance.

### `auto` — watch it and compare it

```bash
pareta auto metrics                                              # requests + success rate (30d), spend, projected savings vs frontier
pareta auto compare "Summarize this clause: …"                   # one prompt: auto vs a frontier vendor, side by side with both bills
pareta auto compare "Summarize this clause: …" --frontier claude-sonnet-4-6
```

`metrics` is read-only and free. `compare` is metered — it makes two real calls, one to `auto` and one to the vendor at the vendor's actual token cost (a failed vendor call bills $0). Allowed vendors: `gpt-5.5`, `gemini-3-5-flash`, `gemini-3-1-pro`, `claude-sonnet-4-6`.

### `rerank` + `embed` — the retrieval lanes

```bash
pareta rerank "governing law" "clause one…" "clause two…" --top-n 3   # docs as arguments
pareta rerank "governing law" --file docs.txt                         # or one document per line
pareta embed "what governs this contract?" --type query              # query-side vector
pareta embed --file passages.txt --out vectors.jsonl                  # document vectors → JSONL
```

`rerank` scores every document against the query (calibrated 0–1 scores, best first; `--top-n` only truncates the output) — metered per document scored. `embed` returns unit-normalized vectors; the table shows sizes only, so grab the vectors with `--out` (JSONL rows `{"index", "vector"}`) or `--json` — metered per input token.

### `audio` — speech in and out

```bash
pareta audio transcribe meeting.wav                         # speech-to-text (prints the transcript)
pareta audio speak "Hello from Pareta" --out hello.wav      # text-to-speech (writes an audio file)
```

Both are metered per minute of audio.

### `image` — text to image

```bash
pareta image "a red fox in the snow" --out fox.png                    # 1024x1024 default
pareta image "wide product banner" --size 2560x1440 --seed 7          # pinned seed
pareta image-edit fox.png "give the fox a red scarf" --out fox2.png   # instruction edit
```

`image` writes a PNG (default `image.png`), billed FLAT per image — every
size costs the same (the model renders at full 2K internally either way).
`image-edit` takes a reference image + a plain-language instruction (no
mask), keeps the reference's aspect ratio, and is billed FLAT per edit.

## Scripting

Because every command takes `--json` and exits non-zero on failure, the CLI composes into shell pipelines and CI. For example, check that a job is routable, benchmark `auto` on your own rows, then run the real call:

```bash
export PARETA_API_KEY="pareta_sk_…"

pareta --json tasks match "pull the key fields out of these contracts" | jq -r '.type'
pareta evals run --intent "extract the key fields from each contract" --file rows.jsonl --models auto --frontier --wait
pareta chat "What is the contract's effective date? …"
```

## Next steps

- [MCP server](mcp.md) — expose the same commands to an AI agent (Claude Desktop, Cursor) as tools.
- [Installation & authentication](installation.md) — the underlying SDK, the `pareta_sk_` key, and `from_env()`.
- [Core concepts](core-concepts.md) — tasks, `model="auto"`, and the metering behind every command.
- [Evaluating on your own data](evaluation.md) — the eval surface behind the `evals` group.



---

<!-- guide/mcp.md -->

# MCP server

`pareta-mcp` is a [Model Context Protocol](https://modelcontextprotocol.io) server that exposes Pareta to an AI agent (Claude Desktop, Cursor, …) as tools — so the agent can call `model="auto"`, match a task, run an eval on your data, and read auto's metrics on your behalf. It ships with the Python package (`pip install "pareta[mcp]"`) and speaks stdio, so any MCP-capable client can drive it regardless of your project's language.

## Install it in its own environment

Like any MCP server, `pareta-mcp` has its own dependency tree (the `mcp` runtime, which pulls in `starlette`). Install it **isolated** — not into an application or project virtualenv, where those dependencies can clash with, say, a FastAPI app, and where the console script may not land on your PATH.

The simplest path is [`uvx`](https://docs.astral.sh/uv/): it runs the server on demand in an ephemeral, isolated environment with nothing to install ahead of time. Point your MCP client's `command` at `uvx`:

```json
{
  "mcpServers": {
    "pareta": {
      "command": "uvx",
      "args": ["--from", "pareta[mcp]", "pareta-mcp"],
      "env": { "PARETA_API_KEY": "pareta_sk_…" }
    }
  }
}
```

Prefer a persistent install? [`pipx`](https://pipx.pypa.io) puts `pareta-mcp` on your PATH in a dedicated venv:

```bash
pipx install "pareta[mcp]"
```

…then point the client's `command` straight at the script:

```json
{
  "mcpServers": {
    "pareta": {
      "command": "pareta-mcp",
      "env": { "PARETA_API_KEY": "pareta_sk_…" }
    }
  }
}
```

Avoid a plain `pip install "pareta[mcp]"` into a shared/app environment — its `mcp`/`starlette` dependencies can clash with the app's FastAPI, and the console script may not land on your PATH.

### Claude Desktop

In Claude Desktop, open **Settings → Developer → Edit Config** to edit `claude_desktop_config.json`, add one of the JSON blocks above, and restart the app. The `pareta` tools then appear in the tool menu.

### Claude Code

[Claude Code](https://docs.claude.com/en/docs/claude-code) speaks MCP natively — add the server in one command. The flags go *before* the `--`; everything after it is the server's launch command:

```bash
claude mcp add pareta --scope user \
  --env PARETA_API_KEY=pareta_sk_… \
  -- uvx --from "pareta[mcp]" pareta-mcp
```

`--scope user` makes it available in every project; `--scope local` (the default) is this project only, and `--scope project` writes a shared `.mcp.json`. Verify with `claude mcp list` (or `/mcp` inside a session): `pareta` should show **connected** with its tools.

To commit it for a team without hardcoding the key, add a project-root `.mcp.json` and reference the key from the environment — Claude Code expands `${PARETA_API_KEY}` at startup:

```json
{
  "mcpServers": {
    "pareta": {
      "command": "uvx",
      "args": ["--from", "pareta[mcp]", "pareta-mcp"],
      "env": { "PARETA_API_KEY": "${PARETA_API_KEY}" }
    }
  }
}
```

### Codex

[Codex](https://developers.openai.com/codex) reads MCP servers from `~/.codex/config.toml`. Add a `[mcp_servers.pareta]` table with the same stdio command:

```toml
[mcp_servers.pareta]
command = "uvx"
args = ["--from", "pareta[mcp]", "pareta-mcp"]

[mcp_servers.pareta.env]
PARETA_API_KEY = "pareta_sk_…"
```

### Cursor and other MCP clients

Any MCP client takes the same stdio command. Use the JSON form from above (in Cursor, **Settings → MCP → Add**): point `command` at `uvx`, `args` at `["--from", "pareta[mcp]", "pareta-mcp"]`, and put `PARETA_API_KEY` in `env`.

## Authenticate

Set `PARETA_API_KEY` (a `pareta_sk_` key from the [dashboard](https://pareta.ai)) in the server's `env`, as shown above; `PARETA_BASE_URL` is optional and defaults to the production API. The key is read lazily on the first tool call, so the server starts even if it's unset — you get a clear error back when a tool runs, never a crashed server.

## Smoke-test it

Run the server directly to confirm it starts. It then waits for an MCP client to connect over stdio — there's no interactive output, so Ctrl-C to exit:

```bash
PARETA_API_KEY=pareta_sk_… uvx --from "pareta[mcp]" pareta-mcp
```

## The tools

The tools are grouped the same way as the SDK and CLI:

- **Inference** — `chat` (metered). The default `model="auto"` is the product: Pareta plans the request, routes it to benchmark-proven open specialists, verifies, and falls back to a frontier model when that's the right call.
- **Discovery** — `match_task`, `list_tasks`, `get_task`, `list_models`. Start with `match_task` to turn a plain-language goal into a task; `list_models` returns exactly one entry, `auto`.
- **Eval** — `run_eval`, `get_eval_run` (bring-your-own-data, metered). Pass `"auto"` among the candidate models to benchmark Pareta's routing itself against frontier baselines on your data.
- **Auto** — `auto_metrics` (read-only, free) and `compare_frontier` (metered: one prompt against a frontier vendor for a side-by-side with `chat`).
- **Audio** — `transcribe`, `speak` (metered per minute).
- **Retrieval** — `rerank` (metered per document scored), `embed` (metered per input token).
- **Images** — `generate_image` (metered flat per image), `edit_image` (metered flat per edit). Both work through disk paths you give them — image bytes never enter the agent's context.

A typical agent flow: `match_task("pull the key fields out of contracts")` → `run_eval(models=["auto"], task, items)` → `chat(prompt)`.

## Spending money is gated by your client's approval

Some tools cost money: `chat` / `run_eval` / `compare_frontier` / `transcribe` / `speak` / `rerank` / `embed` / `generate_image` / `edit_image` debit your org balance. The server deliberately adds **no** second confirmation layer — **your MCP client's per-tool-call approval is the guardrail.** Keep approval prompts on for the `pareta` server, and review the arguments (which task, which rows, which prompt) before approving a metered call. Tool errors — a missing key, an out-of-credit balance — come back as a clean `{"error": …}` message the agent can read, not a crash.

## Next steps

- [The `/pareta` skill](skill.md) — the slash-command alternative: a `SKILL.md` that drives the CLI (Claude Code & Codex), instead of tools-over-a-server.
- [The `pareta` CLI](cli.md) — the same commands from your shell.
- [Core concepts](core-concepts.md) — tasks, `model="auto"`, and the metering the agent is driving.



---

<!-- guide/skill.md -->

# The `/pareta` skill

A [Pareta skill](https://github.com/Pareta-AI/pareta/blob/main/skills/pareta/SKILL.md) teaches an AI coding agent to drive the `pareta` CLI as a slash command — run metered inference against `model="auto"`, match plain-language intent to a benchmarked task, and benchmark auto against frontier models on your own data. It's a single `SKILL.md` that works in both Claude Code and Codex, because they share the same skill format.

## Skill vs. MCP server

Two ways to put Pareta inside a coding agent — and you can use both:

- **The [MCP server](mcp.md)** gives the agent Pareta as structured **tools** (`chat`, `run_eval`, …) it calls directly. Best when you want first-class, auto-discovered tools.
- **This skill** is **instructions** — a `SKILL.md` the agent reads and follows, driving the `pareta` shell command. Best when you want a `/pareta` slash command and a guided workflow, and you've already installed the CLI.

## Prerequisite

The skill drives the CLI, so install it and set a key:

```bash
pipx install "pareta[cli]"            # or: pip install "pareta[cli]"
export PARETA_API_KEY="pareta_sk_…"   # mint one in the dashboard
```

## Install in Claude Code

Copy the skill into your personal skills directory (available from any project):

```bash
mkdir -p ~/.claude/skills/pareta
curl -fsSL https://raw.githubusercontent.com/Pareta-AI/pareta/main/skills/pareta/SKILL.md \
  -o ~/.claude/skills/pareta/SKILL.md
```

For a single repo, drop it at `.claude/skills/pareta/SKILL.md` instead. Then `/pareta` is available — and Claude Code also invokes it automatically when a request matches.

## Install in Codex

Codex uses the same skill format; only the directory differs:

```bash
mkdir -p ~/.codex/skills/pareta
curl -fsSL https://raw.githubusercontent.com/Pareta-AI/pareta/main/skills/pareta/SKILL.md \
  -o ~/.codex/skills/pareta/SKILL.md
```

For a single repo, check it in at `.codex/skills/pareta/SKILL.md`. Codex loads skills automatically when the task matches. (Codex's older custom prompts in `~/.codex/prompts/` are deprecated in favor of skills.)

## What it does

Once installed, ask in plain language — *"check whether Pareta can extract key fields from contracts, prove it on my rows, then run it."* The skill walks the agent through:

1. `pareta tasks match` — resolve the intent to a benchmarked task, a capability, or an honest `unsupported`.
2. `pareta chat` — run inference against `model="auto"` (Pareta plans, routes, verifies, and falls back to frontier when needed).
3. `pareta evals run --models auto --frontier` — benchmark auto against frontier baselines on your own JSONL data.
4. `pareta auto metrics` / `pareta auto compare` — watch spend + projected savings, and run one-prompt frontier side-by-sides.

It bakes in the guardrails: inference / eval / compare **spend your org balance**, so the agent confirms before spending.

## Next steps

- [MCP server](mcp.md) — the tools-over-a-server alternative (Claude Code, Codex, Claude Desktop, Cursor).
- [The `pareta` CLI](cli.md) — the command surface the skill drives.
- [Installation & authentication](installation.md) — install the CLI and mint a `pareta_sk_` key.



---

<!-- guide/agent-openclaw.md -->

# Connect OpenClaw to Pareta

Pareta's Agent Auto endpoint is OpenAI-compatible on the wire. Point OpenClaw — or any agent runtime that speaks the OpenAI chat completions API — at it, set the model to `auto`, and Pareta handles the rest: it routes each turn to the right model, escalates to a frontier model on the turns that need it, and bills one debit per turn.

You don't pick a model, a GPU, or a provider. `auto` is the whole product.

## The three values

Point your runtime's OpenAI-compatible provider at:

| Setting | Value |
| --- | --- |
| Base URL | `https://api.pareta.ai/agent/v1` |
| API key | your `pareta_sk_…` key (mint one in the dashboard) |
| Model | `auto` |

That's the entire integration. Everything else — your system prompt, the full message history, your tool schemas — flows through unchanged.

## What Pareta does per turn

- **One endpoint, one model string.** `model` is the literal string `"auto"`. Pareta reads the shape of each turn and routes it to the right fleet member — general reasoning, coding, or vision — behind that one string. Real model ids never reach you.
- **Full transcript and tools pass through.** Your system prompt, the whole conversation, and your OpenAI-shape `tools` go straight to the chosen model; `tool_calls` come back in the same shape. Nothing is dropped, summarized, or rewritten — the model's output *is* the answer.
- **A frontier floor.** When a turn comes back low-confidence, Pareta re-runs *that turn* on a frontier model (with your tools intact) and returns it as the answer. You get open-weights price on the turns the open fleet handles well, and frontier quality only on the turns that earn it.
- **One debit per turn.** A turn bills once no matter how Pareta routed or escalated it; a turn that errors bills nothing. The `X-Pareta-Billed` response header carries the amount in micro-USD.
- **131,072-token context**, streaming, and tool calling are all supported.

## Connect with the OpenAI SDK

Any OpenAI-compatible client connects the same way OpenClaw does under the hood — set the base URL and key, then call chat completions with `model="auto"`:

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://api.pareta.ai/agent/v1",
    api_key="pareta_sk_…",
)

resp = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "system", "content": "You are a coding agent."},
        {"role": "user", "content": "List the files in the repo, then summarize the README."},
    ],
    tools=[
        {"type": "function", "function": {
            "name": "run_shell",
            "description": "Run a shell command in the workspace.",
            "parameters": {"type": "object",
                           "properties": {"cmd": {"type": "string"}},
                           "required": ["cmd"]}}},
    ],
    tool_choice="auto",
)
print(resp.choices[0].message.tool_calls)
```

Feed the tool results back as `role: "tool"` messages on the next call, exactly as you would with any OpenAI-compatible model. The conversation carries; Pareta keeps routing each turn.

## Connect with raw HTTP

```bash
curl https://api.pareta.ai/agent/v1/chat/completions \
  -H "Authorization: Bearer pareta_sk_…" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "What is the weather in Paris? Use the tool."}],
    "tools": [{"type": "function", "function": {
      "name": "get_weather",
      "parameters": {"type": "object",
                     "properties": {"city": {"type": "string"}},
                     "required": ["city"]}}}],
    "tool_choice": "auto"
  }'
```

## In OpenClaw

OpenClaw configures models through OpenAI-compatible providers. Add Pareta as a provider with the base URL and key above, and set the model to `auto` for any role you want Pareta to serve — the primary model, a subagent, or a utility model. Because `auto` routes per turn, one Pareta provider entry covers coding, general reasoning, and vision without you wiring up separate models.

For a copy-pasteable `openclaw.json` provider block and the full wire contract — request fields, session pinning, streaming shape, the billing header, and error codes — see the [Agent API reference](../reference/agent-api.md).

## How this differs from `/v1`

Pareta's `/v1` chat endpoint is the one-shot **task** lane: send a request, get one synthesized answer. `/agent/v1` is the **conversation-and-tools** lane for agent loops — it passes your transcript and tools through verbatim, turn after turn, and pins a conversation to a consistent route. Same API key, same one-debit-per-turn billing, same `auto` model string. Use `/agent/v1` when a runtime like OpenClaw is driving a multi-turn tool loop.



---

<!-- examples/icd-coding.md -->

# Medical coding (ICD-10)

Turn a clinical discharge summary into ICD-10-CM codes with one chat call. The
summary goes to `model="auto"` as plain text; the prompt pins the output to a
strict JSON array of `{"code", "description"}` objects; you parse the array and
have structured codes.

Medical coding is a text-in, structured-text-out job, so it rides the standard
OpenAI-compatible chat surface — the one interface for every text workload.
`"auto"` is the only model id; the completion is metered against your org
balance, one debit per request regardless of internal routing.

## Setup

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();   // reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

See [installation](../guide/installation.md) for keys and environment.

## Code a discharge summary

The whole contract lives in the prompt: demand a bare JSON array of
`{"code", "description"}` objects and nothing else. Set `temperature=0` —
coding is a deterministic mapping from documentation to codes, and you want the
same summary to produce the same codes every run. `max_tokens` just needs
headroom for the array; 512 covers a typical inpatient stay.

**Python**

```python
PROMPT = (
    "Assign ICD-10-CM codes for the discharge summary below. Respond with "
    'ONLY a JSON array of {"code", "description"} objects — no prose, '
    "no markdown.\n\n" + DISCHARGE_SUMMARY
)

resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": PROMPT}],
    temperature=0,
    max_tokens=512,
)
raw = resp.choices[0].message.content or ""
```

**TypeScript**

```typescript
const PROMPT =
  "Assign ICD-10-CM codes for the discharge summary below. Respond with " +
  'ONLY a JSON array of {"code", "description"} objects — no prose, ' +
  "no markdown.\n\n" + DISCHARGE_SUMMARY;

const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: PROMPT }],
  temperature: 0,
  max_tokens: 512,
});
const raw = resp.choices[0].message.content ?? "";
```

Full runnable example: [python/icd-coding/icd_coding.py](https://github.com/Pareta-AI/examples/blob/main/python/icd-coding/icd_coding.py) · [typescript/icd-coding/icd-coding.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/icd-coding/icd-coding.ts)

## Parse the JSON robustly

The prompt demands bare JSON, but the single most common drift is a response
wrapped in a ` ```json ` fence. Strip a leading fence before parsing, then
validate that the payload really is an array — anything else should fail loudly
rather than flow downstream as half-parsed codes.

**Python**

```python
import json

def parse_codes(text: str) -> list[dict]:
    t = text.strip()
    if t.startswith("```"):                      # ```json fence — the common drift
        t = t.split("\n", 1)[1] if "\n" in t else ""
        t = t.rsplit("```", 1)[0]
    codes = json.loads(t)
    if not isinstance(codes, list):
        raise ValueError(f"expected a JSON array, got {type(codes).__name__}")
    return codes

for c in parse_codes(raw):
    print(f"{c.get('code', '?'):10} {c.get('description', '')}")
```

**TypeScript**

```typescript
function parseCodes(text: string): Array<{ code?: string; description?: string }> {
  let t = text.trim();
  if (t.startsWith("```")) {                     // ```json fence — the common drift
    t = t.includes("\n") ? t.slice(t.indexOf("\n") + 1) : "";
    const close = t.lastIndexOf("```");
    if (close !== -1) t = t.slice(0, close);
  }
  const codes = JSON.parse(t);
  if (!Array.isArray(codes)) throw new Error(`expected a JSON array, got ${typeof codes}`);
  return codes;
}

for (const c of parseCodes(raw)) {
  console.log(`${(c.code ?? "?").padEnd(10)} ${c.description ?? ""}`);
}
```

Full runnable example: [python/icd-coding/icd_coding.py](https://github.com/Pareta-AI/examples/blob/main/python/icd-coding/icd_coding.py) · [typescript/icd-coding/icd-coding.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/icd-coding/icd-coding.ts)

## Nothing to pick

There is no coding model in this example because there is nothing to name:
`"auto"` recognizes medical-coding traffic and routes it internally to the
right serving path, per request, server-side. Your code stays a plain chat call
with a JSON contract — the routing is Pareta's job, not a parameter.

Full runnable example: [python/icd-coding/icd_coding.py](https://github.com/Pareta-AI/examples/blob/main/python/icd-coding/icd_coding.py) · [typescript/icd-coding/icd-coding.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/icd-coding/icd-coding.ts)

## See also

- [Inference (OpenAI-compatible)](../guide/inference.md) — the full chat surface, streaming, extra params.
- [Chat reference](../reference/chat.md) — `chat.completions.create` request and response shapes.
- [Streaming chat](./streaming-chat.md) — token-by-token output for long generations.
- [Evaluating on your data](./evaluate-on-your-data.md) — benchmark `"auto"` on your own summaries, metered like inference.



---

<!-- examples/retrieval.md -->

# Retrieval: reranking and embeddings

Build search over a small support knowledge base three ways: rerank a
candidate list with `pa.rerank(...)`, semantic-search it with
`pa.embeddings(...)`, and compose the two into the classic two-stage RAG
stack — embeddings for recall, reranking for precision.

These are the Retrieval interfaces: `query + documents → ranked list` goes to
`rerank`, `text → vectors` goes to `embeddings`. They are dedicated routes
because ranked lists and vectors don't fit the chat message contract — not
because there is anything to navigate. You never name a serving model on
either; Pareta resolves the lane server-side, the same way `model="auto"`
does for chat. Rerank is metered per document scored, embeddings per input
token, both against your org balance.

## Setup

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();   // reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

See [installation](../guide/installation.md) if the package isn't set up yet.

## Reranking

`rerank(query, documents)` scores every document against the query and returns
`(index, relevance_score)` rows, most relevant first. The scores are
**calibrated** — each is an independent P(relevant) in (0, 1), not a softmax
over the batch — so a fixed cutoff like `>= 0.5` is a real keep/drop filter
that means the same thing across calls and corpora. `top_n` truncates the
response to the best N, but all documents are still scored (and metered).

**Python**

```python
KB = [
    "Refunds: annual plans are refundable in full within 30 days of purchase...",
    "Failed payments: a declined charge is retried after 24 hours...",
    "Seat changes: adding a seat mid-cycle bills the prorated difference...",
    # ...8 help-center passages in the full example
]

ranked = pa.rerank("Can I get my money back if I cancel my annual plan?", KB)
print(ranked.pairs)                        # documents scored — the metered unit
for r in ranked.results:                   # most relevant first
    print(f"{r.relevance_score:.3f}  {KB[r.index][:60]}")

# calibrated scores: threshold to filter, not just sort
keep = [KB[r.index] for r in ranked.results if r.relevance_score >= 0.5]

# top_n truncates the response; top_documents maps indices back to your texts
top3 = pa.rerank("Can I get my money back if I cancel my annual plan?", KB, top_n=3)
print(top3.top_documents(KB))              # the winning texts, best first
```

**TypeScript**

```typescript
const ranked = await pa.rerank("Can I get my money back if I cancel my annual plan?", KB);
console.log(ranked.pairs);                 // documents scored — the metered unit
for (const r of ranked.results) {          // most relevant first
  console.log(`${r.relevanceScore.toFixed(3)}  ${KB[r.index].slice(0, 60)}`);
}

// calibrated scores: threshold to filter, not just sort
const keep = ranked.results.filter((r) => r.relevanceScore >= 0.5).map((r) => KB[r.index]);

// topN truncates the response; topDocuments maps indices back to your texts
const top3 = await pa.rerank("Can I get my money back if I cancel my annual plan?", KB, { topN: 3 });
console.log(top3.topDocuments(KB));        // the winning texts, best first
```

Full runnable example: [python/retrieval/rerank.py](https://github.com/Pareta-AI/examples/blob/main/python/retrieval/rerank.py) · [typescript/retrieval/rerank.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/retrieval/rerank.ts)

## Embeddings (semantic search)

Embedding is asymmetric: index your passages raw (the default), and embed the
incoming question with `input_type="query"` so it gets the retrieval-query
treatment. The vectors come back unit-normalized, so cosine similarity is a
plain dot product — a one-line function, no vector library needed for a corpus
this size.

**Python**

```python
def dot(a, b):                             # unit vectors: this IS cosine
    return sum(x * y for x, y in zip(a, b))

# index side: embed all passages in one order-preserving call, raw
index = pa.embeddings(KB)
doc_vecs = index.vectors
print(index.prompt_tokens)                 # tokens embedded — the metered unit

# search side: the query embeds asymmetrically from the passages
q = pa.embeddings("can I get a refund on my yearly subscription?",
                  input_type="query").vectors[0]

top_k = sorted(range(len(KB)), key=lambda i: dot(q, doc_vecs[i]), reverse=True)[:3]
for i in top_k:
    print(f"{dot(q, doc_vecs[i]):.3f}  {KB[i][:60]}")
```

**TypeScript**

```typescript
const dot = (a: number[], b: number[]) => a.reduce((s, v, i) => s + v * b[i], 0);

// index side: embed all passages in one order-preserving call, raw
const index = await pa.embeddings(KB);
const docVecs = index.vectors;
console.log(index.promptTokens);           // tokens embedded — the metered unit

// search side: the query embeds asymmetrically from the passages
const q = (await pa.embeddings("can I get a refund on my yearly subscription?",
                               { inputType: "query" })).vectors[0];

const topK = KB.map((_, i) => i)
  .sort((a, b) => dot(q, docVecs[b]) - dot(q, docVecs[a]))
  .slice(0, 3);
for (const i of topK) {
  console.log(`${dot(q, docVecs[i]).toFixed(3)}  ${KB[i].slice(0, 60)}`);
}
```

Full runnable example: [python/retrieval/semantic_search.py](https://github.com/Pareta-AI/examples/blob/main/python/retrieval/semantic_search.py) · [typescript/retrieval/semantic-search.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/retrieval/semantic-search.ts)

## The two-stage RAG stack

The two lanes have complementary jobs. Embeddings are the **recall** stage: a
cheap, per-token wide net that scans the whole corpus and won't miss a
relevant passage phrased nothing like the question. The reranker is the
**precision** stage: a per-document scorer that reads each (query, candidate)
pair closely and is far better at deciding what actually answers the question
— too expensive for the whole corpus, exactly right for the shortlist. So:
embed-retrieve a wide top-K, rerank only those, keep the calibrated winners.

**Python**

```python
# stage 1 — recall: cosine top-5 over the whole corpus
doc_vecs = pa.embeddings(KB).vectors
q = pa.embeddings(QUERY, input_type="query").vectors[0]
candidates = sorted(range(len(KB)), key=lambda i: dot(q, doc_vecs[i]),
                    reverse=True)[:5]

# stage 2 — precision: rerank only the 5 candidates (5 metered, not 8)
pool = [KB[i] for i in candidates]
ranked = pa.rerank(QUERY, pool, top_n=2)

for r in ranked.results:                   # r.index points into pool;
    print(f"{r.relevance_score:.3f}  {pool[r.index]}")  # candidates[r.index] recovers the KB index
```

**TypeScript**

```typescript
// stage 1 — recall: cosine top-5 over the whole corpus
const docVecs = (await pa.embeddings(KB)).vectors;
const q = (await pa.embeddings(QUERY, { inputType: "query" })).vectors[0];
const candidates = KB.map((_, i) => i)
  .sort((a, b) => dot(q, docVecs[b]) - dot(q, docVecs[a]))
  .slice(0, 5);

// stage 2 — precision: rerank only the 5 candidates (5 metered, not 8)
const pool = candidates.map((i) => KB[i]);
const ranked = await pa.rerank(QUERY, pool, { topN: 2 });

for (const r of ranked.results) {          // r.index points into pool;
  console.log(`${r.relevanceScore.toFixed(3)}  ${pool[r.index]}`);  // candidates[r.index] recovers the KB index
}
```

Full runnable example: [python/retrieval/rag_search.py](https://github.com/Pareta-AI/examples/blob/main/python/retrieval/rag_search.py) · [typescript/retrieval/rag-search.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/retrieval/rag-search.ts)

## See also

- [rerank reference](../reference/rerank.md) — the full `Rerank` object, limits, and calibration notes.
- [embeddings reference](../reference/embeddings.md) — the `Embeddings` object and input limits.
- [Inference](../guide/inference.md) — `model="auto"` chat, the lane the retrieved context usually feeds.
- Prove it on your own data: [evaluate on your data](./evaluate-on-your-data.md) benchmarks both retrieval stages against your own graded relevance.



---

<!-- examples/extraction.md -->

# Extraction: documents and contracts

Pull structured fields out of documents — a scanned invoice, a PDF, a contract
— and get JSON back. You'll extract vendor/total/line-items from a real invoice
image, then key legal fields from a contract's text.

Extraction is a chat job: image or text in, JSON out, so it all goes through
`chat.completions.create(model="auto")`, OpenAI-compatible on the wire. One
interface, whatever the document looks like; each request is one debit against
the org balance regardless of internal routing.

## Setup

Install the SDK ([installation guide](../guide/installation.md)) and set
`PARETA_API_KEY`.

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();   // reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

## Visual documents (images and PDFs)

A visual document goes in as OpenAI content parts: one `text` part stating the
fields you want, one `image_url` part carrying the pixels as a base64 data URI.
Pin `temperature=0` — extraction wants determinism, not creativity — and ask for
JSON only, so the response parses without ceremony. This example downloads a
real invoice from the public examples dataset and extracts five fields from it.

**Python**

```python
import base64, json, urllib.request

url = ("https://raw.githubusercontent.com/Pareta-AI/example-datasets"
       "/main/invoice-extraction/documents/0.jpg")
img_b64 = base64.b64encode(urllib.request.urlopen(url).read()).decode()

resp = pa.chat.completions.create(
    model="auto",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": 'Extract {"vendor", "invoice_no", "date", '
                                     '"total", "line_items"} as JSON. Return ONLY the JSON object.'},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
        ],
    }],
    temperature=0,
    max_tokens=1024,
)
fields = json.loads(resp.choices[0].message.content)
```

**TypeScript**

```typescript
const url = "https://raw.githubusercontent.com/Pareta-AI/example-datasets" +
            "/main/invoice-extraction/documents/0.jpg";
const imgB64 = Buffer.from(await (await fetch(url)).arrayBuffer()).toString("base64");

const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{
    role: "user",
    content: [
      { type: "text", text: 'Extract {"vendor", "invoice_no", "date", ' +
                            '"total", "line_items"} as JSON. Return ONLY the JSON object.' },
      { type: "image_url", image_url: { url: `data:image/jpeg;base64,${imgB64}` } },
    ],
  }],
  temperature: 0,
  max_tokens: 1024,
});
const fields = JSON.parse(resp.choices[0].message.content ?? "");
```

Expected output for that document:

```json
{
  "vendor": "Bradley-Andrade",
  "invoice_no": "97159829",
  "date": "2015-09-18",
  "total": 978.12,
  "line_items": [ ... ]
}
```

### PDFs

You can hand a PDF into the same `image_url` slot. When a base64 data URI
declares `application/pdf` — or its decoded bytes start with the `%PDF` magic,
so a wrong or missing MIME type is still caught — the platform rasterizes it
server-side before inference: the PDF block is replaced with one PNG image
block per rendered page, up to the **first 8 pages** (sibling keys like
`detail` are preserved on each page). For longer documents, split the PDF or
send pre-rasterized page images. A PDF that cannot be rendered returns a 400
with a clear message rather than forwarding raw bytes a vision model would
reject. Plain `http(s)` image
URLs and URL-encoded (non-base64) data URIs pass through untouched.

**Python**

```python
pdf_b64 = base64.b64encode(open("contract.pdf", "rb").read()).decode()

{"type": "image_url", "image_url": {"url": f"data:application/pdf;base64,{pdf_b64}"}}
```

**TypeScript**

```typescript
const pdfB64 = (await readFile("contract.pdf")).toString("base64");

({ type: "image_url", image_url: { url: `data:application/pdf;base64,${pdfB64}` } });
```

Full runnable example: [python/extraction/visual_document.py](https://github.com/Pareta-AI/examples/blob/main/python/extraction/visual_document.py) · [typescript/extraction/visual-document.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/extraction/visual-document.ts)

## Contract fields (text)

When the document is already text, there is nothing special to do: the contract
goes in as ordinary string content on the same chat surface. Name the fields,
give the model an explicit out (`null`) for anything the text does not state —
otherwise it will guess — and keep `temperature=0`. Sample contract text from
CUAD (Hendrycks et al.), CC-BY-4.0.

**Python**

```python
import json
from pathlib import Path

contract_text = Path("data/sample-contract.txt").read_text()

prompt = ('Extract {"parties", "agreement_date", "effective_date", "governing_law"} '
          "as JSON. parties is a list of legal entity names; dates are YYYY-MM-DD; "
          "use null for anything the text does not state. Return ONLY the JSON object.")

resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": f"{prompt}\n\n---\n\n{contract_text}"}],
    temperature=0,
    max_tokens=512,
)
fields = json.loads(resp.choices[0].message.content)
print(fields["parties"], fields["governing_law"])
```

**TypeScript**

```typescript
import { readFile } from "node:fs/promises";

const contractText = await readFile("data/sample-contract.txt", "utf8");

const prompt = 'Extract {"parties", "agreement_date", "effective_date", "governing_law"} ' +
               "as JSON. parties is a list of legal entity names; dates are YYYY-MM-DD; " +
               "use null for anything the text does not state. Return ONLY the JSON object.";

const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: `${prompt}\n\n---\n\n${contractText}` }],
  temperature: 0,
  max_tokens: 512,
});
const fields = JSON.parse(resp.choices[0].message.content ?? "");
console.log(fields.parties, fields.governing_law);
```

Full runnable example: [python/extraction/contract_fields.py](https://github.com/Pareta-AI/examples/blob/main/python/extraction/contract_fields.py) · [typescript/extraction/contract-fields.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/extraction/contract-fields.ts)

## See also

- [Inference (OpenAI-compatible)](../guide/inference.md) — the full chat surface: streaming, extra params, error handling.
- [Chat reference](../reference/chat.md) — `chat.completions.create` request and response shapes.
- [Document extraction end-to-end](./document-extraction.md) — the full document workflow, from your own files to production.
- Prove it on your own data: [evaluate on your data](./evaluate-on-your-data.md) — benchmark `"auto"` against frontier baselines on your own documents before you commit.



---

<!-- examples/text-classification.md -->

# Text classification

Turn `model="auto"` into a production text classifier: a closed label set in the system prompt, a few labeled examples, `temperature=0`, and a one-word answer your code can branch on. This page builds two of them — a banking-support intent classifier and a hate-speech moderation gate that routes violations to human review.

Classification is a chat-shaped job, so it goes to the one chat interface: `pa.chat.completions.create(model="auto", ...)`, OpenAI-compatible on the wire. There is nothing to pick and nothing to deploy — auto routes every request server-side, and each call is one metered debit against your org balance regardless of internal routing.

## Setup

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();   // reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

See [installation](../guide/installation.md) for getting the SDK and key in place.

## Intent classification

A closed label set is the whole contract: the system prompt names every allowed label and demands the label alone, `temperature=0` makes the answer repeatable, and a membership check after the call guarantees no stray token ever escapes the set. The few-shot pairs are the part worth your attention — on pattern tasks like intent routing, the examples in the prompt move accuracy far more than any sampling knob. When the classifier keeps confusing two intents, add a pair that shows the right answer; that is the tuning loop.

**Python**

```python
LABELS = ("card_arrival", "card_not_working", "lost_or_stolen_card", "transfer_failed",
          "balance_inquiry", "exchange_rate", "top_up_failed", "other")

SYSTEM = (
    "You classify banking-support messages into exactly one intent label: "
    + ", ".join(LABELS)
    + ". Reply with the label only — lowercase, no punctuation, no explanation. "
    "If no label fits, reply: other."
)

FEW_SHOT = (  # the lever on pattern tasks — swap pairs to steer the classifier
    ("My new card was supposed to arrive two weeks ago and it still hasn't", "card_arrival"),
    ("The shop terminal declined my card even though my account has money", "card_not_working"),
    ("I made a transfer to my landlord yesterday and it bounced back", "transfer_failed"),
)

def classify(text: str) -> str:
    messages = [{"role": "system", "content": SYSTEM}]
    for user, label in FEW_SHOT:
        messages.append({"role": "user", "content": user})
        messages.append({"role": "assistant", "content": label})
    messages.append({"role": "user", "content": text})

    resp = pa.chat.completions.create(
        model="auto", messages=messages, temperature=0, max_tokens=8,
    )
    label = (resp.choices[0].message.content or "").strip().lower()
    return label if label in LABELS else "other"   # closed set, enforced
```

**TypeScript**

```typescript
const LABELS = ["card_arrival", "card_not_working", "lost_or_stolen_card", "transfer_failed",
  "balance_inquiry", "exchange_rate", "top_up_failed", "other"] as const;
type Label = (typeof LABELS)[number];

const SYSTEM =
  "You classify banking-support messages into exactly one intent label: " +
  LABELS.join(", ") +
  ". Reply with the label only — lowercase, no punctuation, no explanation. " +
  "If no label fits, reply: other.";

const FEW_SHOT: Array<[string, Label]> = [   // the lever on pattern tasks
  ["My new card was supposed to arrive two weeks ago and it still hasn't", "card_arrival"],
  ["The shop terminal declined my card even though my account has money", "card_not_working"],
  ["I made a transfer to my landlord yesterday and it bounced back", "transfer_failed"],
];

async function classify(text: string): Promise<Label> {
  const messages = [{ role: "system", content: SYSTEM }];
  for (const [user, label] of FEW_SHOT) {
    messages.push({ role: "user", content: user }, { role: "assistant", content: label });
  }
  messages.push({ role: "user", content: text });

  const resp = await pa.chat.completions.create({
    model: "auto", messages, temperature: 0, max_tokens: 8,
  });
  const label = (resp.choices[0].message.content ?? "").trim().toLowerCase();
  return (LABELS as readonly string[]).includes(label) ? (label as Label) : "other";
}
```

`max_tokens=8` caps each answer at the label itself, and one call classifies one utterance — a batch is just a loop. The fallback to `other` matters more than it looks: your downstream `switch` never sees an unexpected string, no matter what the model emits.

Full runnable example: [python/classification/intent.py](https://github.com/Pareta-AI/examples/blob/main/python/classification/intent.py) · [typescript/classification/intent.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/classification/intent.ts)

## Content moderation (hate speech)

Moderation is the same recipe with three labels — `hate`, `offensive`, `neither` — and one extra obligation: acting on the verdict. The prompt carries a one-line definition per label because the hate/offensive boundary (group-directed vs. individual-directed hostility) is exactly what untrained judgment gets wrong, and the two classes usually get different handling downstream. The strict one-word output is what makes the branch after the call safe to write.

**Python**

```python
LABELS = ("hate", "offensive", "neither")

SYSTEM = (
    "You are a content-moderation classifier. Label the text with exactly one of:\n"
    "hate — demeans or attacks a group of people based on a group identity\n"
    "offensive — insulting, hostile, or demeaning toward an individual, but not group-based\n"
    "neither — none of the above\n"
    "Reply with the label only — one lowercase word, no punctuation, no explanation."
)

def moderate(text: str) -> str:
    resp = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user", "content": text}],
        temperature=0,
        max_tokens=4,
    )
    label = (resp.choices[0].message.content or "").strip().lower()
    return label if label in LABELS else "offensive"   # fail closed → human review

review_queue = []
for text in incoming_texts:
    label = moderate(text)
    if label != "neither":
        review_queue.append((label, text))   # violations go to a human
```

**TypeScript**

```typescript
const LABELS = ["hate", "offensive", "neither"] as const;
type Label = (typeof LABELS)[number];

const SYSTEM = [
  "You are a content-moderation classifier. Label the text with exactly one of:",
  "hate — demeans or attacks a group of people based on a group identity",
  "offensive — insulting, hostile, or demeaning toward an individual, but not group-based",
  "neither — none of the above",
  "Reply with the label only — one lowercase word, no punctuation, no explanation.",
].join("\n");

async function moderate(text: string): Promise<Label> {
  const resp = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "system", content: SYSTEM }, { role: "user", content: text }],
    temperature: 0,
    max_tokens: 4,
  });
  const label = (resp.choices[0].message.content ?? "").trim().toLowerCase();
  return (LABELS as readonly string[]).includes(label) ? (label as Label) : "offensive";
}

const reviewQueue: Array<[Label, string]> = [];
for (const text of incomingTexts) {
  const label = await moderate(text);
  if (label !== "neither") reviewQueue.push([label, text]);   // violations go to a human
}
```

Note the failure direction: an answer outside the label set is coerced to `offensive`, not `neither`, so anything the classifier fumbles still reaches human eyes. Fail-open moderation is the one bug this recipe cannot afford.

Full runnable example: [python/classification/moderation.py](https://github.com/Pareta-AI/examples/blob/main/python/classification/moderation.py) · [typescript/classification/moderation.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/classification/moderation.ts)

## See also

- [Inference (OpenAI-compatible)](../guide/inference.md) — the full chat surface behind `model="auto"`.
- [Chat reference](../reference/chat.md) — request params and response schema.
- [Streaming chat completions](./streaming-chat.md) — token-by-token output for longer generations.
- Prove it on your own data: [run an eval](./evaluate-on-your-data.md) that benchmarks `"auto"` against frontier baselines on your own labeled texts.



---

<!-- examples/summarization.md -->

# Summarization

Turn a raw meeting transcript into a three-sentence executive summary plus an owner — item — due action-items list, with one call to `model="auto"`. The output format lives entirely in the prompt, so the same call summarizes support threads, incident timelines, or research notes by swapping the instructions.

Summarization is a chat/text job, so it goes to the one chat interface — `pa.chat.completions.create(model="auto", ...)`, OpenAI-compatible on the wire. There is no model to pick and nothing to deploy: `"auto"` routes every request server-side, and a successful completion is one debit against your org balance regardless of how it routes internally.

## Setup

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();   // reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

See [installation](../guide/installation.md) for getting the SDK and key in place.

## Executive summary + action items

Everything that shapes the output is prompt: a system message that pins the summarizer to the facts, and an instruction block that spells out the exact format — three sentences, then `- owner — item — due date` bullets, with a final bullet for anything explicitly deferred. `temperature=0.2` keeps repeated runs of the same transcript close to each other, and `max_tokens=512` is generous headroom for a one-page meeting.

**Python**

```python
from pathlib import Path

transcript = Path("data/meeting-notes.txt").read_text(encoding="utf-8")

completion = pa.chat.completions.create(
    model="auto",
    messages=[
        {"role": "system", "content": "You summarize meeting transcripts for executives. "
                                      "Be factual; never invent owners, dates, or decisions."},
        {"role": "user", "content": INSTRUCTIONS + "\n" + transcript},
    ],
    temperature=0.2,
    max_tokens=512,
)
print(completion.choices[0].message.content)
print(completion.usage.total_tokens)
```

**TypeScript**

```typescript
import { readFile } from "node:fs/promises";

const transcript = await readFile("data/meeting-notes.txt", "utf8");

const completion = await pa.chat.completions.create({
  model: "auto",
  messages: [
    { role: "system", content: "You summarize meeting transcripts for executives. " +
                               "Be factual; never invent owners, dates, or decisions." },
    { role: "user", content: INSTRUCTIONS + "\n" + transcript },
  ],
  temperature: 0.2,
  max_tokens: 512,
});
console.log(completion.choices[0].message.content);
console.log(completion.usage.totalTokens);
```

A `finish_reason` of `"length"` on the choice means the summary hit `max_tokens` before it finished — raise the cap rather than trusting a truncated action-items list.

Full runnable example: [python/summarization/summarize.py](https://github.com/Pareta-AI/examples/blob/main/python/summarization/summarize.py) · [typescript/summarization/summarize.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/summarization/summarize.ts)

## Stream the summary

For anything a person reads as it generates — a summary panel in your app, a CLI — pass `stream=True` and the same call returns an iterator of chunks instead of one completion. Print `chunk.choices[0].delta.content` as it arrives; the delta is `None` on bookkeeping chunks, so keep the guard. Everything else about the request, including the single debit, is unchanged.

**Python**

```python
stream = pa.chat.completions.create(
    model="auto",
    messages=messages,        # same system + transcript messages as above
    stream=True,
    temperature=0.2,
    max_tokens=512,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()
```

**TypeScript**

```typescript
const stream = pa.chat.completions.create({
  model: "auto",
  messages,                   // same system + transcript messages as above
  stream: true,
  temperature: 0.2,
  max_tokens: 512,
});
for await (const chunk of stream) {
  const delta = chunk.choices[0].delta.content;
  if (delta) {
    process.stdout.write(delta);
  }
}
console.log();
```

Chunk anatomy, accumulation, async streaming, and failure behavior are covered in [streaming chat completions](./streaming-chat.md).

Full runnable example: [python/summarization/summarize.py](https://github.com/Pareta-AI/examples/blob/main/python/summarization/summarize.py) · [typescript/summarization/summarize.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/summarization/summarize.ts)

## See also

- [Inference](../guide/inference.md) — the full chat surface: parameters, errors, and the OpenAI-compatible wire format.
- [Streaming chat completions](./streaming-chat.md) — chunks, accumulation, and async streaming in depth.
- [Error handling](../guide/errors-and-retries.md) — `InsufficientCreditsError` (402) and the rest of the exception hierarchy.
- Prove it on your own data: [evaluate summaries from your real transcripts](./evaluate-on-your-data.md), metered against the same org balance.



---

<!-- examples/text-to-speech.md -->

# Text to speech

Turn a line of text into spoken audio and write it to a `.wav` file — one call
in, one file out. You'll synthesize a short customer notification with
`pa.audio.speech(...)`, save the returned audio, and read back the format,
sample rate, and the duration that was metered.

Speech is text in, audio bytes out — a data shape the chat message contract
can't carry — so it has its own route, `POST /v1/audio/speech`, instead of
`chat.completions`. That is the only reason the route exists: there is no
voice model to name and nothing to deploy. You hand Pareta the text; everything
behind the call is resolved server-side, exactly as `model="auto"` does for chat.

## Setup

Install the SDK and set `PARETA_API_KEY` (see [installation](../guide/installation.md)).

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();   // reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

## Synthesize a notification

`speech(text)` returns a typed `Speech` whose `.audio` is the synthesized audio
already base64-decoded to raw bytes; `.save(path)` writes them to disk. The
response also carries the container format, the sample rate, and `duration_s` —
the length of the *output* audio, which is the metered unit (billed per minute
of output, not per token).

**Python**

```python
speech = pa.audio.speech(
    "Your appointment is confirmed for Thursday at two PM. "
    "Reply R to reschedule, or call us any time."
)
speech.save("welcome.wav")

print(speech.format)        # container/codec, e.g. "wav"
print(speech.sample_rate)   # Hz
print(speech.duration_s)    # output length in seconds — the metered unit
```

**TypeScript**

```typescript
const speech = await pa.audio.speech(
  "Your appointment is confirmed for Thursday at two PM. " +
    "Reply R to reschedule, or call us any time.",
);
await speech.save("welcome.wav");   // Node only — lazy node:fs under the hood

console.log(speech.format);        // container/codec, e.g. "wav"
console.log(speech.sampleRate);    // Hz
console.log(speech.durationS);     // output length in seconds — the metered unit
```

Empty or whitespace-only text raises locally (`ValueError` in Python, a
`ParetaError` in TypeScript) before any request goes out.

Full runnable example: [python/tts/speak.py](https://github.com/Pareta-AI/examples/blob/main/python/tts/speak.py) · [typescript/tts/speak.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/tts/speak.ts)

## Bytes, voice, and metering

You don't have to touch the filesystem: `.audio` is the decoded bytes, ready
for a response body or an object store, and `.save()` returns the same `Speech`
so writing a file and keeping the bytes chains into one expression. `voice=` is
the one optional knob — a voice id; omit it for the default voice. The call
debits your org balance per minute of output audio; a zero balance raises
`InsufficientCreditsError` (402) and no audio is generated.

**Python**

```python
from pareta import InsufficientCreditsError

try:
    audio_bytes = pa.audio.speech(
        "Your order has shipped and is on its way.",
    ).save("shipped.wav").audio      # write the file and keep the bytes

    print(len(audio_bytes), "bytes")
except InsufficientCreditsError:
    print("Org out of credit — top up in the dashboard, then re-run.")
```

**TypeScript**

```typescript
import { InsufficientCreditsError } from "pareta";

try {
  const s = await pa.audio.speech("Your order has shipped and is on its way.");
  await s.save("shipped.wav");
  const audioBytes = s.audio;       // Uint8Array — the same decoded bytes

  console.log(audioBytes.length, "bytes");
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    console.log("Org out of credit — top up in the dashboard, then re-run.");
  } else {
    throw e;
  }
}
```

Full runnable example: [python/tts/speak.py](https://github.com/Pareta-AI/examples/blob/main/python/tts/speak.py) · [typescript/tts/speak.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/tts/speak.ts)

## See also

- [audio reference](../reference/audio.md) — the full `pa.audio` surface: `speech` (TTS) and `transcriptions` (ASR), response objects, async twins.
- [Errors and retries](../guide/errors-and-retries.md) — `InsufficientCreditsError` and the full exception hierarchy.
- [Core concepts](../guide/core-concepts.md) — why speech has its own route while everything chat-shaped goes through `model="auto"`.
- Prove it on your own data: [evaluate on your data](./evaluate-on-your-data.md) benchmarks Pareta against frontier baselines on your own examples, metered the same way.



---

<!-- examples/speech-to-text.md -->

# Speech to text

Turn a recorded audio clip into text with `pa.audio.transcriptions(...)`. One
call sends the clip to `POST /v1/audio/transcriptions` and returns the
transcript, the detected language, and the audio duration that was metered.

Speech has its own route because audio bytes do not fit the chat message
contract — that is the only reason it is not `chat.completions`. Everything
else works the same way `model="auto"` does for chat: you never pick a serving
model or a GPU; Pareta resolves the ASR lane server-side. Transcription is
metered **per minute of input audio** against your org balance, and an empty
balance raises `InsufficientCreditsError` (402).

## Setup

Install the SDK ([installation guide](../guide/installation.md)), export
`PARETA_API_KEY`, and build the client from the environment:

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();   // reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

## Transcribe a file

The common case is a file on disk: pass the path and the SDK reads and encodes
it for you. The `Transcription` you get back carries the transcript on
`.text`, the detected language on `.language`, and the input duration on
`.duration_s` (`.durationS` in TypeScript) — that duration is what the
per-minute meter charged.

**Python**

```python
t = pa.audio.transcriptions("meeting-clip.wav")

print(t.text)         # the transcript
print(t.language)     # detected language, e.g. "en"
print(t.duration_s)   # metered input length in seconds
```

**TypeScript**

```typescript
const t = await pa.audio.transcriptions("meeting-clip.wav");

console.log(t.text);        // the transcript
console.log(t.language);    // detected language, e.g. "en"
console.log(t.durationS);   // metered input length in seconds
```

Full runnable example: [python/asr/transcribe.py](https://github.com/Pareta-AI/examples/blob/main/python/asr/transcribe.py) · [typescript/asr/transcribe.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/asr/transcribe.ts)

## Bytes and base64 input

`audio` accepts three forms, so the clip does not have to touch disk first.
Raw bytes fit audio you already hold in memory — an upload body, a microphone
buffer — and a base64 string passes pre-encoded audio (say, off a webhook or a
queue message) through untouched. One casing trap in TypeScript: a plain
string is always treated as a **file path**, so pre-encoded audio must be
wrapped as `{ base64: ... }`; in Python a non-path string is assumed to
already be base64.

**Python**

```python
# 1. path (str or os.PathLike) — read + encoded for you
t = pa.audio.transcriptions("meeting-clip.wav")

# 2. raw bytes — e.g. an upload body already in memory
raw = open("meeting-clip.wav", "rb").read()
t = pa.audio.transcriptions(raw)

# 3. base64 string — passed through untouched
import base64
b64 = base64.b64encode(raw).decode("ascii")
t = pa.audio.transcriptions(b64)
```

**TypeScript**

```typescript
import { readFile } from "node:fs/promises";

// 1. string = FILE PATH — read + encoded for you (Node)
let t = await pa.audio.transcriptions("meeting-clip.wav");

// 2. raw bytes — Uint8Array, ArrayBuffer, or Blob
const raw = await readFile("meeting-clip.wav");   // Buffer (a Uint8Array)
t = await pa.audio.transcriptions(raw);

// 3. pre-encoded base64 — must be wrapped, a bare string means a path
t = await pa.audio.transcriptions({ base64: raw.toString("base64") });
```

Full runnable example: [python/asr/transcribe.py](https://github.com/Pareta-AI/examples/blob/main/python/asr/transcribe.py) · [typescript/asr/transcribe.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/asr/transcribe.ts)

## The language hint

`language` is an optional ISO hint. Omit it and the lane detects the language
from the audio itself — the right default for mixed or unknown sources. Pass
it when you already know the language: on short or noisy clips the hint
removes the one thing detection can get wrong.

**Python**

```python
t = pa.audio.transcriptions("support-call.wav", language="en")
print(t.language)   # "en" — the hint you gave, confirmed back
```

**TypeScript**

```typescript
const t = await pa.audio.transcriptions("support-call.wav", { language: "en" });
console.log(t.language);   // "en" — the hint you gave, confirmed back
```

Full runnable example: [python/asr/transcribe.py](https://github.com/Pareta-AI/examples/blob/main/python/asr/transcribe.py) · [typescript/asr/transcribe.ts](https://github.com/Pareta-AI/examples/blob/main/typescript/asr/transcribe.ts)

## See also

- [The audio reference](../reference/audio.md) — full `transcriptions` / `speech` signatures, response models, and metering details.
- [Text to speech](./text-to-speech.md) — the companion lane; the runnable example's sample clip was synthesized with it.
- [Error handling](../guide/errors-and-retries.md) — the exception hierarchy, including `InsufficientCreditsError` (402).
- Prove it on your own data: [evaluate on your data](./evaluate-on-your-data.md) benchmarks the same lanes on your own recordings, metered against the same org balance.



---

<!-- examples/evaluate-on-your-data.md -->

# Benchmark `"auto"` on your own data

A public benchmark tells you how `model="auto"` performs on someone else's data. It does not tell you how it performs on *yours*. This page shows how to take your own labeled rows, score `"auto"` against the frontier baselines it replaces, and read back a cost-annotated verdict, all in one `evals.runs.create(...)` call.

The shape is always the same:

1. Pick a task (it carries the scorer and the input schema).
2. Build an eval set from your rows.
3. Run `"auto"` against `frontier="benchmarked"`.
4. Read the results and the dollar cost of the run.

Evals are metered: the org balance is debited for the compute you ran (`"auto"` **and** the frontier baselines). `run.cost` is the billed total in dollars; an empty balance raises `InsufficientCreditsError`. Top-up is browser-only.

## Setup

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()  # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv(); // reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

`from_env()` is the path you want; it keeps the key out of your source. See [Authentication](../guide/installation.md) for the constructor form and key formats.

## 1. Pick a task

A task defines what gets scored and how. Every eval set, run, and result is anchored to one task id. The task also owns the `default_scorer` (the metric your contenders are judged on) and tells you, via `has_blob_input`, whether rows carry documents or images.

If you already know the id, skip ahead. Otherwise, match free text against the catalog:

**Python**

```python
match = pa.tasks.match("extract key fields from a contract", top_k=5)

if match.matched:
    task_id = match.chosen.task_id          # best candidate
    print(task_id, match.chosen.confidence)  # e.g. "contract-key-fields" "high"
else:
    # nothing landed with confidence; inspect the ranked alternates
    for c in match.candidates:
        print(c.task_id, round(c.score, 3), c.confidence)
    raise SystemExit("refine the query")
```

**TypeScript**

```typescript
const match = await pa.tasks.match("extract key fields from a contract", { topK: 5 });

let taskId: string;
if (match.matched) {
  taskId = match.chosen!.taskId!; // best candidate
  console.log(taskId, match.chosen!.confidence); // e.g. "contract-key-fields" "high"
} else {
  // nothing landed with confidence; inspect the ranked alternates
  for (const c of match.candidates) {
    console.log(c.taskId, c.score?.toFixed(3), c.confidence);
  }
  throw new Error("refine the query");
}
```

`match.ambiguous` is `True` when the top two scores are close, worth surfacing to a human before committing. Confirm the scorer and input schema before you build a set:

**Python**

```python
task = pa.tasks.retrieve(task_id)
print(task.default_scorer)   # the metric your run will report (e.g. "macro_joint_f1")
print(task.has_blob_input)   # True → rows attach PDFs/images (see step 2b)
```

**TypeScript**

```typescript
const task = await pa.tasks.retrieve(taskId);
console.log(task.defaultScorer); // the metric your run will report (e.g. "macro_joint_f1")
console.log(task.hasBlobInput);  // true → rows attach PDFs/images (see step 2b)
```

See the [tasks reference](../reference/tasks.md) for the full matching and catalog surface.

## 2. Build an eval set from your rows

An eval set is your labeled data, stored server-side and reusable across runs. Each row is a dict whose fields match the task schema. The exact keys are task-specific, but the universal shape is **inputs the model sees** plus a **target** (the gold label the scorer compares against).

**Python**

```python
items = [
    {
        "text": "This Agreement is made on 3 March 2026 between Acme Corp and Globex LLC...",
        "target": {"effective_date": "2026-03-03", "parties": ["Acme Corp", "Globex LLC"]},
    },
    {
        "text": "Master Services Agreement, dated January 12, 2026, by and between Initech and Hooli...",
        "target": {"effective_date": "2026-01-12", "parties": ["Initech", "Hooli"]},
    },
    # ... more rows. A few dozen labeled rows already give you a usable signal.
]

eval_set = pa.evals.sets.create(
    task=task_id,
    items=items,
    intent="extract the effective date and parties from each contract",
)

print(eval_set.id)                # use this in runs.create(eval_set=...)
print(eval_set.item_count)        # 2
print(eval_set.scoring_strategy)  # e.g. "extraction"
```

**TypeScript**

```typescript
const items = [
  {
    text: "This Agreement is made on 3 March 2026 between Acme Corp and Globex LLC...",
    target: { effective_date: "2026-03-03", parties: ["Acme Corp", "Globex LLC"] },
  },
  {
    text: "Master Services Agreement, dated January 12, 2026, by and between Initech and Hooli...",
    target: { effective_date: "2026-01-12", parties: ["Initech", "Hooli"] },
  },
  // ... more rows. A few dozen labeled rows already give you a usable signal.
];

const evalSet = await pa.evals.sets.create({
  task: taskId,
  items,
  intent: "extract the effective date and parties from each contract",
});

console.log(evalSet.id);              // use this in runs.create({ evalSet: ... })
console.log(evalSet.itemCount);       // 2
console.log(evalSet.scoringStrategy); // e.g. "extraction"
```

`items` must be non-empty (an empty list raises `ValueError` before any request goes out). If you omit `name`, the set is labeled `"sdk eval set (N items)"`.

Reuse a set across many runs, or list and prune as you iterate:

**Python**

```python
for s in pa.evals.sets.list():
    print(s.id, s.task_id, s.item_count, s.name)

# pa.evals.sets.delete(eval_set.id)   # when you are done with it
```

**TypeScript**

```typescript
for (const s of await pa.evals.sets.list()) {
  console.log(s.id, s.taskId, s.itemCount, s.name);
}

// await pa.evals.sets.delete(evalSet.id);   // when you are done with it
```

### 2b. Document tasks: attach the file to each row

When `task.has_blob_input` is `True`, the row carries a binary document. Create the set with the row's text/label fields and a placeholder for the blob, then attach the file to that row by index:

**Python**

```python
doc_task = "invoice-extraction"   # a has_blob_input task

eval_set = pa.evals.sets.create(
    task=doc_task,
    items=[
        {"target": {"invoice_number": "INV-7781", "total": "1240.00"}},
        {"target": {"invoice_number": "INV-7782", "total": "98.50"}},
    ],
    intent="extract the invoice number and total from each invoice",
)

# Attach one PDF per row. idx is the 0-based row; field_name is the blob input
# field from the task schema. MIME is auto-detected from the filename.
pa.evals.sets.upload_document(eval_set.id, "invoices/7781.pdf", idx=0, field_name="document")
pa.evals.sets.upload_document(eval_set.id, "invoices/7782.pdf", idx=1, field_name="document")
```

**TypeScript**

```typescript
const docTask = "invoice-extraction"; // a hasBlobInput task

const evalSet = await pa.evals.sets.create({
  task: docTask,
  items: [
    { target: { invoice_number: "INV-7781", total: "1240.00" } },
    { target: { invoice_number: "INV-7782", total: "98.50" } },
  ],
  intent: "extract the invoice number and total from each invoice",
});

// Attach one PDF per row. idx is the 0-based row; fieldName is the blob input
// field from the task schema. MIME is auto-detected from the filename.
await pa.evals.sets.uploadDocument(evalSet.id, "invoices/7781.pdf", { idx: 0, fieldName: "document" });
await pa.evals.sets.uploadDocument(evalSet.id, "invoices/7782.pdf", { idx: 1, fieldName: "document" });
```

`upload_document` accepts a path (`str`/`Path`), raw `bytes`, or any binary file-like object; anything else raises `TypeError`. Files under 5 MiB upload inline; larger ones go through a signed-URL direct-to-storage flow. Either way the call returns the completion response dict. Pass `mime="application/pdf"` to override detection.

## 3. Run `"auto"` against the frontier baselines

This is the core call. The contender is `"auto"` — Pareta's routing brain, run against every row exactly as it runs in production — and `frontier="benchmarked"` pulls the vendor baselines Pareta has already benchmarked on this task. The run scores everything on the same rows with the same scorer, so the numbers are directly comparable.

**Python**

```python
run = pa.evals.runs.create(
    eval_set=eval_set.id,
    models=["auto"],          # the contender: Pareta's routing brain
    frontier="benchmarked",   # vendor baselines benchmarked on this task
    wait=True,                # block until the run is terminal
)

print(run.status)  # "completed"
```

**TypeScript**

```typescript
const run = await pa.evals.runs.create({
  evalSet: evalSet.id,
  models: ["auto"],         // the contender: Pareta's routing brain
  frontier: "benchmarked",  // vendor baselines benchmarked on this task
  wait: true,               // block until the run is terminal
});

console.log(run.status); // "completed"
```

`models` is required and is always `["auto"]` — individual open-weights models are not part of the eval surface; they stay behind auto's routing. `frontier` controls the baselines:

| `frontier=` | Evaluates against |
|---|---|
| `None` or `"none"` | nothing (`"auto"` alone) |
| `"benchmarked"` | frontier models Pareta has already benchmarked on this task (vision-filtered for document tasks) |
| `"all"` | every frontier model in the eval pool for the task |
| `["gpt-5.5", "claude-sonnet-4-6"]` | exactly these frontier ids |

The `"benchmarked"` and `"all"` keywords need to know the task. With `eval_set=...` the SDK looks it up from the set; if you pass an explicit list of ids it skips the lookup entirely.

GPUs and serving hardware never enter this call. There is no GPU, quantization, or run-mode knob — and no model to pick. You name a task and the baselines; Pareta resolves the rest. Frontier ids are the vendor names in the clear.

### Inline create (skip step 2)

If you do not need a reusable set, hand the rows straight to the run. Pass `items=` and `intent=` instead of `eval_set=`, and the SDK creates the set for you:

**Python**

```python
run = pa.evals.runs.create(
    task=task_id,
    items=items,
    intent="extract the effective date and parties from each contract",
    models=["auto"],
    frontier="benchmarked",
    wait=True,
)
```

**TypeScript**

```typescript
const run = await pa.evals.runs.create({
  task: taskId,
  items,
  intent: "extract the effective date and parties from each contract",
  models: ["auto"],
  frontier: "benchmarked",
  wait: true,
});
```

You must pass either `eval_set=<id>` or `items=` plus `intent=` (`task=` is optional); anything else raises `ValueError`.

### Pinning the frontier roster

To see exactly which baselines a keyword resolves to — or to build an explicit `frontier=[...]` list — enumerate the roster first with `evals.frontier_models`; each entry exposes `.id`, `.vendor`, `.vision`, and `.benchmarked`:

**Python**

```python
roster = pa.evals.frontier_models(task=task_id)
for m in roster:
    print(m.id, m.vendor, "vision" if m.vision else "text", "benchmarked" if m.benchmarked else "-")

# Pin two of them explicitly
ids = [m.id for m in roster if m.benchmarked][:2]
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier=ids, wait=True)
```

**TypeScript**

```typescript
const roster = await pa.evals.frontierModels(taskId);
for (const m of roster) {
  console.log(m.id, m.vendor, m.vision ? "vision" : "text", m.benchmarked ? "benchmarked" : "-");
}

// Pin two of them explicitly
const ids = roster.filter((m) => m.benchmarked).map((m) => m.id).slice(0, 2);
const run = await pa.evals.runs.create({ evalSet: evalSet.id, models: ["auto"], frontier: ids, wait: true });
```

`frontier_models()` annotates `benchmarked` and applies the vision filter only when you pass `task=`. Without a task it returns the full roster, unannotated.

## 4. Read the ranked results

A terminal run carries one `EvalResult` per contender — `"auto"` plus each baseline. Sort by `quality_mean` to see where auto lands, and read `run.cost` to see what the run cost you:

**Python**

```python
ranked = sorted(run.results, key=lambda r: r.quality_mean or 0, reverse=True)

for r in ranked:
    cost_per_item = (r.mean_cost_micro_usd or 0) / 1_000_000  # micro-USD → dollars
    print(
        f"{r.model_id:24}  "
        f"quality={r.quality_mean:.3f}  "
        f"[{r.quality_ci_low:.3f}, {r.quality_ci_high:.3f}]  "
        f"${cost_per_item:.6f}/item  "
        f"ok={r.n_succeeded} err={r.error_count}"
    )

print(f"\nrun cost: ${run.cost}")          # Decimal dollars, floored to cents
print(f"raw micro-USD: {run.cost_micro_usd}")
```

**TypeScript**

```typescript
const ranked = [...run.results].sort((a, b) => (b.qualityMean ?? 0) - (a.qualityMean ?? 0));

for (const r of ranked) {
  const costPerItem = (r.meanCostMicroUsd ?? 0) / 1_000_000; // micro-USD → dollars
  console.log(
    `${(r.modelId ?? "").padEnd(24)}  ` +
      `quality=${r.qualityMean!.toFixed(3)}  ` +
      `[${r.qualityCiLow!.toFixed(3)}, ${r.qualityCiHigh!.toFixed(3)}]  ` +
      `$${costPerItem.toFixed(6)}/item  ` +
      `ok=${r.nSucceeded} err=${r.errorCount}`,
  );
}

console.log(`\nrun cost: $${run.cost}`);     // dollar string, floored to cents
console.log(`raw micro-USD: ${run.costMicroUsd}`);
```

What the fields mean:

- **`quality_mean`**: the contender's mean score on the task's scorer, in `[0, 1]`.
- **`quality_ci_low` / `quality_ci_high`**: the 95% confidence interval. If two contenders' intervals overlap heavily, your eval set is too small to separate them, so add rows.
- **`mean_cost_micro_usd`**: average cost per item, kept in micro-USD (not floored). This is where the auto-vs-frontier comparison lives, so sub-cent precision is preserved: auto matching frontier quality at a fraction of the cost is the whole point.
- **`n_succeeded` / `error_count`**: how many rows scored cleanly. Auto's failures count as errors, not skips — availability is part of what a benchmark should measure.
- **`model_id`**: `"auto"` for Pareta's row; the vendor id for each baseline. `kind` is `"frontier"` on the baseline rows, so you can filter the contender from what it is measured against.

Reading the verdict: auto's quality CI overlapping the frontier's at a lower per-item cost = frontier-grade on your data; a higher mean without overlap = ahead.

### A note on money

`run.cost` is a `Decimal` of dollars, floored to whole cents, so the SDK never overstates a charge and a sub-cent run reads `Decimal("0.00")`. For the exact figure use `run.cost_micro_usd` (an integer, where `1_000_000` micro-USD is `$1.00`). The same convention is why per-item rates like `mean_cost_micro_usd` stay in micro-USD: flooring them to cents would erase the auto-vs-frontier difference you ran the eval to find.

## Not blocking on the run

`wait=True` polls until the run reaches `"completed"` or `"failed"`, then returns. For long sets, tune the cadence and ceiling:

**Python**

```python
run = pa.evals.runs.create(
    eval_set=eval_set.id,
    models=["auto"],
    frontier="benchmarked",
    wait=True,
    poll_interval=5.0,   # seconds between polls (default 3.0)
    timeout=1800.0,      # give up after 30 min (default 900.0); raises ParetaError on timeout
)
```

**TypeScript**

```typescript
const run = await pa.evals.runs.create({
  evalSet: evalSet.id,
  models: ["auto"],
  frontier: "benchmarked",
  wait: true,
  pollInterval: 5,   // seconds between polls (default 3)
  timeout: 1800,     // give up after 30 min (default 900); throws ParetaError on timeout
});
```

Or fire and poll yourself. `wait=False` returns immediately with a run you can retrieve later:

**Python**

```python
run = pa.evals.runs.create(eval_set=eval_set.id,
                           models=["auto"], frontier="benchmarked")
run_id = run.id
# ... later, from anywhere ...
run = pa.evals.runs.retrieve(run_id)
if run.is_terminal:
    print(run.status, run.results)

# equivalently, block on an already-started run:
run = pa.evals.runs.wait(run_id, timeout=1800.0)
```

**TypeScript**

```typescript
let run = await pa.evals.runs.create({
  evalSet: evalSet.id,
  models: ["auto"],
  frontier: "benchmarked",
});
const runId = run.id!;
// ... later, from anywhere ...
run = await pa.evals.runs.retrieve(runId);
if (run.isTerminal) {
  console.log(run.status, run.results);
}

// equivalently, block on an already-started run:
run = await pa.evals.runs.wait(runId, { timeout: 1800 });
```

## Handling an empty balance

Both auto's compute and the frontier baselines are metered. If the org balance cannot cover the run, `create` raises before any work is billed:

**Python**

```python
from pareta import InsufficientCreditsError

try:
    run = pa.evals.runs.create(eval_set=eval_set.id,
                               models=["auto"], frontier="benchmarked", wait=True)
except InsufficientCreditsError:
    print("Out of credit. Top up in the dashboard (billing is browser-only).")
```

**TypeScript**

```typescript
import { InsufficientCreditsError } from "pareta";

try {
  const run = await pa.evals.runs.create({
    evalSet: evalSet.id,
    models: ["auto"],
    frontier: "benchmarked",
    wait: true,
  });
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    console.log("Out of credit. Top up in the dashboard (billing is browser-only).");
  } else {
    throw e;
  }
}
```

`InsufficientCreditsError` is a subclass of `APIStatusError` (status 402), so you can also catch the broader `ParetaError` if you want one handler for every SDK failure.

## Full example

**Python**

```python
from pareta import Pareta, InsufficientCreditsError

pa = Pareta.from_env()

# 1. Pick the task.
task_id = "contract-key-fields"
task = pa.tasks.retrieve(task_id)
print("scoring on:", task.default_scorer)

# 2. Build the eval set from your rows.
items = [
    {"text": "This Agreement is made on 3 March 2026 between Acme Corp and Globex LLC...",
     "target": {"effective_date": "2026-03-03", "parties": ["Acme Corp", "Globex LLC"]}},
    {"text": "Master Services Agreement, dated January 12, 2026, by and between Initech and Hooli...",
     "target": {"effective_date": "2026-01-12", "parties": ["Initech", "Hooli"]}},
]
eval_set = pa.evals.sets.create(
    task=task_id, items=items,
    intent="extract the effective date and parties from each contract",
    name="contract fields v1")

# 3. Run auto against the benchmarked frontier baselines.
try:
    run = pa.evals.runs.create(
        eval_set=eval_set.id,
        models=["auto"],
        frontier="benchmarked",
        wait=True,
    )
except InsufficientCreditsError:
    raise SystemExit("Out of credit. Top up in the dashboard.")

# 4. Read the ranked results.
for r in sorted(run.results, key=lambda r: r.quality_mean or 0, reverse=True):
    print(f"{r.model_id:24} {r.quality_mean:.3f}  ${(r.mean_cost_micro_usd or 0)/1e6:.6f}/item")

print("run cost:", run.cost)  # Decimal dollars, floored to cents
```

**TypeScript**

```typescript
import { Pareta, InsufficientCreditsError } from "pareta";

const pa = Pareta.fromEnv();

// 1. Pick the task.
const taskId = "contract-key-fields";
const task = await pa.tasks.retrieve(taskId);
console.log("scoring on:", task.defaultScorer);

// 2. Build the eval set from your rows.
const items = [
  { text: "This Agreement is made on 3 March 2026 between Acme Corp and Globex LLC...",
    target: { effective_date: "2026-03-03", parties: ["Acme Corp", "Globex LLC"] } },
  { text: "Master Services Agreement, dated January 12, 2026, by and between Initech and Hooli...",
    target: { effective_date: "2026-01-12", parties: ["Initech", "Hooli"] } },
];
const evalSet = await pa.evals.sets.create({
  task: taskId, items,
  intent: "extract the effective date and parties from each contract",
  name: "contract fields v1",
});

// 3. Run auto against the benchmarked frontier baselines.
let run;
try {
  run = await pa.evals.runs.create({
    evalSet: evalSet.id,
    models: ["auto"],
    frontier: "benchmarked",
    wait: true,
  });
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    throw new Error("Out of credit. Top up in the dashboard.");
  }
  throw e;
}

// 4. Read the ranked results.
for (const r of [...run.results].sort((a, b) => (b.qualityMean ?? 0) - (a.qualityMean ?? 0))) {
  console.log(`${(r.modelId ?? "").padEnd(24)} ${r.qualityMean!.toFixed(3)}  $${((r.meanCostMicroUsd ?? 0) / 1e6).toFixed(6)}/item`);
}

console.log("run cost:", run.cost); // dollar string, floored to cents
```

## Async

Every call here has an `async` twin on `AsyncPareta`. The signatures match; the methods are coroutines (`wait` included).

**Python**

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        eval_set = await pa.evals.sets.create(
            task="contract-key-fields", items=items,
            intent="extract the effective date and parties from each contract")
        run = await pa.evals.runs.create(
            eval_set=eval_set.id,
            models=["auto"],
            frontier="benchmarked",
            wait=True,
        )
        for r in run.results:
            print(r.model_id, r.quality_mean)
        print("run cost:", run.cost)

asyncio.run(main())
```

**TypeScript**

In TypeScript there is no separate `AsyncPareta` — the one `Pareta` client is already
async. Every I/O method returns a `Promise`, so you just `await` it; there is no sync/async
split to mirror and no context manager to close.

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

const evalSet = await pa.evals.sets.create({
  task: "contract-key-fields", items,
  intent: "extract the effective date and parties from each contract",
});
const run = await pa.evals.runs.create({
  evalSet: evalSet.id,
  models: ["auto"],
  frontier: "benchmarked",
  wait: true,
});
for (const r of run.results) {
  console.log(r.modelId, r.qualityMean);
}
console.log("run cost:", run.cost);
```

## Next steps

- [Run inference](../guide/inference.md): send production traffic to the same `model="auto"` your eval just measured; inference is metered the same way evals are.
- [Cost & quality monitoring](./cost-and-metrics.md): watch spend, success, and projected savings with `auto.metrics()`.
- [Errors and retries](../guide/errors-and-retries.md): the full exception hierarchy behind `InsufficientCreditsError` and friends.



---

<!-- examples/document-extraction.md -->

# Document extraction (PDF/image)

Pull structured fields out of PDFs and scanned images with `model="auto"` — and prove, on your own documents, that the routing holds frontier quality at a fraction of the cost.

`model="auto"` already routes document requests to Pareta's benchmark-proven extraction specialists — nothing to deploy, no model to pick. This page walks the loop that keeps you honest about it:

1. Find the blob task and check what it expects.
2. Build an eval set from your own documents (one JSONL row per document, with the PDF/image attached to each row).
3. Benchmark `"auto"` against frontier (vision) baselines on those documents.
4. Read the quality and cost verdict.
5. Send production documents to `model="auto"` with OpenAI-compatible inference.

Why do it this way: a document task is a *blob* task. The model reads pixels, not just text, so trusting any router — Pareta's included — on gut is a bad idea. Running an eval on your real documents tells you, in dollars and quality points, how `"auto"` compares to the frontier on exactly your paper. Both evals and inference are metered against your org balance, so the eval also tells you the bill before you commit.

Throughout, there is no model to name and no hardware to size: the only inference id is `"auto"`, and frontier (vendor) ids appear only as eval baselines.

## Setup

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();   // reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

`from_env()` is the path you want in real code. See [Inference](../guide/inference.md) for the OpenAI-compatible alternative.

## 1. Find the document task

Document tasks carry binary inputs, surfaced as `task.has_blob_input == True`. If you know the task id (here, `invoice-extraction`), retrieve it directly. If you only know your intent in words, let the matcher rank candidates.

**Python**

```python
# By id
task = pa.tasks.retrieve("invoice-extraction")
print(task.id, task.default_scorer, task.has_blob_input)
# invoice-extraction  field_f1  True

# Or by intent
m = pa.tasks.match("extract totals and line items from vendor invoices")
if m.matched:
    print("chose:", m.chosen.task_id, m.chosen.confidence)   # 'high' | 'medium' | 'low'
for c in m.candidates:
    print(f"  {c.task_id}  score={c.score:.2f}  {c.confidence}")
```

**TypeScript**

```typescript
// By id
const task = await pa.tasks.retrieve("invoice-extraction");
console.log(task.id, task.defaultScorer, task.hasBlobInput);
// invoice-extraction  field_f1  true

// Or by intent
const m = await pa.tasks.match("extract totals and line items from vendor invoices");
if (m.matched) {
  console.log("chose:", m.chosen.taskId, m.chosen.confidence);   // 'high' | 'medium' | 'low'
}
for (const c of m.candidates) {
  console.log(`  ${c.taskId}  score=${c.score.toFixed(2)}  ${c.confidence}`);
}
```

`task.default_scorer` is the scorer the eval run applies (for a field-extraction task that is typically a field-level F1). You do not invoke it yourself; the run scores each contender against the expected output you provide in step 2.

See the [tasks reference](../reference/tasks.md) for the full catalog and matching surface.

## 2. Build an eval set from your documents

A document eval set is one JSONL row per document. Each row holds the *expected* extraction (what a correct answer looks like) plus a placeholder for the document blob. You attach the actual PDF/image to each row in a second step with `upload_document`.

Create the set first. `items` must be non-empty. The blob field (here `document`) is the input field the document attaches to; the rest of each row is the gold/expected output the scorer grades against.

**Python**

```python
items = [
    {
        "document": None,                       # filled by upload_document below
        "expected": {
            "invoice_number": "INV-4471",
            "invoice_date": "2026-03-14",
            "total": "1284.50",
            "currency": "USD",
            "vendor": "Katana ML",
        },
    },
    {
        "document": None,
        "expected": {
            "invoice_number": "INV-4472",
            "invoice_date": "2026-03-15",
            "total": "962.00",
            "currency": "USD",
            "vendor": "Katana ML",
        },
    },
]

eval_set = pa.evals.sets.create(
    task="invoice-extraction",
    items=items,
    intent="extract invoice_number, invoice_date, total, currency, and vendor from each invoice",
    name="Q1 vendor invoices (10 docs)",   # optional; auto-named if omitted
)
print(eval_set.id, eval_set.item_count, eval_set.scoring_strategy)
# es_…  2  extraction
```

**TypeScript**

```typescript
const items = [
  {
    document: null,                          // filled by uploadDocument below
    expected: {
      invoice_number: "INV-4471",
      invoice_date: "2026-03-14",
      total: "1284.50",
      currency: "USD",
      vendor: "Katana ML",
    },
  },
  {
    document: null,
    expected: {
      invoice_number: "INV-4472",
      invoice_date: "2026-03-15",
      total: "962.00",
      currency: "USD",
      vendor: "Katana ML",
    },
  },
];

const evalSet = await pa.evals.sets.create({
  task: "invoice-extraction",
  items,
  intent: "extract invoice_number, invoice_date, total, currency, and vendor from each invoice",
  name: "Q1 vendor invoices (10 docs)",   // optional; auto-named if omitted
});
console.log(evalSet.id, evalSet.itemCount, evalSet.scoringStrategy);
// es_…  2  extraction
```

Now attach the document for each row. `idx` is the 0-based row index and `field_name` is the blob field you left as `None` above. `file` accepts a path (`str`/`Path`), raw `bytes`, or any binary file-like object; the MIME type is guessed from the filename and can be overridden with `mime=`.

**Python**

```python
from pathlib import Path

invoices = [Path("invoices/INV-4471.pdf"), Path("invoices/INV-4472.png")]

for idx, path in enumerate(invoices):
    pa.evals.sets.upload_document(
        eval_set.id,
        path,
        idx=idx,
        field_name="document",
    )
```

**TypeScript**

```typescript
const invoices = ["invoices/INV-4471.pdf", "invoices/INV-4472.png"];

for (let idx = 0; idx < invoices.length; idx++) {
  await pa.evals.sets.uploadDocument(
    evalSet.id,
    invoices[idx],
    { idx, fieldName: "document" },
  );
}
```

The upload is one call regardless of file size. Files under 5 MiB go up inline; larger files use a signed-URL direct-to-storage flow under the hood. You can also pass bytes or a handle:

**Python**

```python
with open("invoices/INV-4471.pdf", "rb") as fh:
    pa.evals.sets.upload_document(eval_set.id, fh, idx=0, field_name="document")

raw = Path("scan.tiff").read_bytes()
pa.evals.sets.upload_document(
    eval_set.id, raw, idx=1, field_name="document", mime="image/tiff",
)
```

**TypeScript**

```typescript
import { readFile } from "node:fs/promises";

// `file` accepts a path, a Blob, an ArrayBuffer, or a Uint8Array.
const pdf = await readFile("invoices/INV-4471.pdf");   // Buffer (a Uint8Array)
await pa.evals.sets.uploadDocument(evalSet.id, pdf, { idx: 0, fieldName: "document" });

const raw = await readFile("scan.tiff");
await pa.evals.sets.uploadDocument(
  evalSet.id, raw, { idx: 1, fieldName: "document", mime: "image/tiff" },
);
```

## 3. Run the eval

The contender is `"auto"` — the same routing that will serve your production documents — and `frontier=` chooses the baselines it is measured against. For a document task you want vision-capable baselines; `frontier="benchmarked"` resolves to the frontier models Pareta has already benchmarked on this task (vision-filtered for document tasks), so you compare against the right roster automatically.

**Python**

```python
run = pa.evals.runs.create(
    eval_set=eval_set.id,
    models=["auto"],                         # the contender: Pareta's routing brain
    frontier="benchmarked",                  # vision frontier baselines on this task
    wait=True,                               # block until the run is terminal
)
print(run.status, run.id)                    # 'completed'  run_…
```

**TypeScript**

```typescript
const run = await pa.evals.runs.create({
  evalSet: evalSet.id,
  models: ["auto"],                         // the contender: Pareta's routing brain
  frontier: "benchmarked",                  // vision frontier baselines on this task
  wait: true,                               // block until the run is terminal
});
console.log(run.status, run.id);            // 'completed'  run_…
```

`wait=True` polls until the run reaches `completed` or `failed` (default `poll_interval=3.0`s, `timeout=900.0`s), then returns the final run. To fire and poll yourself, leave `wait=False` and call `runs.wait(run.id)` or `runs.retrieve(run.id)` later:

**Python**

```python
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier="benchmarked")
# ... do other work ...
run = pa.evals.runs.wait(run.id, poll_interval=5.0, timeout=1200.0)
```

**TypeScript**

```typescript
let run = await pa.evals.runs.create({ evalSet: evalSet.id, models: ["auto"], frontier: "benchmarked" });
// ... do other work ...
run = await pa.evals.runs.wait(run.id, { pollInterval: 5, timeout: 1200 });
```

If you would rather not pre-create the set, `runs.create` accepts `items=… + intent=…` inline (`task=…` optional) and creates the set for you. You still attach blobs first, so for document tasks the explicit `sets.create` + `upload_document` path above is the one to use.

### What `frontier=` accepts

| Value | Resolves to |
|---|---|
| `None` or `"none"` | no baselines |
| `"all"` | every frontier model for the task (from `pa.evals.frontier_models(task=…)`) |
| `"benchmarked"` | frontier models Pareta has already benchmarked on this task (vision-filtered for document tasks) |
| `["gpt-5.5", "claude-sonnet-4-6"]` | exactly those frontier ids |

Frontier (vendor) ids are in the clear, so you can name them explicitly. To see the roster first:

**Python**

```python
for fm in pa.evals.frontier_models(task="invoice-extraction"):
    print(fm.id, fm.vendor, "vision" if fm.vision else "text", "benchmarked" if fm.benchmarked else "")
```

**TypeScript**

```typescript
for (const fm of await pa.evals.frontierModels("invoice-extraction")) {
  console.log(fm.id, fm.vendor, fm.vision ? "vision" : "text", fm.benchmarked ? "benchmarked" : "");
}
```

### Metering

The run debits your org balance for the compute it consumes, auto and frontier baselines alike. If the balance is empty, `runs.create` raises `InsufficientCreditsError` (402). Top-up is browser-only; the SDK never exposes balance or payment methods.

**Python**

```python
from pareta import InsufficientCreditsError

try:
    run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier="benchmarked", wait=True)
except InsufficientCreditsError:
    print("Org out of credit — top up in the dashboard, then re-run.")
```

**TypeScript**

```typescript
import { InsufficientCreditsError } from "pareta";

try {
  const run = await pa.evals.runs.create({ evalSet: evalSet.id, models: ["auto"], frontier: "benchmarked", wait: true });
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    console.log("Org out of credit — top up in the dashboard, then re-run.");
  } else {
    throw e;
  }
}
```

## 4. Read the verdict

`run.results` is one `EvalResult` per contender: `"auto"` plus each frontier baseline, each with mean quality, a 95% confidence interval, and the average cost per item. The baseline rows carry `kind == "frontier"`; auto's row is the one with `model_id == "auto"`. `run.cost` is the billed total for the whole run.

**Python**

```python
if run.status == "failed":
    raise RuntimeError(run.error_detail)

print(f"run cost: ${run.cost}")              # Decimal dollars, floored to cents

for r in sorted(run.results, key=lambda r: (r.quality_mean or 0), reverse=True):
    print(
        f"{r.model_id:18} {(r.kind or ''):8} "
        f"q={r.quality_mean:.3f} "
        f"[{r.quality_ci_low:.3f}, {r.quality_ci_high:.3f}]  "
        f"{r.mean_cost_micro_usd} uUSD/item  "
        f"ok={r.n_succeeded} err={r.error_count}"
    )
```

**TypeScript**

```typescript
if (run.status === "failed") {
  throw new Error(run.errorDetail ?? "eval run failed");
}

console.log(`run cost: $${run.cost}`);       // dollar string, floored to cents

const sorted = [...run.results].sort((a, b) => (b.qualityMean ?? 0) - (a.qualityMean ?? 0));
for (const r of sorted) {
  console.log(
    `${r.modelId.padEnd(18)} ${(r.kind ?? "").padEnd(8)} ` +
    `q=${r.qualityMean.toFixed(3)} ` +
    `[${r.qualityCiLow.toFixed(3)}, ${r.qualityCiHigh.toFixed(3)}]  ` +
    `${r.meanCostMicroUsd} uUSD/item  ` +
    `ok=${r.nSucceeded} err=${r.errorCount}`,
  );
}
```

```
gpt-5.5            frontier q=0.946 [0.921, 0.968]  41200 uUSD/item  ok=10 err=0
auto                        q=0.938 [0.912, 0.961]   3400 uUSD/item  ok=10 err=0
gemini-3-1-pro     frontier q=0.925 [0.897, 0.950]  28900 uUSD/item  ok=10 err=0
```

Reading it: auto lands within a point of the strongest frontier baseline — the CIs overlap, so the two are not meaningfully different on this sample — at roughly a twelfth of the per-item cost. That is the verdict the eval exists to deliver, and there is nothing to pick or provision off the back of it: the routing that produced auto's row is the routing that serves production.

### A note on money

`run.cost` is a `Decimal` in dollars, floored to whole cents (the SDK never rounds a charge up). The raw integer is on `run.cost_micro_usd` (1,000,000 = $1.00) if you need sub-cent precision. Per-item rates like `result.mean_cost_micro_usd` stay in micro-USD on purpose; flooring them to cents would erase the auto-vs-frontier comparison that just earned its keep above.

**Python**

```python
print(run.cost)             # Decimal('0.07')
print(run.cost_micro_usd)   # 72500
```

**TypeScript**

```typescript
console.log(run.cost);          // "0.07"  (floored-to-cents dollar string)
console.log(run.costMicroUsd);  // 72500
```

## 5. Send documents to `model="auto"` in production

The routing your eval just measured is already live — it is the standard chat surface with `model="auto"`, OpenAI-compatible on the wire. For a vision document task, send the image in the standard OpenAI content-parts shape; PDFs are typically handed in as page images or a data URL, matching whatever the task expects.

**Python**

```python
import base64

img_b64 = base64.b64encode(open("invoices/new-INV.png", "rb").read()).decode()

resp = pa.chat.completions.create(
    model="auto",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract invoice_number, invoice_date, total, currency, vendor as JSON."},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}"}},
            ],
        }
    ],
    temperature=0,
    max_tokens=512,
)
print(resp.choices[0].message.content)
print(resp.usage.total_tokens)
```

**TypeScript**

```typescript
import { readFile } from "node:fs/promises";

const imgB64 = (await readFile("invoices/new-INV.png")).toString("base64");

const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Extract invoice_number, invoice_date, total, currency, vendor as JSON." },
        { type: "image_url", image_url: { url: `data:image/png;base64,${imgB64}` } },
      ],
    },
  ],
  temperature: 0,
  max_tokens: 512,
});
console.log(resp.choices[0].message.content);
console.log(resp.usage.totalTokens);
```

Inference is metered the same way the eval was: a successful completion debits the org balance — one debit per request, no matter how many internal model calls auto's plan makes — and a zero balance raises `InsufficientCreditsError` (402). To stream tokens as they generate:

**Python**

```python
for chunk in pa.chat.completions.create(model="auto", messages=[...], stream=True):
    print(chunk.choices[0].delta.content or "", end="", flush=True)
```

**TypeScript**

```typescript
for await (const chunk of pa.chat.completions.create({ model: "auto", messages: [...], stream: true })) {
  process.stdout.write(chunk.choices[0].delta.content ?? "");
}
```

## Async

Every step mirrors on `AsyncPareta`. `runs.wait` is awaitable; chat streams are `async for`.

**Python**

```python
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        es = await pa.evals.sets.create(
            task="invoice-extraction", items=items,
            intent="extract invoice_number, invoice_date, total, currency, and vendor from each invoice")
        await pa.evals.sets.upload_document(es.id, "invoices/INV-4471.pdf", idx=0, field_name="document")

        run = await pa.evals.runs.create(
            eval_set=es.id, models=["auto"], frontier="benchmarked", wait=True,
        )
        for r in run.results:
            print(r.model_id, r.quality_mean, r.mean_cost_micro_usd)

        resp = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Extract the total as JSON."}],
        )
        print(resp.choices[0].message.content)
```

**TypeScript**

```typescript
// There is no AsyncPareta in TypeScript — the one `Pareta` client is already
// async. Every I/O method returns a Promise you `await`; streams are `for await`.
// No context manager, no `.close()`: there is no owned connection to release.
import { Pareta } from "pareta";

async function main() {
  const pa = Pareta.fromEnv();

  const es = await pa.evals.sets.create({
    task: "invoice-extraction", items,
    intent: "extract invoice_number, invoice_date, total, currency, and vendor from each invoice",
  });
  await pa.evals.sets.uploadDocument(es.id, "invoices/INV-4471.pdf", { idx: 0, fieldName: "document" });

  const run = await pa.evals.runs.create({
    evalSet: es.id, models: ["auto"], frontier: "benchmarked", wait: true,
  });
  for (const r of run.results) {
    console.log(r.modelId, r.qualityMean, r.meanCostMicroUsd);
  }

  const resp = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "Extract the total as JSON." }],
  });
  console.log(resp.choices[0].message.content);
}
```

## The whole loop

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()
TASK = "invoice-extraction"

# 1. eval set from your documents
es = pa.evals.sets.create(
    task=TASK, items=items,
    intent="extract invoice_number, invoice_date, total, currency, and vendor from each invoice",
    name="vendor invoices")
for idx, path in enumerate(invoices):
    pa.evals.sets.upload_document(es.id, path, idx=idx, field_name="document")

# 2. benchmark auto against vision frontier baselines
run = pa.evals.runs.create(
    eval_set=es.id,
    models=["auto"],
    frontier="benchmarked",
    wait=True,
)
print(f"eval cost ${run.cost}")
for r in run.results:
    print(r.model_id, r.quality_mean, r.mean_cost_micro_usd)

# 3. production is the same id the eval just measured
resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Extract the invoice fields as JSON."}],
)
print(resp.choices[0].message.content)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const TASK = "invoice-extraction";

// 1. eval set from your documents
const es = await pa.evals.sets.create({
  task: TASK, items,
  intent: "extract invoice_number, invoice_date, total, currency, and vendor from each invoice",
  name: "vendor invoices",
});
for (let idx = 0; idx < invoices.length; idx++) {
  await pa.evals.sets.uploadDocument(es.id, invoices[idx], { idx, fieldName: "document" });
}

// 2. benchmark auto against vision frontier baselines
const run = await pa.evals.runs.create({
  evalSet: es.id,
  models: ["auto"],
  frontier: "benchmarked",
  wait: true,
});
console.log(`eval cost $${run.cost}`);
for (const r of run.results) {
  console.log(r.modelId, r.qualityMean, r.meanCostMicroUsd);
}

// 3. production is the same id the eval just measured
const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Extract the invoice fields as JSON." }],
});
console.log(resp.choices[0].message.content);
```

## See also

- [Inference (OpenAI-compatible)](../guide/inference.md) — `model="auto"`, streaming, using the `openai` client.
- [The tasks reference](../reference/tasks.md) — the catalog, `match`, and task schemas.
- [Evaluating models on your data](../guide/evaluation.md) — eval sets, runs, frontier baselines, and metering in depth.
- [Cost & quality monitoring](./cost-and-metrics.md) — read run costs and watch live auto traffic with `auto.metrics()`.



---

<!-- examples/streaming-chat.md -->

# Streaming chat completions

Stream tokens as the model generates them instead of waiting for the whole
response. Pass `stream=True` to `chat.completions.create(...)` and you get an
iterator of `ChatCompletionChunk` objects, each carrying one incremental piece
of text on `chunk.choices[0].delta.content`. Use this for chat UIs, agent
loops, long generations, and anywhere a first-token-fast experience matters.

Inference on Pareta is OpenAI-compatible, so the streaming shape here is the
same vLLM-style data-only SSE the `openai` SDK consumes. Use `model="auto"` —
the routing brain streams progress while it plans and executes, then the
answer tokens. There is nothing to deploy first: `"auto"` is live for every
org. Streamed inference is metered against your org balance exactly like a
non-streaming call.

## Quickstart

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()  # reads PARETA_API_KEY (+ optional PARETA_BASE_URL)

stream = pa.chat.completions.create(
    model="auto",             # the routing brain — the only model id
    messages=[{"role": "user", "content": "Write a haiku about throughput."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv(); // reads PARETA_API_KEY (+ optional PARETA_BASE_URL)

const stream = pa.chat.completions.create({
  model: "auto",            // the routing brain — the only model id
  messages: [{ role: "user", content: "Write a haiku about throughput." }],
  stream: true,
});

for await (const chunk of stream) {
  const delta = chunk.choices[0].delta.content;
  if (delta) {
    process.stdout.write(delta);
  }
}
console.log();
```

`stream=True` changes the return type: instead of a single `ChatCompletion`,
`create(...)` returns an `Iterator[ChatCompletionChunk]`. Nothing is sent until
you start iterating, and the connection stays open for the life of the loop.

## Reading a chunk

A streaming chunk has the same schema as a `ChatCompletion`, but each choice
carries a `delta` (the incremental token) instead of a full `message`:

**Python**

```python
chunk.choices[0].delta.content   # str | None — the new text in this chunk
chunk.choices[0].delta.role      # str | None — usually only set on the first chunk
chunk.choices[0].finish_reason   # str | None — "stop" / "length" on the last chunk
chunk.id                         # str | None
chunk.model                      # str | None
```

**TypeScript**

```typescript
chunk.choices[0].delta.content   // string | null — the new text in this chunk
chunk.choices[0].delta.role      // string | null — usually only set on the first chunk
chunk.choices[0].finishReason    // string | null — "stop" / "length" on the last chunk
chunk.id                         // string | null
chunk.model                      // string | null
```

`delta.content` is `None` on chunks that carry no text (for example the opening
role chunk, or a final chunk that only sets `finish_reason`), so always guard
the `if delta:` check before printing or appending. The stream ends when the
server sends `[DONE]`; the SDK consumes that sentinel and stops the iterator for
you, so a plain `for` loop terminates cleanly.

Need the raw server JSON for a field the typed layer does not surface? Every
response object keeps it: `chunk.to_dict()` returns the untouched payload.

## Accumulating the full text

Collect the deltas into a buffer to reconstruct the complete message:

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()

chunks = pa.chat.completions.create(
    model="auto",
    messages=[
        {"role": "system", "content": "You are concise."},
        {"role": "user", "content": "Summarize what an invoice number is."},
    ],
    stream=True,
    temperature=0.2,   # extra OpenAI params pass straight through
    max_tokens=256,
)

parts = []
finish_reason = None
for chunk in chunks:
    choice = chunk.choices[0]
    if choice.delta.content:
        parts.append(choice.delta.content)
    if choice.finish_reason:
        finish_reason = choice.finish_reason

full_text = "".join(parts)
print(full_text)
print("finish_reason:", finish_reason)  # e.g. "stop" or "length"
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

const chunks = pa.chat.completions.create({
  model: "auto",
  messages: [
    { role: "system", content: "You are concise." },
    { role: "user", content: "Summarize what an invoice number is." },
  ],
  stream: true,
  temperature: 0.2, // extra OpenAI params pass straight through
  max_tokens: 256,
});

const parts: string[] = [];
let finishReason: string | null = null;
for await (const chunk of chunks) {
  const choice = chunk.choices[0];
  if (choice.delta.content) {
    parts.push(choice.delta.content);
  }
  if (choice.finishReason) {
    finishReason = choice.finishReason;
  }
}

const fullText = parts.join("");
console.log(fullText);
console.log("finishReason:", finishReason); // e.g. "stop" or "length"
```

A `finish_reason` of `"length"` means the model hit `max_tokens` before it was
done; raise `max_tokens` if you need the full answer.

Note: token usage is not reliably populated on streamed chunks. If you need the
`usage` counts (`prompt_tokens` / `completion_tokens` / `total_tokens`), make
the same call with `stream=False` and read `completion.usage`.

## Extra parameters

Any OpenAI chat parameter you pass as a keyword argument is forwarded verbatim
in the request body: `temperature`, `max_tokens`, `top_p`, `stop`,
`frequency_penalty`, and so on. There is no model knob and no hardware knob —
model choice, GPUs, and quantization are resolved by Pareta per request, so
the only id you pass to `model` is `"auto"`.

**Python**

```python
stream = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "List three GPU-free wins."}],
    stream=True,
    top_p=0.9,
    stop=["\n\n"],
)
```

**TypeScript**

```typescript
const stream = pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "List three GPU-free wins." }],
  stream: true,
  top_p: 0.9,
  stop: ["\n\n"],
});
```

## Async streaming

`AsyncPareta` mirrors the sync client. `create(...)` is a coroutine, so
`await` it once to get the async iterator, then drive it with `async for`:

**Python**

```python
import asyncio
from pareta import AsyncPareta


async def main():
    async with AsyncPareta.from_env() as pa:
        stream = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Stream me a limerick."}],
            stream=True,
        )
        async for chunk in stream:
            delta = chunk.choices[0].delta.content
            if delta:
                print(delta, end="", flush=True)
        print()


asyncio.run(main())
```

**TypeScript**

```typescript
// There is no AsyncPareta in TypeScript: the single `Pareta` client is already
// async. `create({ stream: true })` returns an AsyncIterable<ChatCompletionChunk>
// directly — drive it with `for await`, no separate await for the stream handle.
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

const stream = pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Stream me a limerick." }],
  stream: true,
});

for await (const chunk of stream) {
  const delta = chunk.choices[0].delta.content;
  if (delta) {
    process.stdout.write(delta);
  }
}
console.log();
```

The `async with` block calls `aclose()` for you when the block exits, releasing
the HTTP client. The chunk shape is identical to the sync path:
`chunk.choices[0].delta.content` is the incremental text.

## Metering and errors

Streamed inference debits your org balance on success, the same as a
non-streaming completion. Top-ups are browser-only; the SDK does not expose
balance or payment methods. If the balance is empty, the call raises
`InsufficientCreditsError` (HTTP 402) before any tokens flow:

**Python**

```python
from pareta import Pareta
from pareta import InsufficientCreditsError, EndpointNotReadyError

pa = Pareta.from_env()

try:
    stream = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True,
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)
    print()
except InsufficientCreditsError:
    print("Out of credit — top up in the dashboard.")
except EndpointNotReadyError:
    print("A backend behind auto is briefly unavailable — retry in a moment.")
```

**TypeScript**

```typescript
import { Pareta, InsufficientCreditsError, EndpointNotReadyError } from "pareta";

const pa = Pareta.fromEnv();

try {
  const stream = pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "Hello" }],
    stream: true,
  });
  for await (const chunk of stream) {
    const delta = chunk.choices[0].delta.content;
    if (delta) {
      process.stdout.write(delta);
    }
  }
  console.log();
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    console.log("Out of credit — top up in the dashboard.");
  } else if (e instanceof EndpointNotReadyError) {
    console.log("A backend behind auto is briefly unavailable — retry in a moment.");
  } else {
    throw e;
  }
}
```

A few things to know about how the stream behaves under failure:

- **`model` / `messages` validation is local.** Passing an empty `model` or
  empty `messages` raises `ValueError` immediately, before any network call.
- **Errors surface before the first byte.** Non-2xx responses (402, 401, 404,
  503, and so on) are raised as the matching `ParetaError` subclass when the
  stream starts, not mid-loop. A 503 — a serving backend behind auto warming
  up or briefly unavailable — surfaces as `EndpointNotReadyError` only after
  the SDK's automatic retries are exhausted.
- **Mid-stream drops are not retried.** Retries cover only the initial
  connect/handshake. Once SSE bytes are flowing, a dropped connection raises
  (`APIConnectionError` / `APITimeoutError`) rather than silently resuming,
  because a partial generation cannot be safely continued. Wrap the loop and
  re-issue the request if you need at-least-once delivery.

See [error handling](../guide/errors-and-retries.md) for the full exception hierarchy.

## Related

- [Inference](../guide/inference.md) — the full chat surface; `models.list()`
  returns exactly one entry, `"auto"`.
- [Non-streaming completions](../guide/inference.md) — `stream=False` returns a
  single `ChatCompletion` with `usage` populated.
- [Running evals](../guide/evaluation.md) — benchmark `"auto"` against frontier
  baselines on your own data, also metered against the org balance.



---

<!-- examples/concurrent-async.md -->

# Concurrent calls with AsyncPareta

`AsyncPareta` lets you fire many requests at once instead of one at a time. When
you have a batch of inference prompts to score, or several eval runs to kick off,
running them concurrently turns a wall of sequential round-trips into a single
`asyncio.gather`. The same surface as the sync [`Pareta`](../reference/client.md)
client, with every resource method `async def` and the streaming iterators
async.

This page shows how to:

- run a batch of `chat.completions` concurrently and collect results
- bound concurrency so you do not hammer the API (backpressure)
- handle errors per task so one failure does not sink the batch
- launch and await several eval runs at once

One `AsyncPareta` instance wraps a single pooled `httpx.AsyncClient`. Build it
once, share it across all your coroutines, and close it once. Do not make a
client per request.

**Python**

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:   # reads PARETA_API_KEY
        resp = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Extract the total."}],
        )
        print(resp.choices[0].message.content)

asyncio.run(main())
```

**TypeScript**

The TS SDK has no sync/async split — there is one `Pareta` class, and every
I/O method already returns a `Promise` you `await`. No `AsyncPareta`, no
`asyncio.run`, no `async with`: build one client and share it.

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();                  // reads PARETA_API_KEY

const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Extract the total." }],
});
console.log(resp.choices[0].message.content);
```

Inference is OpenAI-compatible and metered: each successful completion debits
your org balance — one debit per request, no matter how many internal model
calls auto's plan makes — and a zero balance raises `InsufficientCreditsError`
(402). Top-up is browser-only, so the SDK never exposes balance or payment.
`model` is always `"auto"`: Pareta picks the model per request and hides the
hardware, so there is no model or GPU knob to pass.

## Fan out a batch of completions

`asyncio.gather` runs every coroutine concurrently and returns results in input
order. Because all calls share the same client, httpx pools and reuses
connections for you.

**Python**

```python
import asyncio
from pareta import AsyncPareta

PROMPTS = [
    "Extract the invoice total.",
    "Extract the vendor name.",
    "Extract the due date.",
    "Extract the line-item count.",
]

async def classify_one(pa, prompt, document):
    resp = await pa.chat.completions.create(
        model="auto",
        messages=[
            {"role": "system", "content": "You are an invoice parser."},
            {"role": "user", "content": f"{prompt}\n\n{document}"},
        ],
        temperature=0,
        max_tokens=64,
    )
    return resp.choices[0].message.content

async def main():
    document = "INVOICE #4471 ... TOTAL $1,240.00 ..."
    async with AsyncPareta.from_env() as pa:
        answers = await asyncio.gather(
            *(classify_one(pa, p, document) for p in PROMPTS)
        )
    for prompt, answer in zip(PROMPTS, answers):
        print(f"{prompt} -> {answer}")

asyncio.run(main())
```

**TypeScript**

`Promise.all` is the direct analog of `asyncio.gather`: it runs every promise
concurrently and resolves to results in input order. The shared `fetch` keep-alive
pool reuses connections for you.

```typescript
import { Pareta } from "pareta";

const PROMPTS = [
  "Extract the invoice total.",
  "Extract the vendor name.",
  "Extract the due date.",
  "Extract the line-item count.",
];

async function classifyOne(pa: Pareta, prompt: string, document: string) {
  const resp = await pa.chat.completions.create({
    model: "auto",
    messages: [
      { role: "system", content: "You are an invoice parser." },
      { role: "user", content: `${prompt}\n\n${document}` },
    ],
    temperature: 0,
    max_tokens: 64,
  });
  return resp.choices[0].message.content;
}

const document = "INVOICE #4471 ... TOTAL $1,240.00 ...";
const pa = Pareta.fromEnv();
const answers = await Promise.all(
  PROMPTS.map((p) => classifyOne(pa, p, document)),
);
for (let i = 0; i < PROMPTS.length; i++) {
  console.log(`${PROMPTS[i]} -> ${answers[i]}`);
}
```

If any coroutine raises, `gather` propagates the first exception and the rest are
cancelled. That is rarely what you want for a batch. The next two sections fix
both halves of the problem: capacity (backpressure) and partial failure.

## Bound concurrency with a semaphore

Firing 5,000 prompts at `gather` opens as many tasks at once, overruns the
connection pool, and is the fastest way to earn a `RateLimitError` (429). An
`asyncio.Semaphore` caps how many calls are in flight at any moment. The rest
queue and drain as slots free up.

**Python**

```python
import asyncio
from pareta import AsyncPareta

MAX_IN_FLIGHT = 16

async def complete(pa, sem, messages):
    async with sem:                       # acquire a slot; release on exit
        resp = await pa.chat.completions.create(
            model="auto",
            messages=messages,
            temperature=0,
        )
        return resp.choices[0].message.content

async def run_batch(documents):
    sem = asyncio.Semaphore(MAX_IN_FLIGHT)
    async with AsyncPareta.from_env() as pa:
        tasks = [
            complete(pa, sem, [{"role": "user", "content": f"Summarize:\n{d}"}])
            for d in documents
        ]
        return await asyncio.gather(*tasks)

# 1,000 docs, but never more than 16 concurrent requests
results = asyncio.run(run_batch([f"doc {i}" for i in range(1000)]))
print(len(results))
```

**TypeScript**

JS has no built-in `asyncio.Semaphore`, so a small worker-pool does the same job:
spin up `MAX_IN_FLIGHT` workers that each pull from a shared cursor until the
queue drains. That caps in-flight calls without pulling in a dependency.

```typescript
import { Pareta } from "pareta";

const MAX_IN_FLIGHT = 16;

async function complete(pa: Pareta, messages: Array<{ role: string; content: string }>) {
  const resp = await pa.chat.completions.create({
    model: "auto",
    messages,
    temperature: 0,
  });
  return resp.choices[0].message.content;
}

async function runBatch(documents: string[]): Promise<Array<string | null>> {
  const pa = Pareta.fromEnv();
  const results: Array<string | null> = new Array(documents.length);
  let cursor = 0;
  // N workers drain a shared cursor → never more than N requests in flight.
  const worker = async () => {
    for (let i = cursor++; i < documents.length; i = cursor++) {
      results[i] = await complete(pa, [
        { role: "user", content: `Summarize:\n${documents[i]}` },
      ]);
    }
  };
  await Promise.all(
    Array.from({ length: Math.min(MAX_IN_FLIGHT, documents.length) }, worker),
  );
  return results;
}

// 1,000 docs, but never more than 16 concurrent requests
const docs = Array.from({ length: 1000 }, (_, i) => `doc ${i}`);
const results = await runBatch(docs);
console.log(results.length);
```

Pick `MAX_IN_FLIGHT` to match your traffic. 8 to 32 is a sane starting band;
tune it against the hourly p50/p95 latency buckets from
[`auto.metrics()`](cost-and-metrics.md). The SDK already retries `429`,
`503`, and `5xx` with exponential backoff (`max_retries`, default 2), so the
semaphore is your first line of defense and retries are the backstop.

## Handle errors per task

Pass `return_exceptions=True` to `gather` and every coroutine resolves to either
its result or the exception it raised, in order. The batch always completes; you
decide what to do with the failures. This is the right default for fan-out work.

**Python**

```python
import asyncio
from pareta import (
    AsyncPareta,
    ParetaError,
    InsufficientCreditsError,
    EndpointNotReadyError,
    RateLimitError,
    APITimeoutError,
)

MAX_IN_FLIGHT = 16

async def complete(pa, sem, doc):
    async with sem:
        resp = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": f"Extract the total from:\n{doc}"}],
            temperature=0,
        )
        return resp.choices[0].message.content

async def main(documents):
    sem = asyncio.Semaphore(MAX_IN_FLIGHT)
    async with AsyncPareta.from_env() as pa:
        outcomes = await asyncio.gather(
            *(complete(pa, sem, d) for d in documents),
            return_exceptions=True,
        )

    ok, failed = [], []
    for doc, outcome in zip(documents, outcomes):
        if isinstance(outcome, InsufficientCreditsError):
            # Org balance hit zero mid-batch. Nothing else will succeed —
            # stop and top up in the dashboard.
            raise outcome
        if isinstance(outcome, BaseException):
            failed.append((doc, outcome))
        else:
            ok.append((doc, outcome))

    print(f"{len(ok)} succeeded, {len(failed)} failed")
    for doc, err in failed:
        if isinstance(err, EndpointNotReadyError):
            reason = "backend warming"            # 503
        elif isinstance(err, RateLimitError):
            reason = "rate limited after retries"  # 429
        elif isinstance(err, APITimeoutError):
            reason = "timed out"
        elif isinstance(err, ParetaError):
            reason = str(err)
        else:
            reason = repr(err)
        print(f"  retry {doc!r}: {reason}")
    return ok, failed

asyncio.run(main([f"doc {i}" for i in range(50)]))
```

**TypeScript**

`Promise.allSettled` is the analog of `gather(..., return_exceptions=True)`: it
never short-circuits, and every entry resolves to `{status:"fulfilled", value}`
or `{status:"rejected", reason}`. Switch on the error class with `instanceof`.

```typescript
import {
  Pareta,
  ParetaError,
  InsufficientCreditsError,
  EndpointNotReadyError,
  RateLimitError,
  APITimeoutError,
} from "pareta";

const MAX_IN_FLIGHT = 16;

async function complete(pa: Pareta, doc: string) {
  const resp = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: `Extract the total from:\n${doc}` }],
    temperature: 0,
  });
  return resp.choices[0].message.content;
}

async function main(documents: string[]) {
  const pa = Pareta.fromEnv();
  // Worker pool bounds in-flight calls (see the semaphore section above).
  const outcomes: Array<{ ok: true; value: string | null } | { ok: false; error: unknown }> =
    new Array(documents.length);
  let cursor = 0;
  const worker = async () => {
    for (let i = cursor++; i < documents.length; i = cursor++) {
      try {
        outcomes[i] = { ok: true, value: await complete(pa, documents[i]) };
      } catch (error) {
        outcomes[i] = { ok: false, error };
      }
    }
  };
  await Promise.all(
    Array.from({ length: Math.min(MAX_IN_FLIGHT, documents.length) }, worker),
  );

  const ok: Array<[string, string | null]> = [];
  const failed: Array<[string, unknown]> = [];
  for (let i = 0; i < documents.length; i++) {
    const outcome = outcomes[i];
    if (outcome.ok) {
      ok.push([documents[i], outcome.value]);
    } else if (outcome.error instanceof InsufficientCreditsError) {
      // Org balance hit zero mid-batch. Nothing else will succeed —
      // stop and top up in the dashboard.
      throw outcome.error;
    } else {
      failed.push([documents[i], outcome.error]);
    }
  }

  console.log(`${ok.length} succeeded, ${failed.length} failed`);
  for (const [doc, err] of failed) {
    let reason: string;
    if (err instanceof EndpointNotReadyError) {
      reason = "backend warming"; // 503
    } else if (err instanceof RateLimitError) {
      reason = "rate limited after retries"; // 429
    } else if (err instanceof APITimeoutError) {
      reason = "timed out";
    } else if (err instanceof ParetaError) {
      reason = err.message;
    } else {
      reason = String(err);
    }
    console.log(`  retry ${JSON.stringify(doc)}: ${reason}`);
  }
  return { ok, failed };
}

const docs = Array.from({ length: 50 }, (_, i) => `doc ${i}`);
await main(docs);
```

Notes on the error types (all subclass `ParetaError`):

- **`InsufficientCreditsError` (402)** is fatal for the whole batch, not just one
  task. The balance is shared across the org, so once it hits zero every
  remaining call fails the same way. Stop early and top up.
- **`EndpointNotReadyError` (503)** means a serving backend behind auto is
  warming up or briefly unavailable. The SDK already retries 503s automatically;
  anything that still surfaces is safe to retry after a short wait.
- **`RateLimitError` (429)** surfaces only after the SDK exhausts its own
  retries. If you see these, lower `MAX_IN_FLIGHT`.
- **`APITimeoutError`** is raised after `max_retries`. Long generations may need a
  larger `timeout=` on the client (default is 60s, 10s connect).

Because `return_exceptions=True` never cancels siblings, you can re-run just
`failed` on the next pass.

## Streaming under concurrency

Async streaming mirrors the sync path with one twist: `create(...)` is a
coroutine, so you `await` it — and because `stream=True`, the awaited result is
an async iterator you then `async for` over. (Non-streaming `create` is awaited
too, returning the `ChatCompletion`.)

**Python**

```python
import asyncio
from pareta import AsyncPareta

async def stream_into(pa, prompt, sink):
    stream = await pa.chat.completions.create(   # await → returns the async iterator
        model="auto",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    async for chunk in stream:
        sink.append(chunk.choices[0].delta.content or "")

async def main():
    sinks = {p: [] for p in ("Summarize doc A.", "Summarize doc B.")}
    async with AsyncPareta.from_env() as pa:
        await asyncio.gather(
            *(stream_into(pa, p, sink) for p, sink in sinks.items())
        )
    for prompt, parts in sinks.items():
        print(prompt, "->", "".join(parts))

asyncio.run(main())
```

**TypeScript**

In TS, `stream: true` makes `create(...)` return an `AsyncIterable<ChatCompletionChunk>`
directly (not a `Promise` — don't `await` the call), which you drive with
`for await … of`. Run several at once with `Promise.all`, exactly like the batch.

```typescript
import { Pareta } from "pareta";

async function streamInto(pa: Pareta, prompt: string, sink: string[]) {
  const stream = pa.chat.completions.create({   // stream:true → AsyncIterable, no await
    model: "auto",
    messages: [{ role: "user", content: prompt }],
    stream: true,
  });
  for await (const chunk of stream) {
    sink.push(chunk.choices[0].delta.content || "");
  }
}

const sinks = new Map<string, string[]>([
  ["Summarize doc A.", []],
  ["Summarize doc B.", []],
]);
const pa = Pareta.fromEnv();
await Promise.all(
  [...sinks].map(([prompt, sink]) => streamInto(pa, prompt, sink)),
);
for (const [prompt, parts] of sinks) {
  console.log(prompt, "->", parts.join(""));
}
```

`chunk.choices[0].delta.content` is the incremental text. Streams end on
`[DONE]`; the SDK closes them for you. Retries only cover the initial handshake,
so a mid-stream drop raises immediately rather than silently resuming.

## Concurrent eval runs

The same pattern launches several [eval runs](../guide/evaluation.md) at once. With
`wait=True`, each `runs.create(...)` polls the run to completion using
`asyncio.sleep` under the hood, so the coroutines yield the event loop while they
wait. That makes a fan-out of `wait=True` runs genuinely concurrent.

**Python**

```python
import asyncio
from pareta import AsyncPareta

# Benchmark "auto" against frontier baselines on three of your datasets.
async def eval_one(pa, eval_set_id):
    run = await pa.evals.runs.create(
        eval_set=eval_set_id,
        models=["auto"],          # the candidate under test
        frontier="benchmarked",   # frontier models already benchmarked on this task
        wait=True,                # polls until terminal (completed/failed)
        timeout=1200.0,
    )
    return run

async def main(eval_set_ids):
    async with AsyncPareta.from_env() as pa:
        runs = await asyncio.gather(
            *(eval_one(pa, sid) for sid in eval_set_ids),
            return_exceptions=True,
        )

    for outcome in runs:
        if isinstance(outcome, BaseException):
            print("run failed to launch/finish:", outcome)
            continue
        run = outcome
        if run.status == "failed":
            print(f"{run.id}: failed — {run.error_detail}")
            continue
        print(f"{run.id}: {run.status}  cost ${run.cost}")  # Decimal dollars
        for r in run.results:
            print(f"  {r.model_id} ({r.kind}): quality={r.quality_mean}")

# eval_set_ids from earlier pa.evals.sets.create(...) calls
asyncio.run(main(["es_abc", "es_def", "es_ghi"]))
```

**TypeScript**

`runs.create({ wait: true })` returns a `Promise` that polls to terminal, so a
`Promise.allSettled` fans out several runs concurrently. The `timeout` and
`pollInterval` options are in **seconds** (matching the Python eval poller). The
billed total is `run.cost` — a fixed-2dp **string** here, not a `Decimal` — and
`run.costMicroUsd` keeps the raw micro-USD integer.

```typescript
import { Pareta } from "pareta";

// Benchmark "auto" against frontier baselines on three of your datasets.
async function evalOne(pa: Pareta, evalSetId: string) {
  return pa.evals.runs.create({
    evalSet: evalSetId,
    models: ["auto"],        // the candidate under test
    frontier: "benchmarked", // frontier models already benchmarked on this task
    wait: true,              // polls until terminal (completed/failed)
    timeout: 1200,
  });
}

async function main(evalSetIds: string[]) {
  const pa = Pareta.fromEnv();
  const runs = await Promise.allSettled(
    evalSetIds.map((sid) => evalOne(pa, sid)),
  );

  for (const outcome of runs) {
    if (outcome.status === "rejected") {
      console.log("run failed to launch/finish:", outcome.reason);
      continue;
    }
    const run = outcome.value;
    if (run.status === "failed") {
      console.log(`${run.id}: failed — ${run.errorDetail}`);
      continue;
    }
    console.log(`${run.id}: ${run.status}  cost $${run.cost}`); // dollars (string)
    for (const r of run.results) {
      console.log(`  ${r.modelId} (${r.kind}): quality=${r.qualityMean}`);
    }
  }
}

// evalSetIds from earlier pa.evals.sets.create(...) calls
await main(["es_abc", "es_def", "es_ghi"]);
```

Eval runs are metered too: the org balance is debited for the compute (`"auto"`
plus any frontier baselines), and an empty balance raises
`InsufficientCreditsError` (402). `run.cost` is the billed total as `Decimal`
dollars floored to cents; `run.cost_micro_usd` keeps the raw micro-USD integer if
you need sub-cent precision.

If you do not want to block on completion, drop `wait=True` and the call returns
immediately with a queued `EvalRun`; await `pa.evals.runs.wait(run.id)` later, or
poll `pa.evals.runs.retrieve(run.id)` yourself.

## Checklist

- One `AsyncPareta` per process, shared across coroutines. `async with` (or
  `await pa.aclose()`) to release the pool.
- `asyncio.gather(*tasks)` to fan out; `return_exceptions=True` so one failure
  does not cancel the batch.
- `asyncio.Semaphore(N)` to bound in-flight calls — your backpressure valve.
- Treat `InsufficientCreditsError` as batch-fatal; retry `EndpointNotReadyError`
  and the residual `RateLimitError` subset.
- Always `await create()`. For `stream=True` the awaited result is the async
  iterator you `async for` over; for non-streaming it is the `ChatCompletion`.

## See also

- [The client](../reference/client.md) — constructor, `from_env`, retries, timeouts
- [Chat completions](../guide/inference.md) — full inference surface and streaming
- [Core concepts](../guide/core-concepts.md) — the auto surface, metering, and billing
- [Evals](../guide/evaluation.md) — eval sets, runs, and frontier baselines
- [Errors](../guide/errors-and-retries.md) — the full `ParetaError` hierarchy



---

<!-- examples/cost-and-metrics.md -->

# Cost & quality monitoring

Every dollar you spend on Pareta runs through one org balance, and every `model="auto"` request your org sends gets rolled up for you. This page is about reading both: what a call or an eval run actually cost, how `"auto"` stacks up against the frontier baselines it replaces, and how to watch your live auto traffic — volume, success, spend, latency, and projected savings — so you catch a regression before your users do.

Two things to keep straight up front, because they shape every number below:

- **Money is metered against your org balance.** Inference (`chat.completions.create`) and evals (`evals.runs.create`) both debit the balance on success — one debit per request, no matter how many internal model calls auto's plan makes. An empty balance raises `InsufficientCreditsError` (402). The SDK never exposes balance or payment methods — top-up is browser-only, in the dashboard.
- **Models and GPUs are hidden behind `"auto"`.** You never priced a GPU-hour or picked a model; Pareta did, per request. So cost shows up as per-request debits, run totals, and an org-level rollup — and the only model ids in a cost report are `"auto"` and the frontier (vendor) ids in the clear.

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()  # reads PARETA_API_KEY (+ optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv(); // reads PARETA_API_KEY (+ optional PARETA_BASE_URL)
```

## The money convention: dollars are floored to cents

You are billed in whole cents, and the SDK **floors** to cents so it never overstates a charge. That rule shows up in two complementary fields on anything that carries a total:

- `cost: Decimal` — the billed total in dollars, floored to whole cents. A run that truly cost a third of a cent reads `Decimal("0.00")`.
- `cost_micro_usd: int` — the raw integer in micro-USD, where `1_000_000` == `$1.00`. This is the precise number for your own accounting.

**Python**

```python
run = pa.evals.runs.retrieve(run_id)

print(run.cost)            # Decimal('0.07')  — billed dollars, floored to cents
print(run.cost_micro_usd)  # 74211            — raw micro-USD (74,211 uUSD)
```

**TypeScript**

```typescript
const run = await pa.evals.runs.retrieve(runId);

console.log(run.cost);          // "0.07"  — billed dollars (string), floored to cents
console.log(run.costMicroUsd);  // 74211   — raw micro-USD (74,211 uUSD)
```

The flooring is one-directional on purpose: a sub-cent total bills as `$0.00` but keeps its true value on `cost_micro_usd`, so nothing is lost. **Per-unit rates stay in micro-USD** and are never floored — flooring a sub-cent unit rate to whole cents would erase the auto-vs-frontier comparison that the whole exercise is about. You will see this on `result.mean_cost_micro_usd` below.

## What an eval run cost

An eval run is the densest cost signal you get, because it prices `"auto"` and several frontier baselines on the same rows in one shot. The run carries the bill; each `EvalResult` carries that contender's per-item rate.

**Python**

```python
run = pa.evals.runs.create(
    task="contract-key-fields",
    items=[
        {"input": "Effective as of January 1, 2026, ...", "expected": {"effective_date": "2026-01-01"}},
        {"input": "This Agreement terminates on 2027-12-31 ...", "expected": {"termination_date": "2027-12-31"}},
    ],
    intent="extract the key dates from each contract",
    models=["auto"],                 # the contender
    frontier="benchmarked",          # baselines already benchmarked on this task
    wait=True,                       # block until the run is terminal
)

print(f"run {run.id}: {run.status}")
print(f"billed ${run.cost} ({run.cost_micro_usd} uUSD)")  # auto + frontier compute

for r in run.results:
    print(f"{r.model_id:16} {(r.kind or ''):8} "
          f"q={r.quality_mean:.3f} [{r.quality_ci_low:.3f}, {r.quality_ci_high:.3f}]  "
          f"~{r.mean_cost_micro_usd} uUSD/item  "
          f"({r.n_succeeded} ok, {r.error_count} err)")
```

**TypeScript**

```typescript
const run = await pa.evals.runs.create({
  task: "contract-key-fields",
  items: [
    { input: "Effective as of January 1, 2026, ...", expected: { effective_date: "2026-01-01" } },
    { input: "This Agreement terminates on 2027-12-31 ...", expected: { termination_date: "2027-12-31" } },
  ],
  intent: "extract the key dates from each contract",
  models: ["auto"],               // the contender
  frontier: "benchmarked",        // baselines already benchmarked on this task
  wait: true,                     // block until the run is terminal
});

console.log(`run ${run.id}: ${run.status}`);
console.log(`billed $${run.cost} (${run.costMicroUsd} uUSD)`); // auto + frontier compute

for (const r of run.results) {
  console.log(
    `${(r.modelId ?? "").padEnd(16)} ${(r.kind ?? "").padEnd(8)} ` +
      `q=${r.qualityMean?.toFixed(3)} [${r.qualityCiLow?.toFixed(3)}, ${r.qualityCiHigh?.toFixed(3)}]  ` +
      `~${r.meanCostMicroUsd} uUSD/item  ` +
      `(${r.nSucceeded} ok, ${r.errorCount} err)`,
  );
}
```

`run.cost` / `run.cost_micro_usd` is the **total** for the run, across both auto and any frontier baselines — both are metered against your balance. Each `EvalResult` reports `mean_cost_micro_usd`, the average cost per item for that contender in micro-USD. That field is the heart of a cost comparison, so it deliberately stays in raw micro-USD: a 700-uUSD frontier item and a 90-uUSD auto item both floor to `$0.00`, and the gap between them is exactly the thing you came to measure.

If the balance is empty, `create` raises `InsufficientCreditsError` (402) before any compute runs. See [Errors, retries & timeouts](../guide/errors-and-retries.md).

### Quality vs. cost, the actual trade

The point of running `"auto"` next to frontier baselines is to read both axes at once: whether quality holds, and how much money you save. Pick auto's row out by `model_id` and compare it against each baseline (`kind == "frontier"`).

**Python**

```python
run = pa.evals.runs.retrieve(run_id)

auto = next(r for r in run.results if r.model_id == "auto")
baselines = [r for r in run.results if r.kind == "frontier"]

print(f"auto             q={auto.quality_mean:.3f}  {auto.mean_cost_micro_usd} uUSD/item")
for f in sorted(baselines, key=lambda r: r.quality_mean or 0.0, reverse=True):
    line = f"{f.model_id:16} q={f.quality_mean:.3f}  {f.mean_cost_micro_usd} uUSD/item"
    if f.mean_cost_micro_usd and auto.mean_cost_micro_usd:
        # micro-USD ratio — never compute savings off the floored dollar field
        cheaper = f.mean_cost_micro_usd / auto.mean_cost_micro_usd
        dq = (auto.quality_mean or 0.0) - (f.quality_mean or 0.0)
        line += f"  (auto is {cheaper:.1f}x cheaper, dq={dq:+.3f})"
    print(line)
```

**TypeScript**

```typescript
const run = await pa.evals.runs.retrieve(runId);

const auto = run.results.find((r) => r.modelId === "auto")!;
const baselines = run.results.filter((r) => r.kind === "frontier");

console.log(`auto             q=${auto.qualityMean?.toFixed(3)}  ${auto.meanCostMicroUsd} uUSD/item`);
for (const f of [...baselines].sort((a, b) => (b.qualityMean ?? 0) - (a.qualityMean ?? 0))) {
  let line = `${(f.modelId ?? "").padEnd(16)} q=${f.qualityMean?.toFixed(3)}  ${f.meanCostMicroUsd} uUSD/item`;
  if (f.meanCostMicroUsd && auto.meanCostMicroUsd) {
    // micro-USD ratio — never compute savings off the floored dollar field
    const cheaper = f.meanCostMicroUsd / auto.meanCostMicroUsd;
    const dq = (auto.qualityMean ?? 0) - (f.qualityMean ?? 0);
    line += `  (auto is ${cheaper.toFixed(1)}x cheaper, dq=${dq >= 0 ? "+" : ""}${dq.toFixed(3)})`;
  }
  console.log(line);
}
```

Two rules when you read this:

- **Compute savings from `mean_cost_micro_usd`, never from `cost`.** The dollar field is floored to cents and a per-item rate is almost always sub-cent, so a ratio built on it would divide by zero or lie. Stay in micro-USD for any per-unit math.
- **Respect the confidence interval.** `quality_mean` comes with `quality_ci_low` / `quality_ci_high` (a 95% CI). Two contenders whose intervals overlap are not meaningfully different on this sample — add rows before you call the verdict on a hair's-width quality edge.

Full eval mechanics (building sets, frontier roster selection, document tasks, async) live in [Benchmark `"auto"` on your own data](./evaluate-on-your-data.md) and the [Evaluation guide](../guide/evaluation.md).

## What an inference call cost

Inference is OpenAI-compatible, so `chat.completions.create` returns a `ChatCompletion` with a `usage` block. Use it for token accounting; the dollar cost of that traffic lands in your org's auto rollup (next section) rather than inline on each response.

**Python**

```python
resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Extract the effective date from: ..."}],
)

u = resp.usage
print(u.prompt_tokens, u.completion_tokens, u.total_tokens)
print(resp.choices[0].message.content)
```

**TypeScript**

```typescript
const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Extract the effective date from: ..." }],
});

const u = resp.usage;
console.log(u.promptTokens, u.completionTokens, u.totalTokens);
console.log(resp.choices[0].message.content);
```

Each successful call debits your org balance — one debit per request, no matter how many internal model calls auto's plan makes. An empty balance raises `InsufficientCreditsError` (402) here too. The inference surface — streaming, kwargs pass-through, the OpenAI compatibility contract — is covered in [Running inference](../guide/inference.md).

## Monitoring your live auto traffic

Once production traffic is flowing, `auto.metrics()` is your window into it: one org-level rollup of every `model="auto"` request, covering volume, success, spend, latency, and the savings story. One call, no parameters — Python returns the raw rollup dict; TypeScript types it as `AutoMetrics`:

**Python**

```python
m = pa.auto.metrics()   # dict — the org's auto rollup

print(m["requests_30d"], "requests (30d),", m["requests_today"], "today")
print("success rate (30d):", m["success_rate_30d"])
print("billed (30d):", m["billed_micro_usd_30d"], "uUSD")
print("projected savings vs frontier (30d):", m["savings_vs_frontier_micro_usd_30d"], "uUSD")
```

**TypeScript**

```typescript
const m = await pa.auto.metrics(); // typed AutoMetrics

console.log(m.requests_30d, "requests (30d),", m.requests_today, "today");
console.log("success rate (30d):", m.success_rate_30d);
console.log("billed (30d):", m.billed_micro_usd_30d, "uUSD");
console.log("projected savings vs frontier (30d):", m.savings_vs_frontier_micro_usd_30d, "uUSD");
```

What comes back, dimension by dimension:

- **Volume + success** — `requests_30d`, `requests_today`, `success_rate_30d` (`None` with no traffic), and `days_30d`: one cell per day (`{day, n, ok, success_rate}`) over the last 30 days.
- **Spend** — `billed_micro_usd_30d` and `billed_micro_usd_today`, in raw micro-USD. The money convention holds: floor to cents yourself only when you want a billed-dollar figure.
- **Latency + errors** — `performance_hourly_7d`: hourly buckets (`{hour, requests, error_rate, p50_ms, p95_ms}`) over the last 7 days.
- **Projected savings vs frontier** — `savings_vs_frontier_micro_usd_30d` and `savings_multiple_30d`: what the same traffic would have cost at frontier list prices vs what you were billed. It is **projected** — a frontier list-priced counterfactual — and `None` when there is no traffic to project from.
- **`last_request`** — the most recent request's `created_at`, `status_code`, `duration_ms`, and billed cost; a quick liveness check.

### Watching for drift

The rollup is cheap to poll, so put it on a schedule and alert off the health fields: a `days_30d` cell whose `success_rate` dips below your bar, or a `performance_hourly_7d` bucket whose `error_rate` or `p95_ms` creeps up, is your cue to investigate before users notice.

**Python**

```python
m = pa.auto.metrics()

today = m["days_30d"][-1] if m["days_30d"] else None
if today and today["success_rate"] < 0.99:
    print(f"success slipped to {today['success_rate']:.4f} today — investigate")
```

**TypeScript**

```typescript
const m = await pa.auto.metrics();

const today = m.days_30d.at(-1);
if (today && today.success_rate < 0.99) {
  console.log(`success slipped to ${today.success_rate.toFixed(4)} today — investigate`);
}
```

### Spot-check a prompt against a frontier vendor

The rollup's savings number is a projection. For a concrete single-prompt data point, run the same messages against a frontier vendor and compare with what `"auto"` gave you:

**Python**

```python
side = pa.auto.compare_frontier(
    model="gpt-5.5",   # or gemini-3-5-flash, gemini-3-1-pro, claude-sonnet-4-6
    messages=[{"role": "user", "content": "Extract the effective date from: ..."}],
)
print(side["model"], side["cost_micro_usd"], "uUSD,", side["latency_ms"], "ms")
print(side["content"])
```

**TypeScript**

```typescript
const side = await pa.auto.compareFrontier({
  model: "gpt-5.5",   // or gemini-3-5-flash, gemini-3-1-pro, claude-sonnet-4-6
  messages: [{ role: "user", content: "Extract the effective date from: ..." }],
});
console.log(side.model, side.cost_micro_usd, "uUSD,", side.latency_ms, "ms");
console.log(side.content);
```

`compare_frontier` is **metered at the vendor's actual token cost** — one debit per call, and a failed vendor call bills $0. The allowed models are gpt-5.5, gemini-3-5-flash, gemini-3-1-pro, and claude-sonnet-4-6.

## Async

Every method here has an async twin on `AsyncPareta` with the same signatures — `auto.metrics()` and `auto.compare_frontier()` included. Pull the run and the rollup concurrently:

**Python**

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        run, m = await asyncio.gather(
            pa.evals.runs.retrieve(run_id),
            pa.auto.metrics(),
        )
        print("billed", run.cost, "/", run.cost_micro_usd, "uUSD")
        print("30d spend:", m["billed_micro_usd_30d"], "uUSD")
        print("projected savings:", m["savings_vs_frontier_micro_usd_30d"], "uUSD")

asyncio.run(main())
```

**TypeScript**

```typescript
// No AsyncPareta in TS — there's one Promise-only client, so every method is
// already async. Concurrency is just Promise.all over the awaitables.
const [run, m] = await Promise.all([
  pa.evals.runs.retrieve(runId),
  pa.auto.metrics(),
]);
console.log("billed", run.cost, "/", run.costMicroUsd, "uUSD");
console.log("30d spend:", m.billed_micro_usd_30d, "uUSD");
console.log("projected savings:", m.savings_vs_frontier_micro_usd_30d, "uUSD");
```

## Lossless access

Every response object keeps the raw server JSON. `run.to_dict()` and `result.to_dict()` give you everything the API sent, including fields not yet surfaced as typed properties. `auto.metrics()` already hands you the raw rollup (a dict in Python, `AutoMetrics` in TypeScript), so when the backend grows a new key it shows up without an SDK upgrade.

## See also

- [Benchmark `"auto"` on your own data](./evaluate-on-your-data.md) — build eval sets, pick frontier baselines, read per-contender results.
- [Concurrent & async](./concurrent-async.md) — fan-out inference and parallel eval runs.
- [Running inference](../guide/inference.md) — the OpenAI-compatible chat surface and streaming.
- [Errors, retries & timeouts](../guide/errors-and-retries.md) — `InsufficientCreditsError`, the money convention, and the exception hierarchy.



---

<!-- examples/migrate-from-openai.md -->

# Migrating from the OpenAI SDK

Pareta inference is OpenAI-compatible. If you already call `chat.completions.create(...)` through the `openai` SDK, you do not have to rewrite that code to run on Pareta. Point the OpenAI client at your Pareta base URL with a `pareta_sk_` key, set `model="auto"`, and your existing inference keeps working — with Pareta planning each request, routing it to benchmark-proven open specialists, and falling back to a frontier model when that's the right call.

This page covers two things:

1. **Keep using `openai` for inference**, the smallest possible diff: change `base_url`, `api_key`, and `model="auto"`. No deploy step, nothing to provision.
2. **Switch to the `pareta` SDK** for the things OpenAI does not do: evaluating `"auto"` against frontier baselines on your own data, reading your auto metrics, and matching intent to the task catalog.

The mental model: OpenAI gives you one client for one purpose (inference against a model you name). Pareta splits that into a data plane (inference, OpenAI-compatible, where `"auto"` names the routing brain rather than a fixed model) and a control plane (evaluate / match / monitor, which is Pareta-native). You migrate the data plane by changing three strings; you adopt the control plane when you want proof.

## The one-diff migration

The whole migration, as the diff you'd actually commit — removed lines are your OpenAI code today, added lines are the Pareta version:

**Python**

```diff
 from openai import OpenAI

-client = OpenAI(api_key="sk-...")  # talks to api.openai.com
+client = OpenAI(
+    api_key="pareta_sk_...",                 # a Pareta key, not an OpenAI key
+    base_url="https://api.pareta.ai/v1",     # note the /v1 suffix
+)
 resp = client.chat.completions.create(
-    model="gpt-4o-mini",
+    model="auto",                            # the routing brain, not a fixed model
     messages=[{"role": "user", "content": "Extract the invoice total: ..."}],
 )
 print(resp.choices[0].message.content)
```

**TypeScript**

```diff
 import OpenAI from "openai";

-const client = new OpenAI({ apiKey: "sk-..." }); // talks to api.openai.com
+const client = new OpenAI({
+  apiKey: "pareta_sk_...",                 // a Pareta key, not an OpenAI key
+  baseURL: "https://api.pareta.ai/v1",     // note the /v1 suffix
+});
 const resp = await client.chat.completions.create({
-  model: "gpt-4o-mini",
+  model: "auto",                           // the routing brain, not a fixed model
   messages: [{ role: "user", content: "Extract the invoice total: ..." }],
 });
 console.log(resp.choices[0].message.content);
```

Three things changed, nothing else:

- **`api_key`** is a `pareta_sk_...` key (mint it in the dashboard; key management is browser-only). It rides in the same `Authorization: Bearer` header the OpenAI client already sends.
- **`base_url`** is `https://api.pareta.ai/v1`. The OpenAI client appends `/chat/completions` to whatever base URL you give it, and Pareta serves the route at `/v1/chat/completions`, so the base URL must include the `/v1` suffix.
- **`model`** is the literal string `"auto"` — Pareta's routing brain, and the only model id. There is nothing to deploy or provision first, and no model to pick: "which model?" is the question Pareta answers for you, per request.

Streaming, `temperature`, `max_tokens`, `top_p`, `stop`, system messages, and the response shape (`resp.choices[0].message.content`, `resp.usage`) all behave exactly as they do against OpenAI, because the wire format is the same. Your existing response-parsing code does not change.

### Why this works

Pareta serves inference in the vLLM OpenAI-compatible format: data-only SSE for streams, the same request body, the same `ChatCompletion` / `ChatCompletionChunk` JSON shapes. The OpenAI SDK cannot tell the difference. The only Pareta-specific facts that leak through are the key prefix and the fact that `"auto"` names the routing brain rather than a hosted vendor model.

## Where the OpenAI SDK stops

The OpenAI SDK is built around calling a model you name. Pareta's reason to exist is the opposite: `"auto"` answers "which model?" for you, per request — and the SDK's job is to let you prove that routing on your own data, watch what it costs, and check what it covers. None of that has an OpenAI-SDK equivalent:

| You want to... | OpenAI SDK | Pareta SDK |
| --- | --- | --- |
| Call a model | `client.chat.completions.create(...)` | works as-is (OpenAI-compatible) |
| Benchmark `"auto"` vs frontier baselines on your data | not available | `pa.evals.runs.create(...)` |
| Find the grading contract for your eval data | not available | `pa.tasks.match(...)` |
| Watch requests, success rate, spend, projected savings | not available | `pa.auto.metrics()` |
| Run one prompt against a frontier vendor, side-by-side | not available | `pa.auto.compare_frontier(...)` |
| List callable model ids | `client.models.list()` (vendor catalog) | `pa.models.list()` (the single `"auto"` entry) |

For everything in the bottom rows, install and use the `pareta` SDK. It also speaks OpenAI-compatible inference through `pa.chat.completions.create(...)`, so once you adopt it you can drop the second `openai` client entirely and use one library for both planes.

## Switching to the `pareta` SDK

Install it and construct the client from the environment. `Pareta.from_env()` reads `PARETA_API_KEY` and the optional `PARETA_BASE_URL` (default `https://api.pareta.ai`, no `/v1` suffix, the SDK adds route prefixes itself):

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()  # reads PARETA_API_KEY (+ optional PARETA_BASE_URL)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv(); // reads PARETA_API_KEY (+ optional PARETA_BASE_URL)
```

The Python client is a context manager, which releases the HTTP connection cleanly:

**Python**

```python
with Pareta.from_env() as pa:
    resp = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Hello"}],
    )
    print(resp.choices[0].message.content)
```

**TypeScript**

```typescript
// No context manager in TS — there is no owned connection to close; just construct
// the client and await the call.
const pa = Pareta.fromEnv();
const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Hello" }],
});
console.log(resp.choices[0].message.content);
```

### Inference looks the same, with one rename

The OpenAI call maps one-to-one onto the Pareta call. The arguments and the response shape are identical:

**Python**

```python
# OpenAI:
resp = openai_client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "..."}],
    temperature=0,
    max_tokens=512,
)

# Pareta:
resp = pa.chat.completions.create(
    model="auto",              # the routing brain instead of a vendor model name
    messages=[{"role": "user", "content": "..."}],
    temperature=0,             # extra OpenAI params pass straight through
    max_tokens=512,
)

choice = resp.choices[0]
print(choice.message.content)
print(choice.finish_reason)       # "stop", "length", ...
print(resp.usage.total_tokens)    # prompt_tokens + completion_tokens
```

**TypeScript**

```typescript
// OpenAI:
const resp = await openaiClient.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "..." }],
  temperature: 0,
  max_tokens: 512,
});

// Pareta:
const resp = await pa.chat.completions.create({
  model: "auto",              // the routing brain instead of a vendor model name
  messages: [{ role: "user", content: "..." }],
  temperature: 0,             // extra OpenAI params pass straight through
  max_tokens: 512,
});

const choice = resp.choices[0];
console.log(choice.message.content);
console.log(choice.finishReason);      // "stop", "length", ...
console.log(resp.usage.totalTokens);   // promptTokens + completionTokens
```

`model` and `messages` are both required; passing either falsy raises `ValueError` before any request goes out. Any extra OpenAI keyword argument (`temperature`, `max_tokens`, `top_p`, `stop`, `frequency_penalty`, ...) is forwarded verbatim as a request-body field.

Streaming is the same shape as OpenAI too. `stream=True` returns an iterator of `ChatCompletionChunk`, and the incremental text is on `chunk.choices[0].delta.content`:

**Python**

```python
for chunk in pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize this clause: ..."}],
    stream=True,
):
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()
```

**TypeScript**

```typescript
const stream = pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Summarize this clause: ..." }],
  stream: true,
});
for await (const chunk of stream) {
  const delta = chunk.choices[0].delta.content;
  if (delta) process.stdout.write(delta);
}
console.log();
```

See [Streaming chat completions](./streaming-chat.md) for the full streaming details and the async variant.

### Listing models means something different

In the OpenAI SDK, `client.models.list()` returns the vendor's hosted catalog — the menu you pick from. In Pareta there is no menu: `pa.models.list()` returns exactly one entry, `"auto"`, because which model serves a request is decided per request, behind that id. The call exists so OpenAI-style tooling that discovers ids by listing keeps working:

**Python**

```python
for m in pa.models.list():
    print(m.id, m.owned_by)   # auto pareta — m.id is callable as `model=...`
```

**TypeScript**

```typescript
const models = await pa.models.list();
for (const m of models) {
  console.log(m.id, m.ownedBy); // auto pareta — m.id is callable as `model: ...`
}
```

## Three platform facts that have no OpenAI equivalent

These are the differences that matter once you are past the inference call. They are not gotchas; they are the point of the platform.

### 1. There is no model picker, and GPUs are hidden

There is no "pick gpt-4o" step — and no Pareta equivalent of one. Every request goes to `"auto"`, and Pareta plans it, routes each part to benchmark-proven open specialists, verifies checkable outputs, and falls back to a frontier model when that's the right call. Nothing to deploy, no hardware knob anywhere in the API: no GPU, tensor-parallel, quantization, or run-mode setting. Serving is Pareta's problem.

To see *what* auto routes across, browse the task catalog with `pa.tasks.list()` or map a sentence of intent onto it with `pa.tasks.match(...)` — see [Discovery](#discovery-checking-what-auto-covers) below.

### 2. Open-weights models stay behind `"auto"`

OpenAI model names are global and stable (`gpt-4o-mini`). Pareta's open-weights models never cross into the SDK at all — no HuggingFace repo ids, no model roster to browse. The only place model names appear in the clear is the frontier (vendor) side: eval baselines and `pa.auto.compare_frontier(model=...)` take public vendor ids (`gpt-5.5`, `claude-sonnet-4-6`, ...), because those are public names. Everything open-weights is a routing decision behind the one id you already pass.

### 3. Inference and evals are metered against your org balance

OpenAI bills the account behind the key out of band; you never see a price on the response. On Pareta, the same key debits a shared **org balance**, and the eval path surfaces the cost back to you in dollars.

- **Inference debits on success — one debit per request.** Each completed `chat.completions.create()` call debits the org balance once, no matter how many internal model calls auto's plan makes; orchestration overhead is Pareta's cost, not yours. Cost is metered server-side, not returned on the completion object.
- **Evals debit for auto + frontier compute**, and the run reports its spend: `run.cost` is a `Decimal` in dollars (floored to whole cents per Pareta's billing convention, for example 5 micro-USD reads `Decimal("0.00")`), with the raw integer on `run.cost_micro_usd`. A FAILED run is not charged.
- **A zero balance raises `InsufficientCreditsError` (402)** on both the inference and eval paths.
- **Top-up is browser-only.** The SDK consumes credit; it never exposes balance, payment methods, or a way to add funds. There is no API call for it.

**Python**

```python
from pareta import InsufficientCreditsError

try:
    resp = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "..."}],
    )
except InsufficientCreditsError:
    print("Org balance is empty. Top up in the dashboard, then retry.")
```

**TypeScript**

```typescript
import { Pareta, InsufficientCreditsError } from "pareta";

try {
  const resp = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "..." }],
  });
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    console.log("Org balance is empty. Top up in the dashboard, then retry.");
  } else {
    throw e;
  }
}
```

Note that this error reaches the OpenAI client too: if you stayed on the one-diff `openai`-SDK path, a 402 surfaces there as an OpenAI status error rather than as `pareta.InsufficientCreditsError`. Mapping it to a typed exception is one more reason to adopt the `pareta` SDK.

## Evaluate before you commit (the OpenAI SDK can't do this)

The biggest reason to reach for the `pareta` SDK rather than the bare `openai` client: before you trust the routing in production, run your own data through `"auto"` and frontier baselines and read back per-contender quality and cost. There is no OpenAI-SDK analog.

**Python**

```python
from pareta import Pareta

pa = Pareta.from_env()

run = pa.evals.runs.create(
    task="contract-key-fields",
    items=[
        {"input": "...", "expected": "..."},
        {"input": "...", "expected": "..."},
    ],
    intent="extract the key fields from each contract",
    models=["auto"],          # the candidate you ship
    frontier="benchmarked",   # frontier models benchmarked on this task, as baselines
    wait=True,                # poll until terminal, then return
)

print(run.status)            # "completed" or "failed"
print(run.cost)              # Decimal dollars (floored to cents)

for r in run.results:
    print(r.model_id, r.kind, r.quality_mean, r.mean_cost_micro_usd)
```

**TypeScript**

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

const run = await pa.evals.runs.create({
  task: "contract-key-fields",
  items: [
    { input: "...", expected: "..." },
    { input: "...", expected: "..." },
  ],
  intent: "extract the key fields from each contract",
  models: ["auto"],          // the candidate you ship
  frontier: "benchmarked",   // frontier models benchmarked on this task, as baselines
  wait: true,                // poll until terminal, then return
});

console.log(run.status);     // "completed" or "failed"
console.log(run.cost);       // dollar string, floored to cents ("1.23")

for (const r of run.results) {
  console.log(r.modelId, r.kind, r.qualityMean, r.meanCostMicroUsd);
}
```

`frontier=` accepts `None`/`"none"` (no baselines), `"all"` (every frontier model for the task), `"benchmarked"` (the frontier models with a benchmark score on the task), or an explicit list of frontier ids. You can pull the roster with `pa.evals.frontier_models(task="contract-key-fields")` to see what is available, including which entries are `vision`-capable and which are `benchmarked` on that task. See [Evaluate on your data](./evaluate-on-your-data.md) for eval sets, document uploads, and the async path.

## Discovery: checking what auto covers

Benchmarking on your own data needs a grading contract, and `tasks.match(...)` finds it from a plain-English description of your dataset — the `task` an eval run validates rows against and scores with. Again, no OpenAI equivalent:

**Python**

```python
match = pa.tasks.match("pull key fields out of vendor contracts", top_k=5)
if match.type == "task" and match.chosen:
    task_id = match.chosen.task_id          # a benchmarked task, e.g. for evals.runs.create(task=...)
    print("best task:", task_id, "confidence:", match.confidence)
```

**TypeScript**

```typescript
const match = await pa.tasks.match("pull key fields out of vendor contracts", { topK: 5 });
if (match.matched && match.chosen?.taskId) {
  const taskId = match.chosen.taskId;       // a benchmarked task, e.g. for evals.runs.create({ task })
  console.log("best task:", taskId, "confidence:", match.chosen.confidence);
}
```

`match.type` (Python) is one of `"task"` (a benchmarked task fit), `"capability"` (a general lane like chat or coding), `"unsupported"` (a correct "no", not an error), or `"none"`. Browse the full catalog with `pa.tasks.list()` or one task with `pa.tasks.retrieve(task_id)`. `match()` raises `ValueError` on an empty query.

## Errors: from OpenAI exceptions to Pareta exceptions

If you keep the `openai` client, you keep OpenAI's exception types. If you adopt the `pareta` SDK, errors become Pareta exceptions, all subclasses of `ParetaError`, mapped per HTTP status:

**Python**

```python
from pareta import (
    ParetaError,               # base class
    AuthenticationError,       # 401 - bad or missing key
    InsufficientCreditsError,  # 402 - org out of credit; top up in the dashboard
    PermissionDeniedError,     # 403
    NotFoundError,             # 404 - unknown task or resource id
    RateLimitError,            # 429 - throttled (auto-retried)
    EndpointNotReadyError,     # 503 - a serving backend is warming (auto-retried)
    BadRequestError,           # 400/422 - malformed request
)
```

**TypeScript**

```typescript
import {
  ParetaError,              // base class
  AuthenticationError,      // 401 - bad or missing key
  InsufficientCreditsError, // 402 - org out of credit; top up in the dashboard
  PermissionDeniedError,    // 403
  NotFoundError,            // 404 - unknown task or resource id
  RateLimitError,           // 429 - throttled (auto-retried)
  EndpointNotReadyError,    // 503 - a serving backend is warming (auto-retried)
  BadRequestError,          // 400/422 - malformed request
} from "pareta";
```

The rough correspondence to OpenAI: `AuthenticationError` ↔ 401, `RateLimitError` ↔ 429, `BadRequestError` ↔ 400/422, `NotFoundError` ↔ 404. The two without OpenAI analogs are `InsufficientCreditsError` (402, the org-balance gate above) and `EndpointNotReadyError` (503, a serving backend behind auto is warming or briefly unavailable; the client retries 503s automatically, so if it surfaces, wait briefly and retry the call). The client auto-retries 429s and transient 5xx/timeouts with exponential backoff (`max_retries`, default 2).

## Async

`AsyncPareta` mirrors the sync client exactly, same arguments, same response shapes, with `async def` methods and async iterators for streams:

**Python**

```python
import asyncio
from pareta import AsyncPareta


async def main():
    async with AsyncPareta.from_env() as pa:
        resp = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Extract the total: ..."}],
        )
        print(resp.choices[0].message.content)

        # Streaming: await create() once to get the async iterator, then `async for`.
        stream = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Summarize: ..."}],
            stream=True,
        )
        async for chunk in stream:
            print(chunk.choices[0].delta.content or "", end="")
        print()


asyncio.run(main())
```

**TypeScript**

```typescript
// There is no AsyncPareta in TS — the one `Pareta` client is already async.
// Every I/O method returns a Promise (await it); streaming returns an
// AsyncIterable (for await … of it). The sync/async split simply doesn't exist.
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Extract the total: ..." }],
});
console.log(resp.choices[0].message.content);

// Streaming: create({ stream: true }) returns the async iterator directly.
const stream = pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Summarize: ..." }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0].delta.content || "");
}
console.log();
```

This is the same async shape the `openai` SDK uses (`AsyncOpenAI`, `await create(...)`, `async for chunk`), so async migrations are as small as the sync ones. In TypeScript there is no second client at all — `Pareta` is Promise-only, so there is nothing to migrate between sync and async.

## Migration checklist

- [ ] Mint a `pareta_sk_` key in the dashboard.
- [ ] **Staying on `openai`?** Set `base_url="https://api.pareta.ai/v1"`, `api_key="pareta_sk_..."`, and `model="auto"`. Done — nothing to deploy.
- [ ] **Adopting `pareta`?** Swap `OpenAI(...)` for `Pareta.from_env()` and `client.chat...` for `pa.chat...`. Inference args and response shapes are unchanged.
- [ ] Benchmark it: run your own data through `pa.evals` with `models=["auto"]` and a frontier baseline before you commit.
- [ ] Map your error handling: 402 becomes `InsufficientCreditsError`, 503 becomes `EndpointNotReadyError`.
- [ ] Keep your org balance funded (top-up is browser-only); a zero balance stops both inference and evals.
- [ ] (Optional) Watch the routing pay for itself: `pa.auto.metrics()` — requests, success rate, spend, projected savings vs frontier.

## Next steps

- [Evaluate on your data](./evaluate-on-your-data.md), the proof step: `"auto"` vs frontier baselines on your own rows.
- [Streaming chat completions](./streaming-chat.md), the full streaming and async story.
- [Cost & quality monitoring](./cost-and-metrics.md), watch your `"auto"` traffic with `auto.metrics()`.



---

<!-- reference/client.md -->

# Client (`Pareta`, `AsyncPareta`)

The client is the one object you build and the only thing that talks to the network. It holds your API key, the environment URL, the timeout and retry policy, and an HTTP connection pool. Every call you make goes through it: running `model="auto"` inference, browsing the catalog, evaluating auto against the frontier. There are two of them and they are mirror images: `Pareta` is synchronous, `AsyncPareta` is `async`/`await`. Pick one, build it once, reuse it.

```python
from pareta import Pareta

with Pareta.from_env() as pa:                 # reads PARETA_API_KEY
    print(pa.models.list())
```

Nothing else in the SDK is constructed directly. Resources like `pa.chat`, `pa.tasks`, and `pa.evals` are attributes that hang off the client; you never instantiate them yourself.

## Build it from the environment

`from_env()` is the recommended constructor. It reads `PARETA_API_KEY` and an optional `PARETA_BASE_URL`, then builds the client for you. It keeps `pareta_sk_…` secrets out of source and lets the same code run against production or staging by flipping one environment variable.

```bash
export PARETA_API_KEY="pareta_sk_live_…"
```

```python
from pareta import Pareta, AsyncPareta

pa = Pareta.from_env()             # sync
apa = AsyncPareta.from_env()       # async — same call, async client
```

```python
@classmethod
Pareta.from_env(**kwargs) -> Pareta
AsyncPareta.from_env(**kwargs) -> AsyncPareta
```

`from_env()` forwards any extra keyword arguments straight to the constructor, so you can keep the key in the environment and still override the rest in code:

```python
pa = Pareta.from_env(max_retries=5, timeout=120.0)
```

An explicit `api_key=` or `base_url=` passed to `from_env()` wins over the environment variable of the same name.

## Construct it directly

When you are not driving config from the environment, call the constructor. Both clients take the same arguments; they differ only in the type of `http_client`.

```python
from pareta import Pareta

pa = Pareta(
    api_key="pareta_sk_live_…",
    base_url="https://api.pareta.ai",
    timeout=60.0,
    max_retries=2,
    http_client=None,
)
```

```python
Pareta(
    api_key: str | None = None,
    base_url: str | None = None,
    timeout=None,
    max_retries: int = 2,            # DEFAULT_MAX_RETRIES
    http_client: httpx.Client | None = None,
)

AsyncPareta(
    api_key: str | None = None,
    base_url: str | None = None,
    timeout=None,
    max_retries: int = 2,
    http_client: httpx.AsyncClient | None = None,
)
```

| Parameter | Type | Default | What it does |
|-----------|------|---------|--------------|
| `api_key` | `str \| None` | `None` | Your `pareta_sk_…` key. Sent as `Authorization: Bearer <key>`. Required (raises `ParetaError` if missing). |
| `base_url` | `str \| None` | `"https://api.pareta.ai"` | API root. Normalized with `rstrip("/")`. Pass the staging URL to point at staging. |
| `timeout` | `httpx.Timeout \| float \| None` | `httpx.Timeout(60.0, connect=10.0)` | Per-request HTTP timeout. |
| `max_retries` | `int` | `2` | Automatic retries on transient failures. Clamped to `>= 0`. |
| `http_client` | `httpx.Client \| httpx.AsyncClient \| None` | `None` | Bring your own httpx client (proxies, custom transports, pools). |

### `api_key`

The key is the one piece of config you cannot skip. The SDK sends it as a Bearer token on every request. Mint keys in the dashboard; key management is browser-only and the SDK only ever consumes a key.

If the key is falsy (and `PARETA_API_KEY` is unset when using `from_env()`), the constructor raises `ParetaError` before any network call:

```python
from pareta import Pareta, ParetaError

try:
    pa = Pareta(api_key="")
except ParetaError as e:
    print(e)
    # missing API key. Pass api_key=… or set PARETA_API_KEY
    # (mint a pareta_sk_ key in the dashboard).
```

A key that is present but rejected by the server surfaces as `AuthenticationError` (401) on the first request, not at construction time.

### `base_url`

`base_url` selects the environment. It defaults to production and is normalized with a trailing-slash strip, so `https://api.pareta.ai/` and `https://api.pareta.ai` behave identically. Keys are environment-scoped: pair each `base_url` with a key minted for that environment.

```python
prod    = Pareta(api_key="pareta_sk_live_…")                                  # default
staging = Pareta(api_key="pareta_sk_test_…", base_url="https://api-staging.pareta.ai")
```

### `timeout`

Caps how long a single request may take. The default `httpx.Timeout(60.0, connect=10.0)` allows up to 10 seconds to connect and 60 seconds overall. A bare float sets the overall timeout for read, write, and connect alike. Raise it for long completions, or stream the response so tokens arrive incrementally (see [Inference](../guide/inference.md)). Note that `evals.runs.create(..., wait=True)` has its own `timeout` argument governing the poll loop, separate from this HTTP timeout (see [Evals](../guide/evaluation.md)).

```python
import httpx
from pareta import Pareta

pa = Pareta(api_key="pareta_sk_live_…", timeout=httpx.Timeout(120.0, connect=10.0))
```

### `max_retries`

The SDK automatically retries transient failures: HTTP `408, 409, 429, 500, 502, 503, 504`. The default is `2` (up to 3 attempts). Backoff is exponential with jitter, capped at 8 seconds, and honors a server `Retry-After` header when present. Non-transient errors (`401`, `402`, `404`, and so on) raise on the first attempt. Once a stream's bytes are flowing, a mid-stream drop raises immediately and is not retried. See [Errors and retries](../guide/errors-and-retries.md).

```python
pa = Pareta(api_key="pareta_sk_live_…", max_retries=5)   # patient batch job
pa = Pareta(api_key="pareta_sk_live_…", max_retries=0)   # fail fast (tests)
```

### `http_client`

By default the client constructs its own httpx client (configured with your `timeout`) and closes it for you. Pass `http_client=` to control the transport layer: an outbound proxy, mTLS, shared connection pools, or test doubles.

```python
import httpx
from pareta import Pareta

my_client = httpx.Client(
    proxy="http://proxy.internal:8080",
    limits=httpx.Limits(max_connections=50, max_keepalive_connections=10),
    timeout=httpx.Timeout(120.0, connect=10.0),
)
pa = Pareta(api_key="pareta_sk_live_…", http_client=my_client)
```

When you inject a client, you own its lifecycle and its timeout. `pa.close()` will not close a client you passed in, and the constructor's `timeout=` applies only to an SDK-owned client. Set the timeout on your own client, and close it yourself.

## Lifecycle and cleanup

Each client owns an HTTP connection pool. Release it when you are done. The cleanly idiomatic way is the context manager, which closes the pool on exit.

### Sync

```python
close() -> None          # close the HTTP client (only if the SDK owns it)
__enter__() -> Pareta
__exit__(*exc) -> None
```

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    completion = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Extract the parties."}],
    )
    print(completion.choices[0].message.content)
# HTTP client closed on exit
```

Or close it explicitly:

```python
pa = Pareta.from_env()
try:
    pa.models.list()
finally:
    pa.close()
```

### Async

```python
async aclose() -> None
async __aenter__() -> AsyncPareta
async __aexit__(*exc) -> None
```

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        models = await pa.models.list()
        print(models)
    # HTTP client closed on exit

asyncio.run(main())
```

The ownership rule holds in both: if you passed `http_client=`, neither `close()`/`aclose()` nor exiting the context manager touches it. Close your own client.

## Resource namespaces

The client is a namespace router. Every capability hangs off it as an attribute. The sync client exposes the sync resources; the async client exposes the async mirrors. The method shapes match one-to-one, async methods are `async def`, and streaming methods return async iterators on the async client.

| Namespace | Sync type | Async type | What it does | Reference |
|-----------|-----------|------------|--------------|-----------|
| `chat` | `Chat` | `AsyncChat` | OpenAI-compatible inference via `chat.completions.create(model="auto", ...)`. Metered. | [chat](./chat.md) |
| `models` | `Models` | `AsyncModels` | `models.list()` — the OpenAI-compatible model listing: exactly one entry, `"auto"`. | [models](./models.md) |
| `tasks` | `Tasks` | `AsyncTasks` | The grading-contract directory for evals: `list`, `retrieve`, `match`. | [tasks](./tasks.md) |
| `evals` | `Evals` | `AsyncEvals` | `evals.sets`, `evals.runs`, and `evals.frontier_models(...)`. Metered. | [evals](./evals.md) |
| `audio` | `Audio` | `AsyncAudio` | Speech: `audio.transcriptions(...)` (ASR) and `audio.speech(...)` (TTS). Metered per minute. | [audio](./audio.md) |
| `rerank` | callable | callable | `pa.rerank(query, documents, top_n=...)` — calibrated document reranking. Metered per document. | [rerank](./rerank.md) |
| `embeddings` | callable | callable | `pa.embeddings(texts, input_type=...)` — unit-normalized vectors. Metered per input token. | [embeddings](./embeddings.md) |
| `auto` | `Auto` | `AsyncAuto` | `auto.metrics()` — your org's auto-traffic rollup — and `auto.compare_frontier(...)`, a metered frontier side-by-side. | [quickstart](../guide/quickstart.md) |

The TypeScript client mirrors all of them — `chat`, `models`, `tasks`, `evals`, `audio`, `auto` as camelCase namespaces, `rerank`/`embeddings` as the same callable fields — on one Promise-only `Pareta` (full capability parity as of `pareta` 1.2.0 on npm).

A tour of the core namespaces against one client:

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    # tasks — which grading contract scores my dataset?
    match = pa.tasks.match("extract key fields from contracts")
    print(match.type, match.chosen.task_id if match.chosen else None)

    # chat — OpenAI-compatible inference; "auto" is the only model id
    resp = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Say hello."}],
    )
    print(resp.choices[0].message.content)

    # models — the OpenAI-compatible list: exactly one entry, "auto"
    for m in pa.models.list():
        print(m.id, m.owned_by)

    # evals — benchmark "auto" against frontier baselines on your own data
    run = pa.evals.runs.create(
        task="contract-key-fields",
        items=[{"input": "…", "expected": "…"}],
        models=["auto"],
        frontier="benchmarked",
        wait=True,
    )
    print("run cost:", run.cost)        # Decimal dollars, floored to cents

    # auto — the org-level rollup of your routed traffic
    metrics = pa.auto.metrics()
```

The same code on the async client, with `await` and the async context manager:

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        resp = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Say hello."}],
        )
        print(resp.choices[0].message.content)

asyncio.run(main())
```

See [Async](../guide/async.md) for the full sync-vs-async mapping.

## Platform truths the client makes concrete

These hold no matter how you build the client. They are why there is no GPU knob, no balance API, and no model picker in the SDK.

- **Models and GPUs are hidden.** You configure a key, a URL, timeouts, and retries — never hardware, and never a model pick. `"auto"` is the only model id; Pareta plans each request and resolves the models, GPUs, tensor-parallelism, and quantization behind it. There is no hardware parameter anywhere in the SDK.
- **Frontier ids are in the clear; open models stay behind `"auto"`.** The vendor ids you read — in `evals.frontier_models()`, on frontier rows of `run.results`, in `auto.compare_frontier(model=…)` — are public products. The open specialists auto routes to never surface as ids you pass or read.
- **Inference and evals are metered against your org balance.** A successful `pa.chat.completions.create()` debits your balance — one debit per request, no matter how many internal model calls auto's plan makes; `pa.evals.runs.create()` debits for both auto and frontier compute. An `EvalRun` reports its billed total on `run.cost` (a `Decimal` in dollars, floored to whole cents, so a sub-cent run reads `Decimal("0.00")`) and the raw value on `run.cost_micro_usd`. When the balance hits zero, both paths raise `InsufficientCreditsError` (402). Top-up is browser-only; the SDK exposes neither balance nor payment methods.

  ```python
  from pareta import InsufficientCreditsError

  try:
      pa.chat.completions.create(model="auto", messages=[{"role": "user", "content": "ping"}])
  except InsufficientCreditsError:
      print("Out of credit — top up in the dashboard.")
  ```

- **Inference is OpenAI-compatible.** `base_url` plus your `pareta_sk_…` key is a drop-in OpenAI endpoint. You can point the `openai` SDK at the same `base_url` and send `model="auto"`; this SDK adds the control plane (evals, catalog match, auto metrics) the `openai` client cannot do. See [Inference](../guide/inference.md).

## See also

- [Configuration](../guide/configuration.md) — the full configuration guide: `from_env`, `base_url`, timeouts, retries, custom transports, and the configuration cookbook.
- [Inference](../guide/inference.md) — `chat.completions.create(model="auto", ...)`, streaming, and metering.
- [tasks](./tasks.md) — the grading contracts evals score against; `match` finds the right one for your dataset.
- [Evaluation](../guide/evaluation.md) — benchmark `"auto"` on your own data, including `run.cost`.
- [Errors and retries](../guide/errors-and-retries.md) — the `ParetaError` hierarchy and retry behavior.
- [Async](../guide/async.md) — the sync-vs-async mapping for every resource.



---

<!-- reference/chat.md -->

# chat.completions

Run inference on Pareta. `chat.completions.create(...)` is the one call you make to get tokens out. It has the same shape as the OpenAI chat completions API: pass `model="auto"`, a list of `messages`, and you get a `ChatCompletion` back. Set `stream=True` and you get an iterator of token deltas instead.

Inference on Pareta is OpenAI-compatible on the wire (vLLM-style SSE), so this exact surface works whether you call it through this SDK, the `openai` package, or raw HTTP. This SDK's added value is the control plane around it (evals, catalog match, auto metrics); for plain inference the clients are interchangeable.

Two platform truths shape this page:

- **There is no model to pick, and GPUs are hidden.** The `model` you pass is the literal string `"auto"` — Pareta plans each request, routes it to benchmark-proven open specialists, verifies, and falls back to a frontier model when that's the right call. Open-weights model ids never cross to you, and you never pick a GPU, quantization, or tensor-parallel setting. The backend resolves all of that.
- **Inference is metered against your org balance.** A successful completion debits your balance in dollars — one debit per request, no matter how many internal model calls auto's plan makes. If the balance is empty, the call raises [`InsufficientCreditsError`](exceptions.md) (402). Top-up is browser-only; the SDK exposes no balance or payment surface.

**Route:** `POST /v1/chat/completions`

## Signature

```python
class Completions:
    def create(
        self,
        *,
        model: str,
        messages: list[dict[str, Any]],
        stream: bool = False,
        **kwargs: Any,
    ) -> ChatCompletion | Iterator[ChatCompletionChunk]
```

All arguments are keyword-only.

| Parameter | Type | Default | Notes |
|-----------|------|---------|-------|
| `model` | `str` | required | The literal string `"auto"` — the routing brain, and the only model id. Validated server-side at call time. |
| `messages` | `list[dict]` | required | Non-empty list of OpenAI-format message dicts (`{"role": ..., "content": ...}`). |
| `stream` | `bool` | `False` | `False` returns a `ChatCompletion`; `True` returns an iterator of `ChatCompletionChunk`. |
| `**kwargs` | `Any` | — | Any extra OpenAI body field (`temperature`, `max_tokens`, `top_p`, `stop`, `seed`, ...) passes through unchanged. |

`model` and `messages` are both required. The SDK raises `ValueError` before sending if `model` is falsy or `messages` is empty, so a malformed call fails fast without burning a request or a charge.

## Basic completion

```python
from pareta import Pareta

with Pareta.from_env() as pa:   # reads PARETA_API_KEY (+ optional PARETA_BASE_URL)
    resp = pa.chat.completions.create(
        model="auto",                # the routing brain — the only model id
        messages=[
            {"role": "system", "content": "You extract structured fields from documents."},
            {"role": "user", "content": "What is the invoice total?\n\nINVOICE\nTotal due: $4,210.00"},
        ],
    )

    print(resp.choices[0].message.content)
    print(resp.usage.total_tokens, "tokens")
```

`Pareta.from_env()` is the recommended constructor; it reads `PARETA_API_KEY` and the optional `PARETA_BASE_URL`. You can also pass the key explicitly with `Pareta(api_key="pareta_sk_...")`. The client is a context manager, so `with` releases the HTTP connection for you.

### Where `model` comes from

One place: `model` is always the literal string `"auto"`. Pareta plans the
request, routes it to benchmark-proven open specialists, verifies, and falls
back to a frontier model when that's the right call. Nothing to deploy, no
menu to pick from — [`models.list()`](./models.md) confirms it by returning
the single `"auto"` entry.

## Return type: ChatCompletion

With `stream=False` (the default), `create(...)` returns a `ChatCompletion`. Fields mirror OpenAI:

```python
resp.id                            # str | None
resp.model                         # str | None — echoes "auto"
resp.created                       # int | None — Unix timestamp
resp.choices                       # list[Choice]
resp.choices[0].index              # int | None
resp.choices[0].finish_reason      # str | None — "stop", "length", ...
resp.choices[0].message.role       # str | None — "assistant"
resp.choices[0].message.content    # str | None — the generated text
resp.usage.prompt_tokens           # int | None
resp.usage.completion_tokens       # int | None
resp.usage.total_tokens            # int | None
```

Every response object keeps the untouched server JSON. If a field is not surfaced as a typed property, reach it with `resp.to_dict()` or `resp["..."]`. Nothing the API returns is lost behind the typed layer.

## Passthrough parameters

Any extra keyword goes straight into the request body, so the full OpenAI parameter set is available without the SDK enumerating it:

```python
resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize this contract clause: ..."}],
    temperature=0.2,
    max_tokens=512,
    top_p=0.9,
    stop=["\n\n"],
    seed=7,
)
```

These fields are not validated SDK-side; they are forwarded as-is and validated by the serving model. An unsupported field comes back as a `BadRequestError` (400/422).

## Streaming

Set `stream=True` and `create(...)` returns an `Iterator[ChatCompletionChunk]` instead of a single `ChatCompletion`. Each chunk carries a `delta` (not a `message`); the incremental text is at `chunk.choices[0].delta.content`.

```python
with Pareta.from_env() as pa:
    stream = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Draft a one-paragraph status update."}],
        stream=True,
    )
    for chunk in stream:
        print(chunk.choices[0].delta.content or "", end="", flush=True)
    print()
```

A chunk has the same schema as a `ChatCompletion`. `ChatCompletionChunk` exists as a distinct type only for hinting:

```python
chunk.choices[0].delta.content    # str | None — the new text in this chunk
chunk.choices[0].delta.role       # str | None — usually only set on the first chunk
chunk.choices[0].finish_reason    # str | None — "stop" / "length" on the last chunk
chunk.id                          # str | None
chunk.model                       # str | None
```

Guard `delta.content` with `or ""` (or an `if delta:` check): the opening role chunk and the final `finish_reason` chunk carry no text, so `delta.content` is `None` there. To accumulate the full text:

```python
text = "".join(c.choices[0].delta.content or "" for c in stream)
```

The stream is data-only SSE and always terminates on a `[DONE]` sentinel, which the SDK consumes for you, so the iterator simply ends and a plain `for` loop exits cleanly.

**Mid-stream behavior:** retries (see below) cover only the initial connect and status-line handshake. Once tokens are flowing, a mid-stream connection drop raises immediately rather than silently resuming. Nothing is sent until you start iterating, and the connection stays open for the life of the loop.

## Async

`AsyncPareta` mirrors the sync client. `create(...)` is `async def`. For streaming you `await` the call once, then `async for` over the chunks.

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        # Non-streaming: await returns a ChatCompletion
        resp = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "What is the invoice total?"}],
        )
        print(resp.choices[0].message.content)

        # Streaming: await once, then async-for the chunks
        stream = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Stream me a haiku about ledgers."}],
            stream=True,
        )
        async for chunk in stream:
            print(chunk.choices[0].delta.content or "", end="", flush=True)
        print()

asyncio.run(main())
```

The async return type is `ChatCompletion | AsyncIterator[ChatCompletionChunk]`. The async client also exposes `aclose()` and works as an `async with` context manager.

## Metering

Every successful completion (streaming or not) debits your org balance — one debit per request, however many internal model calls auto's plan makes; orchestration overhead is Pareta's cost, not yours. The `chat.completions` surface does not return a per-call cost field; spend is rolled up org-wide via `pa.auto.metrics()` (see [Cost & quality monitoring](../examples/cost-and-metrics.md)). If the org balance is empty, the call raises `InsufficientCreditsError` (402, a subclass of `ParetaError`). Top up in the dashboard; billing is browser-only and the SDK has no balance or payment surface.

## Errors

`create(...)` raises specific subclasses of `ParetaError`. A single `except ParetaError` is a fine catch-all; the named classes let you branch. The two cases specific to running inference are an empty balance and a warming backend:

```python
from pareta import (
    Pareta,
    InsufficientCreditsError,   # 402: org balance empty
    EndpointNotReadyError,      # 503: a serving backend is warming / briefly unavailable
)

with Pareta.from_env() as pa:
    try:
        resp = pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Hello"}],
        )
        print(resp.choices[0].message.content)
    except InsufficientCreditsError:
        # Balance hit zero. Top up in the dashboard (billing is browser-only).
        print("Out of credit — top up in the dashboard, then retry.")
    except EndpointNotReadyError:
        # A serving backend behind auto is warming. The SDK already retried
        # the 503 with backoff; wait briefly and retry the call.
        print("Backend warming — retry shortly.")
```

| Raised | Status | When |
|--------|--------|------|
| `ValueError` | — | `model` falsy or `messages` empty (SDK-side, before sending) |
| `BadRequestError` | 400 / 422 | Malformed request or unsupported passthrough field |
| `AuthenticationError` | 401 | Invalid or missing API key |
| `InsufficientCreditsError` | 402 | Org balance empty |
| `PermissionDeniedError` | 403 | Caller lacks permission |
| `NotFoundError` | 404 | `model` is not a recognized model id (send `"auto"`) |
| `RateLimitError` | 429 | Rate limited (after retries) |
| `EndpointNotReadyError` | 503 | A serving backend behind `auto` is warming or briefly unavailable (after retries) |
| `APITimeoutError` | — | No response within the client timeout (after retries) |
| `APIConnectionError` | — | DNS, TCP, or TLS failure |

`APIStatusError` subclasses expose `status_code`, `detail`, `request_id` (the `x-request-id` header), and the raw `response` for debugging. See [Errors](exceptions.md) for the full hierarchy.

### Retries

Transient failures (408, 409, 429, 500, 502, 503, 504) are retried automatically with exponential backoff and jitter, up to `max_retries` times (default 2), honoring a `Retry-After` header when present. You only see `RateLimitError`, `EndpointNotReadyError`, or `APITimeoutError` after retries are exhausted. For streaming, retries cover only the initial handshake, never a mid-stream drop.

## Using the OpenAI SDK instead

Because Pareta is one OpenAI-compatible endpoint, you do not need this SDK to call it. Point the `openai` client at Pareta's base URL with your `pareta_sk_` key. Note the `/v1` suffix the OpenAI client expects:

```python
from openai import OpenAI

client = OpenAI(api_key="pareta_sk_...", base_url="https://api.pareta.ai/v1")

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What is the invoice total?"}],
)
print(resp.choices[0].message.content)
```

Streaming, `temperature`, `max_tokens`, and the rest work exactly as they do against OpenAI. Metering still applies; a zero balance returns a 402, which the `openai` client surfaces as its own status error. Reach for the Pareta SDK when you want typed errors and the control plane: [matching intent to the catalog](./tasks.md) and [running evals](./evals.md).

## See also

- [`models`](./models.md) — the OpenAI-compatible model list: the single `"auto"` entry.
- [`tasks`](./tasks.md) — browse and match the catalog of tasks auto routes across.
- [`evals`](./evals.md) — benchmark `"auto"` against frontier baselines on your own data.
- [Errors](exceptions.md) — the full exception hierarchy and retry policy.



---

<!-- reference/models.md -->

# models

`client.models` is the OpenAI-compatible model index: `GET /v1/models`. On Pareta it returns exactly one entry — **`"auto"`** — because there is only one model id to call. The resource exists so OpenAI-style tooling that discovers ids by listing keeps working unchanged; the id it discovers is the one you pass to [`chat.completions.create(model=...)`](../guide/inference.md).

Two platform truths show up here:

- **There is no model menu.** "Which model?" is the question `"auto"` answers for you, per request. The open-weights models auto routes across never appear in this list — they stay behind the one id, and you never see or pick a GPU.
- **Calling a model is metered.** Listing is free, but each completion against `"auto"` debits your org balance — one debit per request, no matter how many internal model calls auto's plan makes. An empty balance raises `InsufficientCreditsError` (402) at call time. Top-up is browser-only.

## list

```python
def list(self) -> ModelList
```

**Route:** `GET /v1/models`

Returns a [`ModelList`](#modellist) of the callable model ids — the single `"auto"` entry. There are no parameters and no pagination.

```python
from pareta import Pareta

with Pareta.from_env() as pa:          # reads PARETA_API_KEY (+ optional PARETA_BASE_URL)
    models = pa.models.list()          # ModelList

    for m in models:                   # ModelList is directly iterable
        print(m.id, "·", m.owned_by)   # auto · pareta
```

`m.id` is exactly what you feed to inference. Listing and calling compose directly, which is the contract generic OpenAI tooling relies on:

```python
with Pareta.from_env() as pa:
    first = pa.models.list().data[0]   # the "auto" entry
    resp = pa.chat.completions.create(
        model=first.id,                # == "auto"
        messages=[{"role": "user", "content": "What is the invoice total?"}],
    )
    print(resp.choices[0].message.content)
```

### Async

`AsyncModels.list` is the same call, awaited:

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        models = await pa.models.list()
        for m in models:
            print(m.id, m.owned_by)

asyncio.run(main())
```

## ModelList

The return value of `list()`. It wraps the raw `{"data": [...]}` payload and behaves like a lightweight collection.

| Member | Type | Description |
| --- | --- | --- |
| `data` | `list[Model]` | The callable model entries — `"auto"`. |
| `__iter__()` | `Iterable[Model]` | Iterate models directly: `for m in models`. |
| `__len__()` | `int` | Number of entries: `len(models)`. |

```python
models = pa.models.list()

len(models)          # int
models.data          # list[Model]: the underlying list
list(models)         # same elements, via __iter__
[m.id for m in models]
```

`ModelList` is not indexable directly. To grab one element, go through `.data` (`models.data[0]`) or iterate.

Like every Pareta response object, it keeps the raw server JSON. Reach anything not surfaced as a property with `models.to_dict()` or `models["data"]`.

## Model

One element of `ModelList.data`. It is the OpenAI-compatible model record, so it carries only three fields.

| Property | Type | Description |
| --- | --- | --- |
| `id` | `str \| None` | The model id — `"auto"`. Pass it as `chat.completions.create(model=...)`. |
| `owned_by` | `str \| None` | `"pareta"`. |
| `created` | `int \| None` | Unix timestamp (seconds). |

```python
for m in pa.models.list():
    print(m.id)         # str | None: usable as the `model` arg in inference
    print(m.owned_by)   # str | None: "pareta"
    print(m.created)    # int | None: Unix seconds

    m.to_dict()         # full raw record, nothing lost behind the typed layer
```

The ids of the open-weights models behind `"auto"` never cross into the SDK. That is by design: routing happens server-side, per request, and hardware is resolved for you. See [Core concepts](../guide/core-concepts.md) for the routing and metering model.

## Errors

`list()` makes a plain authenticated GET, so the failure modes are the standard ones. A bad or missing key raises `AuthenticationError` (401); transient 429/5xx and connection timeouts are retried automatically (`max_retries`, default 2) before surfacing as `RateLimitError`, `APIStatusError`, or `APITimeoutError`. All inherit from `ParetaError`.

```python
from pareta import Pareta, AuthenticationError, ParetaError

try:
    with Pareta.from_env() as pa:
        models = pa.models.list()
except AuthenticationError:
    print("Check PARETA_API_KEY (it should start with pareta_sk_).")
except ParetaError as e:
    print("Listing failed:", e)
```

`InsufficientCreditsError` (402) does **not** fire here. Listing is free; metering happens when you call a model. See [Errors and retries](../guide/errors-and-retries.md) for the full hierarchy.

## See also

- [Running inference](../guide/inference.md) — pass `"auto"` to `chat.completions.create`.
- [`tasks`](./tasks.md) — browse and match the catalog of tasks auto routes across.
- [Core concepts](../guide/core-concepts.md) — the routing brain, hidden models, and org-balance metering.



---

<!-- reference/tasks.md -->

# tasks

`client.tasks` is the **grading-contract directory** for evals. A task names
how a dataset is scored: the input/output shape your rows must follow and the
scorer that grades outputs against your labels (field-F1 for extraction,
nDCG@10 for ranking, WER for transcripts, a judge panel for open-ended text).

You never need a task for inference. Production traffic is
`chat.completions.create(model="auto", ...)` — it takes no task id, and every
routing decision happens on Pareta's side. You need a task in exactly one
moment: **when you benchmark on your own data**, because grading requires a
declared contract — [`evals.runs.create(task=...)`](./evals.md) uses it to
validate your rows and score every candidate the same way.

- `match` maps a plain-English description of your dataset to the right
  grading contract, so you don't read a scorer list.
- `list` / `retrieve` browse the contracts and a contract's row schema.

Catalog reads are free: `list`, `retrieve`, and `match` are not metered. The
meter starts when you run compute (inference and eval runs). See
[Errors](exceptions.md) for `InsufficientCreditsError`.

All snippets assume:

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

---

## tasks.match

```python
def match(self, query: str, *, top_k: int = 5) -> TaskMatch
```

**Route:** `POST /v1/tasks/match`

Turns a free-text description of your data or job into the grading contract
that fits it. The matcher is an LLM reasoning router: it reasons about intent
(not keyword overlap). If the router is unavailable it falls back to a
deterministic keyword scorer.

- `query` (required): free-text description, e.g. `"vendor invoices with
  labeled line items and totals"`. Raises `ValueError` if empty or
  whitespace-only.
- `top_k` (default `5`): how many ranked candidates the keyword fallback
  returns. The reasoning router returns a single chosen match (it does not
  rank).

Returns a [`TaskMatch`](#taskmatch). Read `match.type` for the outcome:

- `"task"` — a benchmarked grading contract fits; `.chosen.task_id` names it.
- `"capability"` — the job is a general lane (chat, coding, vision, speech,
  retrieval) rather than a labeled-dataset job; `.capability` describes it.
  General lanes have judge- or metric-scored general tasks when you want to
  benchmark them anyway.
- `"unsupported"` / `"none"` — no grading contract fits the description.
  This is a statement about *scoring*, not about serving: generation work
  can always go to `model="auto"`.

```python
match = pa.tasks.match("pull line items and totals out of vendor invoices")

if match.type == "task":
    task_id = match.chosen.task_id          # the grading contract
    print(f"grade with {task_id} via {match.matcher} "
          f"(confidence={match.confidence})")
elif match.type == "capability":
    print(f"general lane: {match.capability.label}")
else:
    print(f"{match.type}: {match.reasoning}")
```

A robust pattern handles the no-match and ambiguous cases rather than blindly
trusting `chosen`:

```python
match = pa.tasks.match("classify support tickets by urgency")

if not match.matched:
    raise SystemExit(f"no contract matched; closest: "
                     f"{[c.task_id for c in match.candidates]}")
if match.ambiguous:
    # Top two scores are close: a good moment to ask the user to disambiguate.
    print("ambiguous, top candidates:",
          [(c.task_id, round(c.score or 0, 2)) for c in match.candidates[:2]])

task_id = match.chosen.task_id
```

The matched `task_id` goes to
[`evals.runs.create(task=task_id, ...)`](./evals.md); inference itself stays
`model="auto"` — there is nothing to switch.

---

## tasks.retrieve

```python
def retrieve(self, task_id: str, *, examples_n: int | None = None) -> Task
```

**Route:** `GET /v1/tasks/{task_id}`

Fetches a single contract's row schema. The field that matters most is
`has_blob_input`: `True` means the rows carry documents or images (PDFs,
scans), which determines how you build eval sets and which frontier baselines
can run them (vision-capable only).

- `examples_n` (optional): request N example rows from the task's bundled
  golden set — the fastest way to see the exact input/expected shape your
  rows must follow. The typed layer surfaces `id`, `default_scorer`, and
  `has_blob_input`; reach the examples through the raw record with
  `task.to_dict()`.

Returns a [`Task`](#task).

```python
task = pa.tasks.retrieve(task_id, examples_n=3)
print(task.id, task.default_scorer, "blob_input=", task.has_blob_input)

# examples come back on the raw record:
examples = task.to_dict().get("examples", [])
```

---

## tasks.list

```python
def list(self) -> list[Task]
```

**Route:** `GET /v1/tasks`

Returns every grading contract as a `list[Task]`. Use this to browse when you
do not have a free-text query for `match`.

```python
for task in pa.tasks.list():
    kind = "document" if task.has_blob_input else "text"
    print(f"  {task.id:<28} {kind:<10} scorer={task.default_scorer}")
```

---

## From dataset to proof

End to end: a description of your data in, a grading contract out, `"auto"`
proven against the frontier on your own rows — while inference stays
`model="auto"` throughout.

```python
from pareta import Pareta

pa = Pareta.from_env()

# 1. dataset description -> grading contract
match = pa.tasks.match("extract key fields from contracts")
if not match.matched:
    raise SystemExit(f"no contract matched: {[c.task_id for c in match.candidates]}")
task_id = match.chosen.task_id

# 2. inspect the contract (document rows? which scorer?)
task = pa.tasks.retrieve(task_id)
print(f"task={task.id}  scorer={task.default_scorer}  blob={task.has_blob_input}")

# 3. the proof: benchmark "auto" against the frontier on your own rows
run = pa.evals.runs.create(
    task=task_id,
    items=[{"input": "…", "expected": {"effective_date": "2026-01-01"}}],
    models=["auto"],
    frontier="benchmarked",
    wait=True,
)
print("billed:", run.cost)

# production is the same call you started with: model="auto"
```

There is nothing to switch on at the end of this pass: production traffic is
the same `chat.completions.create(model="auto", ...)` call, and the eval is
the proof it holds up on your data. To pick the vendor baselines to measure
against, see `evals.frontier_models` in [evals](./evals.md).

---

## Async

`AsyncTasks` mirrors the sync surface. Every method is `async def` and awaited:

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        match = await pa.tasks.match("extract key fields from contracts")
        if not match.matched:
            return
        task = await pa.tasks.retrieve(match.chosen.task_id)
        print(task.id, task.default_scorer, task.has_blob_input)

        catalog = await pa.tasks.list()
        print(f"{len(catalog)} grading contracts")

asyncio.run(main())
```

See the [async guide](../guide/async.md) for the full sync-vs-async story.

---

## Response models

Every response object keeps the raw server JSON: call `.to_dict()` (or index it
like a dict) to reach any field the typed layer does not surface yet.

### Task

From `GET /v1/tasks` and `GET /v1/tasks/{id}`.

| Field | Type | Notes |
|---|---|---|
| `id` | `str \| None` | Task id, e.g. `"contract-key-fields"` |
| `default_scorer` | `str \| None` | The scorer used to grade outputs on this task |
| `has_blob_input` | `bool` | `True` if the rows carry documents/images (vision tasks) |

### TaskMatch

From `POST /v1/tasks/match`.

| Field | Type | Notes |
|---|---|---|
| `query` | `str \| None` | The echoed query |
| `type` | `str \| None` | `"task"`, `"capability"`, `"unsupported"`, or `"none"` |
| `matched` | `bool` | A high-confidence task was found |
| `chosen` | `TaskMatchCandidate \| None` | The best candidate, or `None` if nothing cleared the bar |
| `capability` | `Capability \| None` | The general lane, when `type == "capability"` |
| `candidates` | `list[TaskMatchCandidate]` | The top-`top_k` ranked alternates |
| `reasoning` | `str \| None` | Why the router picked this match (reasoning matcher only) |
| `confidence` | `str \| None` | `"high"` / `"medium"` / `"low"` (reasoning matcher only) |
| `ambiguous` | `bool` | `True` when the top two scores are close |
| `matcher` | `str \| None` | Which matcher answered: `"reason"` (LLM router) or `"keyword"` (fallback) |

See [`Capability`](types.md#capability) for the capability lane fields (`id`,
`label`, `category`, `category_id`, `desc`).

### TaskMatchCandidate

| Field | Type | Notes |
|---|---|---|
| `task_id` | `str \| None` | The candidate task id |
| `score` | `float \| None` | Match score in `[0, 1]` |
| `confidence` | `str \| None` | `"high"`, `"medium"`, or `"low"` |

---

See also: [evals](./evals.md) · [inference](../guide/inference.md) ·
[errors](exceptions.md) · [core concepts](../guide/core-concepts.md)



---

<!-- reference/evals.md -->

# `evals`: evaluate models on your own data

`client.evals` runs the only benchmark that matters: how `model="auto"` scores on **your** rows. You hand Pareta a task — the [grading contract](./tasks.md) that names your rows' shape and scorer (`tasks.match` finds it from a plain-English description of your dataset) — and a list of labeled items, name the candidates — `"auto"`, plus the frontier baselines to beat — and get back per-candidate quality with 95% confidence intervals and per-item cost. The platform scores everything with the task's scorer, runs every candidate on the same items, and meters the compute against your org balance. No GPUs to size, no scorer to wire up, no judge to host.

The namespace has three parts:

- [`evals.sets`](#evalssets-evaluation-datasets): turn your rows into a reusable eval set (and attach documents for blob tasks).
- [`evals.runs`](#evalsruns-evaluation-runs): run candidates over a set and read aggregated results.
- [`evals.frontier_models`](#evalsfrontier_models-frontier-baseline-roster): list the vendor baselines you can evaluate against.

All examples use the synchronous `Pareta` client. Every method has an `async` twin with the same signature on `AsyncPareta`; see [Async](#async).

```python
from pareta import Pareta

pa = Pareta.from_env()  # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

## The shape of an eval

1. Turn your rows into an **eval set** (`evals.sets.create`), or pass them inline to the run.
2. Kick off an **eval run** over a list of models (`evals.runs.create`), optionally blocking until it finishes.
3. Read `run.results` to compare quality and cost; read `run.cost` for the bill.

```python
run = pa.evals.runs.create(
    task="contract-key-fields",
    items=[
        {"input": "Effective as of January 1, 2026, ...", "expected": {"effective_date": "2026-01-01"}},
        {"input": "This Agreement terminates on 2027-12-31 ...", "expected": {"termination_date": "2027-12-31"}},
    ],
    models=["auto"],         # the product under test
    frontier="benchmarked",  # vendor baselines benchmarked on this task
    wait=True,               # block until the run is terminal
)

print(run.status)             # "completed"
print(f"billed ${run.cost}")  # Decimal dollars, floored to cents

for r in run.results:
    print(f"{r.model_id:16} q={r.quality_mean:.3f} "
          f"[{r.quality_ci_low:.3f}, {r.quality_ci_high:.3f}]  "
          f"~{r.mean_cost_micro_usd} uUSD/item  ({r.n_succeeded} ok, {r.error_count} err)")
```

That single call created the eval set inline, started the run, polled it to completion, and returned an `EvalRun` with one aggregate per candidate. The sections below unpack each piece.

The candidates are `"auto"` plus vendor ids: `models=["auto"]` puts the product under test, and `frontier=` adds the vendor baselines to beat. Frontier ids are in the clear because they are public products — take them from [`frontier_models`](#evalsfrontier_models-frontier-baseline-roster). The open specialists auto routes to are never individually named: the thing you measure is the routed product, not its parts.

## `evals.sets`: evaluation datasets

An eval set is your rows bound to a task, stored server-side and reusable across runs. Create one explicitly when you want to reuse it; otherwise pass `task=` + `items=` straight to `runs.create` (see [inline create](#inline-create)).

### `sets.create`

```python
def create(self, *, task: str, items: list[dict], name: str | None = None) -> EvalSet
```

`POST /v1/eval-sets`

- `task` (required): the task id. Carries the scorer and the input schema.
- `items` (required, non-empty): your evaluation rows. Each is a dict in the task's input schema; the SDK serializes them to JSONL on the wire. An empty list raises `ValueError` before any request goes out.
- `name` (optional): defaults to `f"sdk eval set ({len(items)} items)"`.

```python
eval_set = pa.evals.sets.create(
    task="contract-key-fields",
    items=[
        {"input": "Effective as of January 1, 2026, ...", "expected": {"effective_date": "2026-01-01"}},
        {"input": "This Agreement terminates on 2027-12-31 ...", "expected": {"termination_date": "2027-12-31"}},
    ],
    name="Q2 contracts sample",
)

print(eval_set.id)               # pass this to runs.create(eval_set=...)
print(eval_set.task_id)          # "contract-key-fields"
print(eval_set.item_count)       # 2
print(eval_set.scoring_strategy) # e.g. "extraction": how this task is scored
```

The exact row fields (`input`, `expected`, and any others) follow the task you chose. To inspect a task's schema and pull sample rows before formatting yours, use `pa.tasks.retrieve(task_id, examples_n=...)`. See [`tasks`](./tasks.md).

Returns an [`EvalSet`](#evalset).

### `sets.list`

```python
def list(self) -> list[EvalSet]
```

`GET /v1/eval-sets`: every eval set the org can access.

```python
for s in pa.evals.sets.list():
    print(s.id, s.task_id, s.item_count, s.name)
```

### `sets.retrieve`

```python
def retrieve(self, eval_set_id: str) -> EvalSet
```

`GET /v1/eval-sets/{eval_set_id}`: one set by id.

```python
eval_set = pa.evals.sets.retrieve("evalset_abc123")
```

### `sets.delete`

```python
def delete(self, eval_set_id: str) -> None
```

`DELETE /v1/eval-sets/{eval_set_id}`: remove a set.

```python
pa.evals.sets.delete(eval_set.id)
```

### `sets.upload_document`

```python
def upload_document(
    self,
    eval_set_id: str,
    file,
    *,
    idx: int,
    field_name: str,
    mime: str | None = None,
) -> dict
```

Attaches a binary document (PDF, image, scan) to one row's blob field. Use this for tasks where `task.has_blob_input == True`: create the set with each row's labels (and a placeholder for the blob), then attach the file to that row by index.

- `eval_set_id`: the set to attach to.
- `file`: a path (`str` / `pathlib.Path`), raw `bytes`/`bytearray`, or any binary file-like object with `.read()`. Anything else raises `TypeError`.
- `idx` (required): 0-based row index.
- `field_name` (required): the blob input field on the task schema.
- `mime` (optional): MIME type; guessed from the filename when omitted, falling back to `application/octet-stream`.

```python
eval_set = pa.evals.sets.create(
    task="invoice-extraction",
    items=[
        {"expected": {"total": "1240.00", "vendor": "Katana ML"}},  # doc attached next
        {"expected": {"total": "89.50", "vendor": "Acme"}},
    ],
)

# Attach the PDF for row 0's `document` field.
pa.evals.sets.upload_document(
    eval_set.id,
    "invoices/katana-0001.pdf",  # path, bytes, or binary file-like
    idx=0,
    field_name="document",
)

# Bytes or a file handle work too; override the guessed MIME when needed.
with open("invoices/scan.tiff", "rb") as f:
    pa.evals.sets.upload_document(eval_set.id, f, idx=1, field_name="document", mime="image/tiff")
```

`upload_document` collapses the upload into one call. Files under 5 MiB go up inline via `attach-blob`; larger files mint a signed URL (`blob-upload-url`), stream straight to storage with a `PUT`, then confirm (`blob-upload-complete`). Either way it returns the completion endpoint's response dict. A failed storage `PUT` raises `ParetaError`.

Frontier baselines on document tasks are automatically vision-filtered, so you never accidentally score a scan against a text-only model.

## `evals.runs`: evaluation runs

A run evaluates a list of models over an eval set and returns per-model aggregates.

### `runs.create`

```python
def create(
    self,
    *,
    eval_set: str | None = None,
    task: str | None = None,
    items: list[dict] | None = None,
    models,
    frontier=None,
    name: str | None = None,
    wait: bool = False,
    poll_interval: float = 3.0,
    timeout: float = 900.0,
) -> EvalRun
```

`POST /v1/eval-runs`

You drive it one of two ways. Pass **`eval_set=<id>`** to run against an existing set, **or** pass **`task=...` + `items=...`** to create a set inline in the same call. Passing neither raises `ValueError`.

- `models` (required): the candidate list — pass `["auto"]`. Required even when `frontier` is set; an empty `models` with no frontier ids raises `ValueError`.
- `frontier` (default `None`): the vendor baselines to score alongside your candidates. Keyword or explicit list, [resolved SDK-side](#frontier-resolution).
- `name` (optional): run label; also used as the inline set's name.
- `wait` (default `False`): when `False`, returns as soon as the run is queued (status `"running"` or queued). When `True`, blocks via [`runs.wait`](#runswait) and returns the terminal run.
- `poll_interval` (default `3.0`): seconds between polls when `wait=True`.
- `timeout` (default `900.0`): max seconds to wait; exceeding it raises `ParetaError`.

```python
# Against an existing set
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], wait=True)
```

<a id="inline-create"></a>
**Inline create**: skip `sets.create` entirely and hand the rows to the run; the SDK creates the set for you:

```python
run = pa.evals.runs.create(
    task="contract-key-fields",
    items=[{"input": "...", "expected": {"effective_date": "2026-01-01"}}],
    models=["auto"],
    frontier="benchmarked",
    wait=True,
)
```

**Metering.** Each run is metered: the org balance is debited for the compute across **auto and frontier** candidates. If the balance cannot cover the run, `create` raises `InsufficientCreditsError` (402) before any work is billed. Top-up is browser-only; the SDK never exposes balance or payment methods.

```python
from pareta import InsufficientCreditsError

try:
    run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"],
                               frontier="benchmarked", wait=True)
except InsufficientCreditsError:
    raise SystemExit("Out of credit. Top up in the dashboard (billing is browser-only).")
```

`InsufficientCreditsError` subclasses `APIStatusError`; catch `ParetaError` for one handler over every SDK failure. See [Errors and metering](exceptions.md).

Returns an [`EvalRun`](#evalrun).

#### `frontier=` resolution

`frontier=` controls which vendor models get scored alongside `"auto"`, so the report shows exactly what quality the routing holds and what cost it saves. The SDK resolves the keyword to a concrete list of ids before sending the run:

| `frontier=` | Baselines scored |
| --- | --- |
| `None` or `"none"` (default) | none, your `models=` candidates only (`[]`) |
| `"all"` | every frontier model available for the task |
| `"benchmarked"` | frontier models benchmarked on the task (vision-filtered for document tasks) |
| `["gpt-5.5", "claude-..."]` | exactly these frontier ids, passed through as-is |

```python
# Auto only, no baselines
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier="none", wait=True)

# Everything in the frontier pool for the task
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier="all", wait=True)

# A hand-picked baseline
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier=["gpt-5.5"], wait=True)
```

The `"all"` and `"benchmarked"` keywords need the task to fetch the roster. When you create inline (`task=...`) the SDK already has it; when you pass `eval_set=...` it looks the task up from the set. If it still cannot resolve a task it raises `ValueError`. An unrecognized keyword (anything other than `"all"` / `"benchmarked"` / `"none"`) raises `ValueError`, and a `frontier` that is not `None`, a list/tuple, or a string raises `TypeError`. An explicit list skips the roster lookup entirely.

To enumerate and pin the roster yourself, see [`frontier_models`](#evalsfrontier_models-frontier-baseline-roster).

### `runs.retrieve`

```python
def retrieve(self, run_id: str) -> EvalRun
```

`GET /v1/eval-runs/{run_id}`: full run state, including `results` once the run is terminal.

```python
run = pa.evals.runs.retrieve("evalrun_xyz789")
if run.is_terminal:
    print(run.status, run.results)
```

### `runs.wait`

```python
def wait(self, run_id: str, *, poll_interval: float = 3.0, timeout: float = 900.0) -> EvalRun
```

Polls `runs.retrieve(run_id)` every `poll_interval` seconds until `run.is_terminal` (status `"completed"` or `"failed"`), then returns the final `EvalRun`. Raises `ParetaError` if `timeout` seconds elapse first. This is exactly what `create(..., wait=True)` calls internally, so you can fire a run and block on it later:

```python
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"])   # returns immediately
run = pa.evals.runs.wait(run.id, poll_interval=5.0, timeout=1800.0)  # block on it
```

Or poll on your own schedule without `wait`:

```python
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"])
while not run.is_terminal:
    run = pa.evals.runs.retrieve(run.id)
```

### Reading results

A terminal `EvalRun` carries one [`EvalResult`](#evalresult) per candidate in `run.results`, plus the bill.

```python
run = pa.evals.runs.retrieve(run_id)

if run.status == "failed":
    print("run failed:", run.error_detail)
else:
    ranked = sorted(run.results, key=lambda r: r.quality_mean or 0.0, reverse=True)
    for r in ranked:
        cost_per_item = (r.mean_cost_micro_usd or 0) / 1_000_000  # micro-USD to dollars
        print(f"{r.model_id:24} q={r.quality_mean:.3f} "
              f"[{r.quality_ci_low:.3f}, {r.quality_ci_high:.3f}]  "
              f"${cost_per_item:.6f}/item  ok={r.n_succeeded} err={r.error_count}")

    print(f"run cost: ${run.cost}")          # Decimal dollars, floored to cents
    print(f"raw micro-USD: {run.cost_micro_usd}")
```

Use the confidence interval: two candidates whose CIs overlap are not meaningfully different on this sample, so add rows before calling the comparison. A high `error_count` on one candidate usually means malformed output, not a bad model, so inspect before trusting its quality number. When `"auto"` holds the frontier's quality at a fraction of its per-item cost, production is the call you already have: keep sending `model="auto"`.

**On money.** `run.cost` is a `Decimal` in dollars, **floored to whole cents** (the SDK never rounds a charge up), so a sub-cent run reads `Decimal("0.00")`. `run.cost_micro_usd` is the raw integer (`1_000_000` micro-USD = `$1.00`) for exact accounting. Per-item rates like `result.mean_cost_micro_usd` stay in micro-USD on purpose: flooring sub-cent unit rates to whole cents would erase the auto-vs-frontier cost gap the eval exists to find. Same convention SDK-wide; see [Errors and metering](exceptions.md).

## `evals.frontier_models`: frontier baseline roster

```python
def frontier_models(self, task: str | None = None) -> list[FrontierModel]
```

`GET /v1/eval/frontier-models`: the vendor (frontier) models you can evaluate against. Feed the `.id`s into `runs.create(frontier=[...])`.

- `task` (optional): when given, each entry is annotated `benchmarked` (it has been benchmarked on that task) and the roster is vision-filtered for document tasks. Without a task the full roster comes back unannotated.

```python
roster = pa.evals.frontier_models(task="contract-key-fields")
for m in roster:
    print(m.id, m.vendor, "vision" if m.vision else "text",
          "benchmarked" if m.benchmarked else "-")

# Pin two benchmarked baselines explicitly
ids = [m.id for m in roster if m.benchmarked][:2]
run = pa.evals.runs.create(eval_set=eval_set.id, models=["auto"], frontier=ids, wait=True)
```

Returns a list of [`FrontierModel`](#frontiermodel).

## Async

Every method above has an `async` twin on `AsyncPareta` with an identical signature; the methods are coroutines (`wait` included). Document uploads are async too.

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        eval_set = await pa.evals.sets.create(
            task="contract-key-fields",
            items=[{"input": "...", "expected": {"effective_date": "2026-01-01"}}],
        )
        run = await pa.evals.runs.create(
            eval_set=eval_set.id,
            models=["auto"],
            frontier="benchmarked",
            wait=True,
        )
        for r in run.results:
            print(r.model_id, r.quality_mean)
        print("billed", run.cost)

asyncio.run(main())
```

`await pa.evals.runs.wait(run_id)`, `await pa.evals.frontier_models(task=...)`, and `await pa.evals.sets.upload_document(...)` all work the same way.

## Response objects

Every object keeps the raw server JSON: call `.to_dict()` for lossless access to anything not yet surfaced as a typed field, and index it dict-style (`run["..."]`) as an escape hatch.

### `EvalSet`

From `sets.create`, `sets.list`, `sets.retrieve`.

| Field | Type | Notes |
| --- | --- | --- |
| `id` | `str \| None` | Pass to `runs.create(eval_set=...)` |
| `task_id` | `str \| None` | The task this set is bound to |
| `name` | `str \| None` | Label |
| `item_count` | `int \| None` | Number of rows |
| `scoring_strategy` | `str \| None` | How the task is scored (e.g. `"extraction"`) |

### `EvalRun`

From `runs.create`, `runs.retrieve`, `runs.wait`. Wraps the `{"run": {...}, "results": [...]}` envelope.

| Field | Type | Notes |
| --- | --- | --- |
| `id` | `str \| None` | Run id; pass to `runs.retrieve` / `runs.wait` |
| `eval_set_id` | `str \| None` | The set evaluated |
| `status` | `str \| None` | `"running"`, `"evaluating"`, `"completed"`, `"failed"` |
| `is_terminal` | `bool` | `True` when status is `"completed"` or `"failed"` |
| `candidate_models` | `list[str]` | The candidates evaluated (`"auto"` + frontier ids) |
| `error_detail` | `str \| None` | Error message when `status == "failed"` |
| `cost` | `Decimal` | Billed total in dollars, floored to cents |
| `cost_micro_usd` | `int` | Raw total cost in micro-USD (`1_000_000` = `$1.00`) |
| `results` | `list[EvalResult]` | One aggregate per model (populated once terminal) |

### `EvalResult`

One candidate's aggregate on a run; from `run.results`.

| Field | Type | Notes |
| --- | --- | --- |
| `model_id` | `str \| None` | `"auto"`, or a frontier vendor id |
| `kind` | `str \| None` | `"frontier"` on vendor baseline rows; unset on `"auto"` rows |
| `quality_mean` | `float \| None` | Mean score in `[0, 1]`, your ranking key |
| `quality_ci_low` | `float \| None` | 95% CI lower bound |
| `quality_ci_high` | `float \| None` | 95% CI upper bound |
| `mean_cost_micro_usd` | `int \| None` | Avg cost per item in micro-USD (not floored) |
| `n_succeeded` | `int \| None` | Rows that scored cleanly |
| `error_count` | `int \| None` | Rows that errored |

### `FrontierModel`

A vendor baseline; from `frontier_models`.

| Field | Type | Notes |
| --- | --- | --- |
| `id` | `str \| None` | Pass to `runs.create(frontier=[...])` |
| `vendor` | `str \| None` | `"openai"`, `"anthropic"`, etc. |
| `vision` | `bool` | `True` if vision-capable |
| `benchmarked` | `bool` | `True` if benchmarked on the task (only set when `task=` is given) |

## See also

- [`tasks`](./tasks.md): match intent to a task, inspect its schema, pull example rows.
- [`chat`](./chat.md): the OpenAI-compatible inference surface, metered the same way evals are — production is the same `model="auto"` call you just benchmarked.
- [Errors and metering](exceptions.md): `InsufficientCreditsError`, the money convention, and the full exception hierarchy.



---

<!-- reference/audio.md -->

# audio

`client.audio` is the Speech surface: turn recorded audio into text, and turn
text into spoken audio. It exposes the two general **capability lanes** that are
not chat — `asr` (speech-to-text) and `tts` (text-to-speech) — as two methods:

- [`audio.transcriptions`](#audiotranscriptions): transcribe an audio clip to text (ASR).
- [`audio.speech`](#audiospeech): synthesize speech from text and save it to a file (TTS).

Two facts set this namespace apart from the rest of the SDK:

- **Its own routes, not `chat.completions`.** Speech does not go through
  `chat.completions.create` — `audio.transcriptions(...)` and
  `audio.speech(...)` hit their own dedicated routes directly. You never pick a
  voice model, a GPU, or a quantization; Pareta resolves the serving model behind
  the lane, exactly as `model="auto"` does for chat.
- **Metered per minute of audio.** Both lanes are metered against your org
  balance by **audio duration** — input length for transcription, output length
  for synthesis — not by tokens. An empty balance raises
  [`InsufficientCreditsError`](exceptions.md) (402). Top-up is browser-only; the
  SDK exposes neither balance nor payment methods.

Both calls go through the client's transport, so auth, retries, and typed error
mapping apply exactly as they do everywhere else.

All examples use the synchronous `Pareta` client. Every method has an `async`
twin with the same signature on `AsyncPareta`; see [Async](#async).

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
```

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const t = await pa.audio.transcriptions("clip.wav", { language: "en" });
t.text;                                              // the transcript
const s = await pa.audio.speech("hello there");
await s.save("out.wav");                             // decoded bytes (Node)
```

---

## audio.transcriptions

```python
def transcriptions(self, audio, *, language: str | None = None) -> Transcription
```

**Route:** `POST /v1/audio/transcriptions`

Speech-to-text (the `asr` lane). Hands your audio to Pareta, returns the
transcript plus the detected language and the metered duration.

- `audio` (required): the clip to transcribe, in any of three forms — a **path**
  (`str` / `os.PathLike`) to an audio file, raw audio **bytes**, or an already
  **base64**-encoded string. A path or bytes are read and base64-encoded for you;
  a string that does not name a file is assumed to be base64 and passed through.
  An empty base64 string raises `ValueError`; anything that is not a path, bytes,
  or string raises `TypeError`.
- `language` (optional): an ISO language hint (e.g. `"en"`, `"es"`). Omit it to
  auto-detect across the supported languages.

Metered per minute of **input** audio.

```python
result = pa.audio.transcriptions("meeting-clip.wav")

print(result.text)          # the transcript
print(result.language)      # detected (or the hint you passed)
print(result.duration_s)    # input length that was metered (per minute)
```

`audio=` is flexible about where the bytes come from — a path, an in-memory
buffer, or a pre-encoded string all work — and `language` is a hint, not a
requirement:

```python
# Raw bytes (e.g. from a recorder or an upload), with a language hint.
with open("call.ogg", "rb") as f:
    result = pa.audio.transcriptions(f.read(), language="en")

# A Transcription stringifies to its transcript.
print(str(result))          # same as result.text (empty string if None)
```

Returns a [`Transcription`](#transcription).

---

## audio.speech

```python
def speech(self, text: str, *, voice: str | None = None) -> Speech
```

**Route:** `POST /v1/audio/speech`

Text-to-speech (the `tts` lane). Synthesizes spoken audio from text and returns a
[`Speech`](#speech) whose `.audio` is the decoded bytes — call `.save(path)` to
write a file.

- `text` (required): the text to speak. Empty or whitespace-only text raises
  `ValueError` before any request goes out.
- `voice` (optional): a voice id. Omit it for the default (Kokoro) voice.

Metered per minute of **output** audio.

```python
speech = pa.audio.speech("Pareta turns text into speech in one call.")
speech.save("out.wav")

print(speech.format)        # container/codec, e.g. "wav"
print(speech.sample_rate)   # Hz
print(speech.duration_s)    # output length that was metered (per minute)
```

`.save()` returns the same `Speech`, so you can chain it; pick a voice with
`voice=`:

```python
audio_bytes = pa.audio.speech(
    "Your contract has been processed.",
    voice="af_heart",
).save("notice.wav").audio       # write the file and keep the bytes
```

Returns a [`Speech`](#speech).

---

## Async

Every method above has an `async` twin on `AsyncPareta` with an identical
signature; the methods are coroutines. `Speech.save(...)` is a local file write,
not a network call, so it is the same on both clients.

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        result = await pa.audio.transcriptions("clip.wav", language="en")
        print(result.text, result.duration_s)

        speech = await pa.audio.speech("Hello from Pareta.")
        speech.save("hello.wav")
        print(speech.format, speech.sample_rate)

asyncio.run(main())
```

---

## Response objects

Every object keeps the raw server JSON: call `.to_dict()` for lossless access to
anything not yet surfaced as a typed field, and index it dict-style
(`result["..."]`) as an escape hatch.

### Transcription

From `audio.transcriptions`. Stringifies to `.text` (or `""` when absent).

| Field | Type | Notes |
| --- | --- | --- |
| `text` | `str \| None` | The transcript |
| `language` | `str \| None` | Detected language, or the hint you passed |
| `duration_s` | `float \| None` | Input audio length that was metered (per minute) |

### Speech

From `audio.speech`.

| Field | Type | Notes |
| --- | --- | --- |
| `audio` | `bytes` | The synthesized audio, base64-decoded to raw bytes (`b""` if empty) |
| `audio_base64` | `str \| None` | The raw base64 payload as returned by the server |
| `sample_rate` | `int \| None` | Sample rate in Hz |
| `duration_s` | `float \| None` | Output audio length that was metered (per minute) |
| `format` | `str \| None` | Container/codec of the returned audio, e.g. `"wav"` |

`Speech` also has one method:

| Method | Returns | Notes |
| --- | --- | --- |
| `save(path)` | `Speech` | Write the decoded `.audio` bytes to `path` (`str` / `os.PathLike`); returns `self` for chaining |

## See also

- [`tasks`](./tasks.md): `tasks.match(...)` routes a free-text job to a
  capability lane (including `speech-to-text` / `text-to-speech`) when no
  benchmarked task fits.
- [`chat`](./chat.md): the OpenAI-compatible `model="auto"` inference surface for
  the chat-style capabilities, metered the same way.
- [Errors and metering](exceptions.md): `InsufficientCreditsError`, the per-minute
  metering, and the full exception hierarchy.



---

<!-- reference/images.md -->

# images

`client.images` is the image surface: turn a text prompt into a PNG, or edit
an existing image with a plain-language instruction. It exposes the
`image-gen` **capability lane** as two methods:

- [`images.generate`](#imagesgenerate): generate an image from a prompt and
  save it to a file.
- [`images.edit`](#imagesedit): edit a reference image with an instruction
  (no mask).

Two facts set this namespace apart from the rest of the SDK:

- **Its own routes, not `chat.completions`.** Generation hits
  `POST /v1/images/generations` and editing hits `POST /v1/images/edits`
  directly. You never pick a serving model, a GPU, or a step count; Pareta
  resolves the lane (`hidream-1` today), exactly as `model="auto"` does for
  chat.
- **Metered flat per call.** Every generation debits the same per-image
  price against your org balance — the model renders at full 2K quality
  internally regardless of the delivery size, so every size costs the same.
  Every edit debits the same per-edit price (edits cost more than
  generations: the reference roughly doubles the model's work). The
  `X-Pareta-Billed` response header carries the receipt in micro-USD. An
  empty balance raises [`InsufficientCreditsError`](exceptions.md) (402).

Calls go through the client's transport, so auth, retries, and typed error
mapping apply exactly as they do everywhere else.

All examples use the synchronous `Pareta` client. The method has an `async`
twin with the same signature on `AsyncPareta`.

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (and optional PARETA_BASE_URL)
img = pa.images.generate("a lighthouse on a rocky coast at dusk")
img.save("lighthouse.png")
img.size                        # "1024x1024" — the ACTUAL delivered size
```

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const img = await pa.images.generate("a lighthouse on a rocky coast at dusk");
await img.save("lighthouse.png");            // decoded bytes (Node)
```

---

## images.generate

```python
pa.images.generate(prompt, *, size=None, seed=None) -> ImageGeneration
```

| Parameter | Type | Description |
|---|---|---|
| `prompt` | `str` | What to render. Required, ≤4000 chars. |
| `size` | `str \| None` | Delivery size. Omit for `1024x1024`. Also: `2048x2048`, `2304x1728`, `1728x2304`, `2560x1440`, `1440x2560`. Every size bills the same flat price. |
| `seed` | `int \| None` | Pin the noise seed for reproducibility. |

TypeScript: `pa.images.generate(prompt, { size?, seed? })`.

A generation takes ~12s when the lane is warm; the first request after a
quiet spell can take a few minutes while the model boots.

## images.edit

```python
pa.images.edit(image, prompt, *, seed=None) -> ImageGeneration
```

| Parameter | Type | Description |
|---|---|---|
| `image` | `str \| PathLike \| bytes` | The reference image: a file path, raw bytes, or an already-base64 string. PNG/JPEG, ≤25MB decoded. |
| `prompt` | `str` | The edit instruction, in plain language. Required, ≤4000 chars. Instruction-only — there is no mask parameter. |
| `seed` | `int \| None` | Pin the noise seed for reproducibility. |

TypeScript: `pa.images.edit(image, prompt, { seed? })` — `image` is a file
path (Node), `Uint8Array`/`ArrayBuffer`/`Blob`, or `{ base64 }`.

```python
pa.images.edit("product.png", "put the bottle on a marble surface").save("v2.png")
```

The output keeps the reference's aspect ratio (a ~1MP reference renders at
~4MP). A warm edit takes ~30s. Billed flat per edit.

## The ImageGeneration object

| Accessor | Type | Description |
|---|---|---|
| `.image` | `bytes` / `Uint8Array` | The generated image, base64-decoded PNG bytes. |
| `.b64_json` / `.b64Json` | `str \| None` | The raw base64 payload. |
| `.size` | `str \| None` | The actual delivered size (e.g. `"1024x1024"`). |
| `.model` | `str \| None` | The lane's public model name (`hidream-1`). |
| `.created` | `int \| None` | Unix timestamp. |
| `.save(path)` | — | Write the PNG to `path`. Returns the object for chaining. |

## CLI

```bash
pareta image "a red fox in the snow" --out fox.png --size 2048x2048
```

Billed flat per image — the table in [`pareta --help`](../guide/cli.md) lists
the options.



---

<!-- reference/rerank.md -->

# rerank

`client.rerank` is the Retrieval surface: rank a list of candidate documents
by relevance to a query. It exposes the `rerank` **capability lane** — the
Cohere-shaped workload behind search, RAG context selection, and citation
finding — as a single callable:

- [`rerank(query, documents, top_n=None)`](#rerank-1): score and order the
  documents, most relevant first.

Two facts set this namespace apart from the rest of the SDK:

- **Its own route, not `chat.completions`.** Reranking does not go through
  `chat.completions.create` — `rerank(...)` hits `POST /v1/rerank` directly.
  You never pick a serving model, a GPU, or a quantization; Pareta resolves
  the model behind the lane (`pareta-rerank-1`, a purpose-trained pointwise
  reranker), exactly as `model="auto"` does for chat.
- **Metered per document scored.** Each call is metered against your org
  balance by the number of documents it scores — not by tokens, and not by
  how many results `top_n` keeps. An empty balance raises
  [`InsufficientCreditsError`](exceptions.md) (402). Top-up is browser-only;
  the SDK exposes neither balance nor payment methods.

The call goes through the client's transport, so auth, retries, and typed
error mapping apply exactly as they do everywhere else.

All examples use the synchronous `Pareta` client. The `async` twin has the
same signature on `AsyncPareta` (`await pa.rerank(...)`).

```python
from pareta import Pareta

pa = Pareta()  # PARETA_API_KEY from the environment

docs = [
    "This Agreement shall be governed by the laws of the State of Delaware.",
    "Either party may terminate upon thirty (30) days written notice.",
    "All notices shall be delivered to the addresses set forth above.",
]

ranked = pa.rerank("Which state's law governs this contract?", docs)

ranked.results[0].index            # 0 — position in YOUR docs list
ranked.results[0].relevance_score  # calibrated P(relevant), e.g. 0.97
ranked.top_documents(docs)[0]      # the winning text itself
```

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const ranked = await pa.rerank("Which state's law governs this contract?", docs);
ranked.results[0].index;           // position in YOUR docs array
ranked.results[0].relevanceScore;  // calibrated P(relevant)
ranked.topDocuments(docs)[0];      // the winning text itself
```

## rerank()

```python
pa.rerank(query, documents, *, top_n=None) -> Rerank
```

| Parameter | Type | Notes |
|---|---|---|
| `query` | `str` | What to rank against. Required, non-empty. |
| `documents` | `Sequence[str]` | The candidate texts. Required, non-empty; up to 1,000 per call. Only the first ~512 tokens of each document are scored. |
| `top_n` | `int \| None` | Truncate the response to the best N. All documents are still scored (and metered). |

Returns a [`Rerank`](#the-rerank-object). Raises `ValueError` locally on an
empty query/documents (before any network call); server-side validation
errors surface as typed API errors like everywhere else.

## The `Rerank` object

| Accessor | Type | Meaning |
|---|---|---|
| `.results` | `list[RerankResult]` | Ordered most-relevant-first. Each row has `.index` (position in your `documents`) and `.relevance_score` (calibrated P(relevant) ∈ (0, 1) — thresholdable, not just ordinal). |
| `.model` | `str` | The lane's serving model (`pareta-rerank-1`). |
| `.pairs` | `int` | Documents scored — the metered unit. |
| `.top_documents(documents)` | `list[str]` | Convenience: map the ranked indices back onto the list you sent, best first. |

## Scores are calibrated

The reranker is a pointwise yes/no scorer: each `relevance_score` is an
independent probability, not a softmax over the batch. That means a fixed
threshold (say, `>= 0.5`) means the same thing across calls — use it to
*filter* ("keep only actually-relevant passages"), not just to sort:

```python
relevant = [docs[r.index] for r in ranked.results if r.relevance_score >= 0.5]
```

## Benchmark it on your data

`document-reranking` is a catalog task: upload your own (query, documents,
graded relevance) items as an eval set and the benchmark scores nDCG@10 per
candidate — the same metric, math, and serving path this route uses. See
[Evaluation](../guide/evaluation.md).



---

<!-- reference/embeddings.md -->

# embeddings

`client.embeddings` is the Retrieval surface's recall lane: turn text into
vectors for semantic search and RAG. It exposes the `embed` **capability
lane** as a single callable:

- [`embeddings(input, input_type=None)`](#embeddings-1): embed one string or
  a list, order-preserving.

Two facts set this namespace apart from the rest of the SDK:

- **Its own route, not `chat.completions`.** `embeddings(...)` hits
  `POST /v1/embeddings` directly (OpenAI-shaped request and response). You
  never pick a serving model or GPU; Pareta resolves the model behind the
  lane (`qwen-embed-1` — Qwen3-Embedding-0.6B, the open embedder that beats
  `text-embedding-3-large` on the measured CUAD recall benchmark).
- **Metered per input token.** Each call debits your org balance by the
  tokens embedded — $0.004 per 1M tokens, 5× under OpenAI
  `text-embedding-3-small` and 32× under `3-large`. An empty balance raises
  [`InsufficientCreditsError`](exceptions.md) (402).

```python
from pareta import Pareta

pa = Pareta()  # PARETA_API_KEY from the environment

# Index side: embed documents raw.
docs = ["Delaware law governs.", "30-day termination notice.", "Notices in writing."]
doc_vecs = pa.embeddings(docs).vectors

# Search side: embed the QUERY with input_type="query" — retrieval queries
# embed differently from passages (the model's retrieval instruction).
q = pa.embeddings("which state's law applies?", input_type="query").vectors[0]

# Vectors are unit-normalized: cosine similarity is a plain dot product.
best = max(range(len(docs)), key=lambda i: sum(a * b for a, b in zip(q, doc_vecs[i])))
```

```typescript
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const docVecs = (await pa.embeddings(docs)).vectors;
const q = (await pa.embeddings("which state's law applies?", { inputType: "query" })).vectors[0];
// unit vectors: cosine is a plain dot product
const dot = (a: number[], b: number[]) => a.reduce((s, v, i) => s + v * b[i], 0);
const best = docVecs.reduce((bi, v, i) => (dot(q, v) > dot(q, docVecs[bi]) ? i : bi), 0);
```

## embeddings()

```python
pa.embeddings(input, *, input_type=None) -> Embeddings
```

| Parameter | Type | Notes |
|---|---|---|
| `input` | `str \| Sequence[str]` | Text(s) to embed. Up to 512 per call; each is truncated at 512 tokens. |
| `input_type` | `"query" \| "document" \| None` | `"query"` applies the retrieval-query embedding; default (`None`/`"document"`) embeds raw. |

Returns an [`Embeddings`](#the-embeddings-object). Raises `ValueError`
locally on empty input (before any network call).

## The `Embeddings` object

| Accessor | Type | Meaning |
|---|---|---|
| `.vectors` | `list[list[float]]` | Unit-normalized 1024-dim vectors, in your input order. |
| `.model` | `str` | The lane's serving model (`qwen-embed-1`). |
| `.prompt_tokens` | `int` | Tokens embedded — the metered unit. |
| `len(result)` | `int` | Number of vectors. |

## The RAG stack on Pareta

Embeddings are the recall stage; pair them with [`rerank`](rerank.md) for
precision — retrieve a wide candidate set by cosine, then let
`pareta-rerank-1` re-score the top 50–100. Both stages are benchmarkable on
your own data (the `text-embedding` and `document-reranking` catalog tasks
score nDCG@10 against your graded relevance), and both are measured on the
same public pools, so the leaderboard shows exactly what each stage buys.



---

<!-- reference/exceptions.md -->

# Exceptions

Every error the Pareta SDK raises is a subclass of `ParetaError`. That single
base class is the contract: one `except ParetaError` catches anything the SDK
can throw, and a narrower `except InsufficientCreditsError` catches exactly the
case you care about. Server errors carry the HTTP `status_code`, the server's
`detail` message, and a `request_id` you can quote in a support ticket.

This page is the class-by-class reference: the full hierarchy, the
status-code-to-class mapping, and the attributes on each error. For the
narrative version (what gets retried automatically, how to tune timeouts and
the retry budget), see [Errors, retries & timeouts](../guide/errors-and-retries.md).

## Import

All exception classes are exported from the top-level package. Import the ones
you handle directly.

```python
from pareta import (
    ParetaError,                # base class for everything below
    APIConnectionError,         # never reached the server (DNS/TCP/TLS)
    APITimeoutError,            # subclass of APIConnectionError
    APIStatusError,             # any non-2xx from the server
    BadRequestError,            # 400, 422
    AuthenticationError,        # 401
    InsufficientCreditsError,   # 402 — org out of balance
    PermissionDeniedError,      # 403
    NotFoundError,              # 404
    ConflictError,              # 409
    RateLimitError,             # 429
    EndpointNotReadyError,      # 503 — a backend behind auto warming/briefly down
)
```

## The hierarchy

```
Exception
└── ParetaError                      base class for every SDK error
    ├── APIConnectionError           request never reached the server
    │   └── APITimeoutError          timed out before any response
    └── APIStatusError               server returned a non-2xx status
        ├── BadRequestError          400, 422
        ├── AuthenticationError      401
        ├── InsufficientCreditsError 402
        ├── PermissionDeniedError    403
        ├── NotFoundError            404
        ├── ConflictError            409
        ├── RateLimitError           429
        └── EndpointNotReadyError    503
```

Two facts that fall out of this tree and are worth holding onto:

- `APITimeoutError` is a subclass of `APIConnectionError`, so catching
  `APIConnectionError` also catches timeouts.
- Every status-mapped class (`BadRequestError`, `InsufficientCreditsError`, and
  the rest) is a subclass of `APIStatusError`, so catching `APIStatusError`
  catches all of them and gives you `.status_code`, `.detail`, and `.request_id`.

## Status code mapping

When the server returns a non-2xx response, the SDK builds the most specific
`APIStatusError` subclass for that status. Anything not in the table below
(other 5xx, unexpected codes) surfaces as a plain `APIStatusError` carrying the
raw `status_code`.

| Status | Exception | When |
| --- | --- | --- |
| 400 | `BadRequestError` | Request rejected by the server |
| 401 | `AuthenticationError` | Missing or invalid `pareta_sk_` API key |
| 402 | `InsufficientCreditsError` | Org balance is empty — top up in the dashboard |
| 403 | `PermissionDeniedError` | Authenticated, but not allowed |
| 404 | `NotFoundError` | Task, eval set, or run id does not exist |
| 409 | `ConflictError` | Conflict (transient lock/contention) |
| 422 | `BadRequestError` | FastAPI request validation failed |
| 429 | `RateLimitError` | Rate limited; honors `Retry-After` |
| 503 | `EndpointNotReadyError` | A serving backend behind `auto` is warming or briefly unavailable |
| other 5xx | `APIStatusError` | Generic server error |

Note that `400` and `422` both map to `BadRequestError`, so a single clause
covers both client-side and FastAPI-validation rejections.

## Class reference

### `ParetaError`

The base class for every error the SDK raises. Catch this to handle any SDK
failure with one clause. It is also raised directly in one non-HTTP case:
constructing a client with no API key.

```python
from pareta import Pareta, ParetaError

try:
    pa = Pareta()            # no api_key arg, PARETA_API_KEY unset
except ParetaError as e:
    print(e)                 # "missing API key. Pass api_key=… or set PARETA_API_KEY …"
```

### `APIConnectionError(ParetaError)`

The request never reached the server: DNS failure, TCP refusal, TLS error, or a
dropped connection. Connection failures on the initial handshake are retried up
to `max_retries` times; this is raised only after the retry budget is spent.

```python
APIConnectionError(message: str = "connection error", *, cause: BaseException | None = None)
```

The underlying `httpx` exception is attached as `.__cause__`, so a traceback
shows the original network error.

### `APITimeoutError(APIConnectionError)`

The request did not complete within the client `timeout` (default
`httpx.Timeout(60.0, connect=10.0)`). Because it subclasses
`APIConnectionError`, an `except APIConnectionError` clause also catches it.

```python
APITimeoutError(message: str = "request timed out", *, cause: BaseException | None = None)
```

### `APIStatusError(ParetaError)`

The server returned a non-2xx status. This is the parent of every
status-mapped class below, and is also raised directly for any status not in
the [mapping table](#status-code-mapping).

**Attributes**

```python
status_code: int                  # the HTTP status
detail: Any                       # server's `detail` message, or the raw body
request_id: str | None            # value of the x-request-id response header
response: httpx.Response | None   # the full response, for advanced use
```

`detail` is the FastAPI `{"detail": "..."}` message when the body is JSON;
otherwise it falls back to the raw response text. `request_id` comes from the
`x-request-id` header and is the value to quote when reporting a problem.

```python
from pareta import Pareta, APIStatusError

pa = Pareta.from_env()
try:
    pa.tasks.retrieve("nonexistent-task")
except APIStatusError as e:
    print(e.status_code)     # 404
    print(e.detail)          # server's explanation
    print(e.request_id)      # quote this in a support ticket
```

### `BadRequestError(APIStatusError)` — 400, 422

The request was rejected by the server, either as a bad request (400) or a
FastAPI validation failure (422). Inspect `.detail` for the specific field or
reason.

### `AuthenticationError(APIStatusError)` — 401

The `pareta_sk_` API key is missing or invalid. Mint a fresh key in the
dashboard and pass it via `Pareta.from_env()` (reads `PARETA_API_KEY`) or
`Pareta(api_key="pareta_sk_…")`.

### `InsufficientCreditsError(APIStatusError)` — 402

The org balance is empty. Both metered paths raise this: inference
(`chat.completions.create()`) and eval runs (`evals.runs.create()`, billed for
auto and frontier compute combined). Top-up is browser-only — the SDK never
exposes balance or payment methods, so the fix is to add credit in the
dashboard and retry.

```python
from pareta import Pareta, InsufficientCreditsError

pa = Pareta.from_env()
try:
    pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Extract the parties."}],
    )
except InsufficientCreditsError:
    print("Org is out of credit — top up at https://pareta.ai in the dashboard, then retry.")
```

### `PermissionDeniedError(APIStatusError)` — 403

The key is valid but the org or user lacks permission for the requested
resource or action.

### `NotFoundError(APIStatusError)` — 404

The referenced resource does not exist: an unknown task id, eval set id, or
run id.

### `ConflictError(APIStatusError)` — 409

A conflict on the server — a transient lock or contention on the resource. A
409 is retried automatically (see [the retry list below](#what-gets-retried));
a stable 409 surfaces here after the retry budget is spent.

### `RateLimitError(APIStatusError)` — 429

Too many requests. The client retries 429s automatically, honoring the server's
`Retry-After` header when present; this is raised only after retries are
exhausted.

### `EndpointNotReadyError(APIStatusError)` — 503

A serving backend behind `auto` is warming up (a cold start) or briefly
unavailable. 503 is in the automatic retry set, so a brief warm-up often
resolves before this is raised. If it still surfaces, there is nothing to fix
on your side — wait briefly and re-issue the request.

```python
import time

from pareta import Pareta, EndpointNotReadyError

pa = Pareta.from_env()
try:
    pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "ping"}],
    )
except EndpointNotReadyError:
    time.sleep(10)   # still warming after the SDK's own retries — pause, then re-issue
```

## What gets retried

The client retries transient failures before raising, so most of the errors
above only surface after the retry budget (`max_retries`, default `2`) is spent.

- **Retried automatically:** connection and timeout errors on the initial
  handshake, and HTTP statuses **408, 409, 429, 500, 502, 503, 504**. Backoff is
  exponential with jitter — `min(0.5 * 2**attempt, 8.0)` seconds — and honors a
  `Retry-After` header when the server sends one.
- **Never retried:** `400`, `401`, `402`, `403`, `404`, `422`, and any other
  status not in the list. These are deterministic — retrying would not change the
  outcome — so they raise immediately.
- **Mid-stream drops are not retried.** Retries apply only to the initial
  handshake; once SSE bytes are flowing (a streaming chat completion) a dropped
  connection raises immediately, since the stream cannot be safely resumed.

Tune the budget per client:

```python
from pareta import Pareta

pa = Pareta.from_env(max_retries=5)   # default is 2; set 0 to disable retries
```

## Handling errors

Order `except` clauses from most specific to least specific. The base
`ParetaError` is the safety net at the bottom.

```python
from pareta import (
    Pareta,
    InsufficientCreditsError,
    EndpointNotReadyError,
    RateLimitError,
    APIStatusError,
    APIConnectionError,
    ParetaError,
)

pa = Pareta.from_env()

try:
    completion = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Summarize this contract."}],
    )
    print(completion.choices[0].message.content)
except InsufficientCreditsError:
    print("Out of credit — top up in the dashboard, then retry.")
except EndpointNotReadyError:
    print("A backend is still warming — wait briefly, then retry.")
except RateLimitError as e:
    print(f"Rate limited; the client already retried. request_id={e.request_id}")
except APIStatusError as e:
    print(f"Server returned {e.status_code}: {e.detail} (request_id={e.request_id})")
except APIConnectionError:
    print("Could not reach the API after retries — check the network.")
except ParetaError as e:
    print(f"Unexpected SDK error: {e}")
```

The same classes and hierarchy apply to the async client — `await`ed
`AsyncPareta` calls raise the exact same exception types, so error handling code
is identical across sync and async.

```python
import asyncio
from pareta import AsyncPareta, APIStatusError

async def main():
    async with AsyncPareta.from_env() as pa:
        try:
            await pa.tasks.retrieve("nonexistent-task")
        except APIStatusError as e:
            print(e.status_code, e.detail)

asyncio.run(main())
```

## Pre-flight `ValueError`

A few methods validate arguments before any network call and raise the
standard-library `ValueError` (not a `ParetaError`) when an argument is plainly
unusable. These are programming errors caught early, not server responses:

- `chat.completions.create()` — empty `model` or `messages`
- `tasks.match()` — empty `query`
- `evals.sets.create()` — empty `items`
- `evals.runs.create(frontier=…)` — a `frontier` value that cannot be resolved

## See also

- [Errors, retries & timeouts](../guide/errors-and-retries.md) — the narrative
  guide: retry behavior, backoff, and tuning the timeout and retry budget.
- [Configuration](../guide/configuration.md) — setting `api_key`, `base_url`,
  `timeout`, and `max_retries`.
- [Inference](../guide/inference.md) — the metered chat path that raises
  `InsufficientCreditsError`.
- [Evaluation](../guide/evaluation.md) — eval runs, also metered against the org
  balance.



---

<!-- reference/types.md -->

# Response types

Every method that talks to the API hands you back a typed object, not a bare
dict. These objects give you attribute access and autocomplete over the shapes
the API returns: a chat completion's `choices`, a task match's `type`, an eval
run's `cost`. They are thin: each one wraps the raw server JSON and exposes the
fields you actually use as properties.

This page is the field-by-field reference for those objects. For how the methods
that return them work, see [Running inference](../guide/inference.md),
[tasks](./tasks.md), and [Evaluating models](../guide/evaluation.md).

## The shared base: every object keeps the raw JSON

All response objects inherit from one base. Two things are true of every object
on this page:

- `.to_dict()` returns the exact JSON the server sent, losslessly. The typed
  properties are a convenience layer over it; nothing is dropped.
- `obj["some_key"]` and `obj.get("some_key", default)` read raw fields directly.
  This is the escape hatch for any field the platform adds before the typed layer
  catches up.

```python
from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (+ optional PARETA_BASE_URL)

resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "ping"}],
)

resp.choices[0].message.content   # typed access
resp.to_dict()                    # the full raw JSON, lossless
resp["id"]                        # raw-key access for anything not yet typed
```

Properties return `None` (or an empty list) when a field is absent rather than
raising, so reading an optional field is always safe.

## Inference types

These come back from `chat.completions.create` (route `POST /v1/chat/completions`).
Inference is OpenAI-compatible, so the schema matches the OpenAI chat objects.

### ChatCompletion

The non-streaming result of `chat.completions.create(...)`.

| Property | Type | Notes |
|---|---|---|
| `id` | `str \| None` | Completion id |
| `model` | `str \| None` | The model id on the completion |
| `created` | `int \| None` | Unix timestamp |
| `choices` | `list[Choice]` | One entry per generated choice |
| `usage` | `Usage` | Token counts |

```python
resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Extract the effective date."}],
    temperature=0,
)
print(resp.choices[0].message.content)
print(resp.usage.total_tokens)
```

### ChatCompletionChunk

One delta from a streaming completion. Returned (one per SSE event) when you pass
`stream=True`. It has the same schema as `ChatCompletion`; it is a distinct type
purely for hinting. The incremental text lives on `choices[0].delta.content`, not
`choices[0].message`.

```python
for chunk in pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize this contract."}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="", flush=True)
```

The `or ""` guard matters: the first and last chunks often carry no content (role
preamble, finish marker), so `delta.content` can be `None` mid-stream.

### Choice

One element of `completion.choices`.

| Property | Type | Notes |
|---|---|---|
| `index` | `int \| None` | Position in the choices list |
| `finish_reason` | `str \| None` | `"stop"`, `"length"`, etc. |
| `message` | `Message` | The full message. Populated on **non-streaming** results |
| `delta` | `Message` | The incremental token. Populated on **streaming** chunks |

`message` and `delta` always return a `Message` (empty if absent), so reading
`choice.delta.content` on a non-streaming result, or vice versa, returns `None`
rather than blowing up.

### Message

The content of a `Choice`.

| Property | Type | Notes |
|---|---|---|
| `role` | `str \| None` | `"assistant"`, `"user"`, etc. |
| `content` | `str \| None` | The text |

### Usage

Token accounting on a `ChatCompletion`.

| Property | Type |
|---|---|
| `prompt_tokens` | `int \| None` |
| `completion_tokens` | `int \| None` |
| `total_tokens` | `int \| None` |

## Model listing types

Returned from `models.list()` (route `GET /v1/models`). This is the
OpenAI-compatible model listing: it returns exactly one entry, `"auto"`, so
any OpenAI-style tooling pointed at Pareta gets a sensible `/models` response
with the one id you send.

### ModelList

| Property | Type |
|---|---|
| `data` | `list[Model]` |

`ModelList` is directly iterable and has a length, so you usually skip `.data`:

```python
models = pa.models.list()
print(len(models))
for m in models:                       # iterates m in models.data
    print(m.id, m.owned_by)
```

### Model

One element of a `ModelList`.

| Property | Type | Notes |
|---|---|---|
| `id` | `str \| None` | `"auto"`. Pass straight into `chat.completions.create(model=...)` |
| `owned_by` | `str \| None` | `"pareta"` |
| `created` | `int \| None` | Unix timestamp |

## Discovery types

These come from the `tasks` namespace and name the grading contract before you
send traffic.

### Task

Returned from `tasks.list()` and `tasks.retrieve(id)`. One benchmarked job.

| Property | Type | Notes |
|---|---|---|
| `id` | `str \| None` | Stable task id, e.g. `"contract-key-fields"` |
| `default_scorer` | `str \| None` | The function that grades model output for this task |
| `has_blob_input` | `bool` | `True` when the task takes documents or images, not just text |

```python
for t in pa.tasks.list():
    print(t.id, t.default_scorer, "doc" if t.has_blob_input else "text")
```

`has_blob_input` tells you whether you will need
`evals.sets.upload_document(...)` to attach binaries when you evaluate on this
task.

### TaskMatch

Returned from `tasks.match(query, top_k=...)`. The ranked result of matching
free-text intent to a task.

| Property | Type | Notes |
|---|---|---|
| `query` | `str \| None` | The query, echoed back |
| `type` | `str \| None` | `"task"`, `"capability"`, `"unsupported"`, or `"none"` |
| `matched` | `bool` | `True` when a high-confidence match was found |
| `chosen` | `TaskMatchCandidate \| None` | The best candidate, or `None` if nothing matched confidently |
| `capability` | `Capability \| None` | The general lane, when `type == "capability"` |
| `candidates` | `list[TaskMatchCandidate]` | Top-K ranked alternates |
| `reasoning` | `str \| None` | The router's rationale (reasoning matcher only) |
| `confidence` | `str \| None` | `"high"` / `"medium"` / `"low"` (reasoning matcher only) |
| `ambiguous` | `bool` | `True` when the top two scores are close |
| `matcher` | `str \| None` | Which strategy answered: `"reason"` (LLM router) or `"keyword"` (fallback) |

```python
m = pa.tasks.match("pull totals and dates out of vendor invoices", top_k=5)
if m.type == "task" and m.chosen:
    print("best:", m.chosen.task_id, m.chosen.score, m.chosen.confidence)
elif m.type == "capability" and m.capability:
    print("capability:", m.capability.id, m.capability.label)
else:
    print(m.type, "—", m.reasoning)         # "unsupported" / "none"
print("via", m.matcher)
```

See [tasks.match](./tasks.md#tasksmatch) for the full matching semantics.

### Capability

The general capability lane a match resolved to — on `TaskMatch.capability` when
`TaskMatch.type == "capability"`.

| Property | Type | Notes |
|---|---|---|
| `id` | `str \| None` | The lane id (`chat`/`coding`/`agentic`/`vision`/`asr`/`tts`) |
| `label` | `str \| None` | Human-readable label |
| `category` | `str \| None` | Catalog category name |
| `category_id` | `str \| None` | Catalog category id |
| `desc` | `str \| None` | One-line description |

### TaskMatchCandidate

An element of `match.candidates` (and the type of `match.chosen`).

| Property | Type | Notes |
|---|---|---|
| `task_id` | `str \| None` | The candidate task's id |
| `score` | `float \| None` | Match score in `[0, 1]` |
| `confidence` | `str \| None` | `"high"`, `"medium"`, or `"low"` |

### FrontierModel

Returned from `evals.frontier_models(task=...)`. A vendor model you can evaluate
against — the baseline `"auto"` is measured by.

| Property | Type | Notes |
|---|---|---|
| `id` | `str \| None` | Vendor model id. Feed into `evals.runs.create(frontier=[...])` |
| `vendor` | `str \| None` | `"openai"`, `"anthropic"`, etc. |
| `vision` | `bool` | `True` if vision-capable |
| `benchmarked` | `bool` | `True` if it is benchmarked on the task. Only meaningful when you passed `task=` |

Frontier ids are shown in the clear because they are public products. The open
specialists auto routes to are not — they never surface as ids.

```python
for fm in pa.evals.frontier_models(task="contract-key-fields"):
    flag = "vision" if fm.vision else "text"
    note = " (benchmarked on this task)" if fm.benchmarked else ""
    print(fm.id, fm.vendor, flag, note)
```

## Audio types

The Speech lanes (`asr`, `tts`) return these from the `audio` namespace.

### Transcription

Returned from `audio.transcriptions(audio, language=...)`. Speech-to-text.

| Property | Type | Notes |
|---|---|---|
| `text` | `str \| None` | The transcript (also `str(transcription)`) |
| `language` | `str \| None` | Detected (or supplied) language |
| `duration_s` | `float \| None` | Input audio length, metered per minute |

```python
t = pa.audio.transcriptions("call.wav")   # path | bytes | base64
print(t.text, t.language, t.duration_s)
```

### Speech

Returned from `audio.speech(text, voice=...)`. Text-to-speech.

| Property | Type | Notes |
|---|---|---|
| `audio` | `bytes` | The synthesized audio, base64-decoded |
| `audio_base64` | `str \| None` | The raw base64 the server returned |
| `sample_rate` | `int \| None` | Sample rate of the audio |
| `duration_s` | `float \| None` | Output audio length, metered per minute |
| `format` | `str \| None` | Container/codec (e.g. `"wav"`) |

`save(path)` writes the decoded bytes to a file and returns `self`.

```python
pa.audio.speech("Hello from Pareta.").save("out.wav")
```

## Evaluation types

These come from the `evals` namespace and carry the cost numbers you compare
`"auto"` against the frontier with.

### EvalSet

Returned from `evals.sets.create(...)`, `evals.sets.list()`, and
`evals.sets.retrieve(id)`. A reusable evaluation dataset.

| Property | Type | Notes |
|---|---|---|
| `id` | `str \| None` | Eval set id. Pass to `evals.runs.create(eval_set=...)` |
| `task_id` | `str \| None` | The task this set is graded against |
| `name` | `str \| None` | Label (auto-generated if you did not pass one) |
| `item_count` | `int \| None` | Number of rows |
| `scoring_strategy` | `str \| None` | The strategy used to grade rows, e.g. `"extraction"`, `"classification"` |

```python
es = pa.evals.sets.create(
    task="contract-key-fields",
    items=[{"input": "...contract...", "expected": {"effective_date": "2026-01-01"}}],
    name="my contracts v1",
)
print(es.id, es.task_id, es.item_count, es.scoring_strategy)
```

### EvalRun

Returned from `evals.runs.create(...)`, `evals.runs.retrieve(id)`, and
`evals.runs.wait(id)`. The state of an evaluation, including per-model results
once it is terminal. The object wraps the server's `{"run": {...}, "results":
[...]}` envelope and flattens it for you.

| Property | Type | Notes |
|---|---|---|
| `id` | `str \| None` | Run id |
| `eval_set_id` | `str \| None` | The set being evaluated |
| `status` | `str \| None` | `"running"`, `"evaluating"`, `"completed"`, `"failed"` |
| `is_terminal` | `bool` | `True` when `status` is `"completed"` or `"failed"` |
| `candidate_models` | `list[str]` | The candidates evaluated: `"auto"` and any frontier ids |
| `error_detail` | `str \| None` | Failure message when `status == "failed"` |
| `cost_micro_usd` | `int` | Raw total cost in **micro-USD** |
| `cost` | `Decimal` | Billed total in **dollars, floored to cents**. See [money](#money-cost-vs-cost_micro_usd) |
| `results` | `list[EvalResult]` | Per-model aggregates (populated once terminal) |

```python
run = pa.evals.runs.create(
    task="contract-key-fields",
    items=[{"input": "...", "expected": {"effective_date": "2026-01-01"}}],
    models=["auto"],                  # the product under test
    frontier="benchmarked",           # vendor baselines benchmarked on this task
    wait=True,                        # block until terminal
)

if run.status == "failed":
    print("eval failed:", run.error_detail)
else:
    for r in sorted(run.results, key=lambda r: r.quality_mean or 0, reverse=True):
        print(r.model_id, r.kind, r.quality_mean, r.mean_cost_micro_usd, f"n={r.n_succeeded}")
    print("billed:", run.cost, "| raw µUSD:", run.cost_micro_usd)
```

Eval compute is metered against your org balance (both the auto runs and any
frontier baselines). An empty balance raises `InsufficientCreditsError` (402);
top up in the browser, since the SDK never exposes balance or payment.

### EvalResult

One element of `run.results`: a single candidate's aggregate over the run.

| Property | Type | Notes |
|---|---|---|
| `model_id` | `str \| None` | The candidate evaluated: `"auto"` or a frontier vendor id |
| `kind` | `str \| None` | `"frontier"` on vendor baseline rows; unset on `"auto"` rows |
| `quality_mean` | `float \| None` | Mean score in `[0, 1]` |
| `quality_ci_low` | `float \| None` | 95% CI lower bound |
| `quality_ci_high` | `float \| None` | 95% CI upper bound |
| `mean_cost_micro_usd` | `int \| None` | Average per-item cost in **micro-USD** (not floored) |
| `n_succeeded` | `int \| None` | Rows that scored without error |
| `error_count` | `int \| None` | Rows that errored |

The point of a result row is the comparison: read `quality_mean` against the
confidence interval to know whether `"auto"` genuinely matches the frontier on
your data, and `mean_cost_micro_usd` to see what each call costs.

```python
auto_row = next(r for r in run.results if r.model_id == "auto")

for r in run.results:
    if r.kind != "frontier":
        continue
    matches = (
        auto_row.quality_ci_high is not None
        and r.quality_mean is not None
        and auto_row.quality_ci_high >= r.quality_mean
    )
    print(f"{r.model_id}: q={r.quality_mean:.3f} at {r.mean_cost_micro_usd} µUSD/item"
          f" — auto ({auto_row.quality_mean:.3f}) "
          f"{'matches it within the CI' if matches else 'trails it'}")
```

## Money: `.cost` vs `.cost_micro_usd`

Money on these objects follows one convention (SDK_PLAN §6): the **billed total
is floored to whole cents**, while sub-cent unit rates stay in micro-USD. The SDK
floors rather than rounds, so it never overstates a charge.

Three fields, two representations:

- `run.cost` is a `Decimal` in **dollars, floored to cents**. A 5 µUSD run reads
  `Decimal("0.00")`; a 420,715 µUSD run reads `Decimal("0.42")`. This is what the
  org is billed.
- `run.cost_micro_usd` is the **raw integer** in micro-USD. `1_000_000` = `$1.00`.
  Use it when you need the exact charge below cent precision.
- Per-item **unit rates** stay in micro-USD on purpose:
  `result.mean_cost_micro_usd` (and the `cost_micro_usd` on an
  `auto.compare_frontier()` result). Flooring a fraction-of-a-cent unit rate to
  whole cents would collapse it to zero and erase the auto-vs-frontier
  comparison that is the whole reason you ran the eval.

```python
from decimal import Decimal

print(run.cost)                       # Decimal("0.42") — billed dollars, floored
print(run.cost_micro_usd)             # 420715 — raw micro-USD
assert run.cost == Decimal("0.42")

# Convert any micro-USD unit rate to dollars yourself when you want to display it:
mean = run.results[0].mean_cost_micro_usd        # e.g. 850 µUSD per item
print(f"${mean / 1_000_000:.6f} per item")       # $0.000850 per item
```

Both inference and evals debit the org balance on success; an empty balance
raises `InsufficientCreditsError`. The SDK only ever consumes credit and surfaces
the 402; topping up is browser-only.

## See also

- [Running inference](../guide/inference.md) — `ChatCompletion`, streaming chunks, and the async iterator form
- [tasks](./tasks.md) — `Task` and `TaskMatch` in depth, and the dataset-to-contract flow
- [Evaluating models](../guide/evaluation.md) — building `EvalSet`s, running evals, and reading `EvalRun` cost
- [Core concepts](../guide/core-concepts.md) — the auto story, hidden hardware, and metering, end to end



---

<!-- reference/http-api.md -->

# Underlying HTTP API

The Pareta SDKs (Python and TypeScript) are thin, typed wrappers over a plain JSON-over-HTTPS API
served at `https://api.pareta.ai` under the `/v1/` prefix. Every method you call
maps to exactly one route (a couple of ergonomic helpers fan out to two or
three). This page is the lookup table: for each SDK method, the HTTP method,
path, request shape, and response shape it wraps.

Reach for it when you are debugging a request in a proxy log, calling Pareta from
a language without an SDK, or you just want to know what goes over the wire.
Everywhere else, prefer the SDK: it handles auth, retries, SSE parsing, and the
cost flooring convention for you.

A few platform truths shape every route below:

- **One model id.** Inference takes `model: "auto"` — the only entry
  `GET /v1/models` returns. Every request is planned, routed to
  benchmark-proven open specialists, verified, and falls back to a frontier
  model when that's the right call. There is nothing to deploy, and no GPU,
  hardware, or model knob anywhere on the wire; Pareta resolves all of it
  server-side, per request.
- **The models behind `auto` stay behind `auto`.** Which model served a given
  request never crosses this boundary. Frontier (vendor) ids appear in the
  clear only where you pick them deliberately: eval baselines and the frontier
  compare.
- **Inference and evals are metered against your org balance.**
  `POST /v1/chat/completions` debits **once per request** — however many
  internal model calls auto's plan makes, orchestration overhead is Pareta's
  cost, not yours. `POST /v1/eval-runs` debits for the auto and frontier
  compute it runs; `POST /v1/playground/frontier` debits at the vendor's
  actual token cost. The speech routes (`/v1/audio/*`) bill **per minute** of
  audio. An empty balance returns `402`. Top-up is browser-only; there is no
  balance or payment route.
- **Inference is OpenAI-compatible.** `/v1/chat/completions` and `/v1/models`
  speak the OpenAI wire format, so existing OpenAI clients point at Pareta by
  swapping the base URL and key.

## Base URL and versioning

| | |
|---|---|
| Base URL | `https://api.pareta.ai` (override with `PARETA_BASE_URL`) |
| Prefix | `/v1/` |
| Content type | `application/json` (JSON bodies); `multipart/form-data` for uploads |
| Streaming | `text/event-stream` (chat streaming) |

The SDK normalizes the base URL with `rstrip("/")`, so a trailing slash is
harmless.

## Authentication

Every request carries a bearer token in the `Authorization` header. The token is
your `pareta_sk_…` secret key, minted in the dashboard.

```
Authorization: Bearer pareta_sk_…
User-Agent: pareta-python/<version>
Accept: application/json            # or text/event-stream for streaming routes
Content-Type: application/json      # JSON bodies only; multipart sets its own
```

The SDK reads the key from the `api_key=` argument or the `PARETA_API_KEY`
environment variable. Prefer `Pareta.from_env()`, which reads both
`PARETA_API_KEY` and the optional `PARETA_BASE_URL`:

```python
from pareta import Pareta

# Reads PARETA_API_KEY (+ optional PARETA_BASE_URL) from the environment.
with Pareta.from_env() as pa:
    print([m.id for m in pa.models.list()])
```

A raw `curl` against the same route:

```bash
curl https://api.pareta.ai/v1/models \
  -H "Authorization: Bearer $PARETA_API_KEY"
```

Constructing a client with no key raises `ParetaError` before any request goes
out. A key that reaches the server and is rejected returns `401`
(`AuthenticationError`). See [Errors, retries & timeouts](../guide/errors-and-retries.md).

## Route map at a glance

| SDK call | Method | Path |
|---|---|---|
| `chat.completions.create(...)` | `POST` | `/v1/chat/completions` |
| `models.list()` | `GET` | `/v1/models` |
| `tasks.list()` | `GET` | `/v1/tasks` |
| `tasks.retrieve(id)` | `GET` | `/v1/tasks/{id}` |
| `tasks.match(query)` | `POST` | `/v1/tasks/match` |
| `auto.metrics()` | `GET` | `/v1/auto/metrics` |
| `auto.compare_frontier(...)` | `POST` | `/v1/playground/frontier` |
| `audio.transcriptions(audio)` | `POST` | `/v1/audio/transcriptions` |
| `audio.speech(text)` | `POST` | `/v1/audio/speech` |
| `evals.frontier_models(task)` | `GET` | `/v1/eval/frontier-models` |
| `evals.sets.create(...)` | `POST` | `/v1/eval-sets` |
| `evals.sets.list()` | `GET` | `/v1/eval-sets` |
| `evals.sets.retrieve(id)` | `GET` | `/v1/eval-sets/{id}` |
| `evals.sets.delete(id)` | `DELETE` | `/v1/eval-sets/{id}` |
| `evals.sets.upload_document(...)` | `POST` | `/v1/eval-sets/{id}/attach-blob` (small) or `/blob-upload-url` + `PUT` + `/blob-upload-complete` (large) |
| `evals.runs.create(...)` | `POST` | `/v1/eval-runs` |
| `evals.runs.retrieve(id)` / `evals.runs.wait(id)` | `GET` | `/v1/eval-runs/{id}` |

## Inference: chat completions

### `POST /v1/chat/completions`

OpenAI-compatible chat completions. Wrapped by
[`chat.completions.create()`](../guide/inference.md). Metered: a successful
completion debits the org balance **once per request** — however many internal
model calls auto's plan makes, orchestration overhead is Pareta's cost, not
yours — and an empty balance returns `402` (`InsufficientCreditsError`).

`model` is `"auto"`. Extra OpenAI fields (`temperature`, `max_tokens`,
`top_p`, ...) pass straight through as body fields.

Request body:

```json
{
  "model": "auto",
  "messages": [{"role": "user", "content": "Extract the parties."}],
  "temperature": 0.0
}
```

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    resp = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Extract the parties."}],
        temperature=0.0,
    )
    print(resp.choices[0].message.content)   # ChatCompletion -> Choice -> Message
    print(resp.usage.total_tokens)           # Usage
```

The same request as `curl`:

```bash
curl https://api.pareta.ai/v1/chat/completions \
  -H "Authorization: Bearer $PARETA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Extract the parties."}]
  }'
```

#### Cost receipt headers

Every non-streamed completion carries its own receipt (micro-USD integers,
same unit as the agent lane's billing header):

| Header | Meaning |
| --- | --- |
| `X-Pareta-Billed` | What this request debited. Follows the ledger exactly — an idempotent retry (same `Idempotency-Key`) that replays a prior debit reads `0`. |
| `X-Pareta-Frontier-Would-Have-Cost` | The counterfactual: what one list-priced frontier call on the same prompt would have cost. Billed vs. this is your per-request savings. |

```bash
curl -si https://api.pareta.ai/v1/chat/completions ... | grep -i x-pareta
# x-pareta-billed: 653
# x-pareta-frontier-would-have-cost: 11795
```

Streamed responses carry the same two numbers as SSE comment lines right
before `data: [DONE]` (comments are invisible to OpenAI SDK parsers, so
stock clients are unaffected):

```
: pareta-billed-micro-usd 653

: pareta-frontier-would-have-cost-micro-usd 11795

data: [DONE]
```

#### Streaming

Set `"stream": true`. The response is a data-only SSE stream in vLLM format:
each `data:` line is one JSON chunk, and the stream ends with `data: [DONE]`.

```
data: {"choices": [{"delta": {"content": "The"}}]}
data: {"choices": [{"delta": {"content": " parties"}}]}
data: [DONE]
```

The SDK yields `ChatCompletionChunk` objects;
`chunk.choices[0].delta.content` is the incremental text.

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    for chunk in pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Summarize the contract."}],
        stream=True,
    ):
        piece = chunk.choices[0].delta.content
        if piece:
            print(piece, end="", flush=True)
```

Retries cover only the initial handshake. Once SSE bytes are flowing a
mid-stream drop raises immediately (`APIConnectionError`) and cannot be resumed.

### `GET /v1/models`

OpenAI-compatible model listing. Wrapped by `models.list()`. Returns exactly
one entry — `"auto"` — shaped as
`{"data": [{"id", "owned_by", "created"}, ...]}`. The `id` is what you pass to
`chat.completions.create(model=...)`.

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    models = pa.models.list()          # ModelList (iterable, has len)
    for m in models:
        print(m.id, m.owned_by)        # Model
```

## Tasks (benchmark catalog)

### `GET /v1/tasks`

List the benchmark catalog. Wrapped by `tasks.list()`. The server returns
`{"tasks": [...]}`; the SDK maps each to a `Task` (`id`, `default_scorer`,
`has_blob_input`).

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    for t in pa.tasks.list():
        print(t.id, t.default_scorer, t.has_blob_input)
```

### `GET /v1/tasks/{task_id}`

Retrieve one task's schema and default scorer. Wrapped by
`tasks.retrieve(task_id, examples_n=None)`. The optional `examples_n` query param
requests N example items when available.

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    task = pa.tasks.retrieve("contract-key-fields", examples_n=3)
    print(task.id, task.has_blob_input)
```

### `POST /v1/tasks/match`

Map free-text intent to one match. Wrapped by `tasks.match(query, top_k=5)`. The
matcher is an LLM reasoning router that maps intent to a benchmarked task, a
general capability lane (`"capability:<id>"`), or `"unsupported"`, degrading to a
keyword scorer if the router is unavailable. An empty `query` raises `ValueError`
client-side. The response keeps the legacy keys (`matched`/`chosen`/`candidates`/
`ambiguous`/`matcher`) and adds `type` (`"task"`/`"capability"`/`"unsupported"`/
`"none"`), `reasoning`, and `capability` (when `type == "capability"`).

Request body:

```json
{"query": "pull key fields out of vendor contracts", "top_k": 5}
```

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    match = pa.tasks.match("pull key fields out of vendor contracts")
    if match.matched and match.chosen:
        print(match.chosen.task_id, match.chosen.confidence)
    for c in match.candidates:        # ranked alternates
        print(c.task_id, c.score)
```

## Auto (metrics and frontier compare)

The routes around the `model: "auto"` call itself: an org-level rollup of your
auto traffic, and a metered one-prompt comparison against a frontier vendor.
The SDK wraps both in the `pa.auto` namespace.

### `GET /v1/auto/metrics`

Your org's `model: "auto"` traffic, rolled up. Wrapped by `auto.metrics()`,
which returns the raw JSON dict: requests + success rate (30d), billed spend,
hourly p50/p95/error buckets (7d), daily success cells (30d), and the
**projected** savings vs frontier (a frontier list-priced counterfactual,
labeled as projected in the dashboard too).

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    m = pa.auto.metrics()
    print(m["requests_30d"], m["success_rate_30d"])
    print(m["savings_vs_frontier_micro_usd_30d"])   # projected, micro-USD
```

### `POST /v1/playground/frontier`

Run one prompt against a frontier vendor for a side-by-side with
`model: "auto"`. Wrapped by `auto.compare_frontier(model=..., messages=...)`.
Allowed `model` values: `gpt-5.5`, `gemini-3-5-flash`, `gemini-3-1-pro`,
`claude-sonnet-4-6` (anything else returns `400`); `messages` takes 1–40
entries. Metered at the vendor's **actual token cost** — one debit per call; a
failed vendor call returns `502` and bills $0. An empty balance returns `402`.

Request body:

```json
{
  "model": "gpt-5.5",
  "messages": [{"role": "user", "content": "Extract the parties."}]
}
```

Returns `{"model": ..., "content": ..., "cost_micro_usd": ..., "latency_ms": ...}`.

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    side = pa.auto.compare_frontier(
        model="gpt-5.5",
        messages=[{"role": "user", "content": "Extract the parties."}],
    )
    print(side["content"], side["cost_micro_usd"], side["latency_ms"])
```

## Speech (audio)

The Speech capability lanes (`asr`, `tts`) run on dedicated services, not the
chat-completions path, so they have their own routes. The SDK wraps them in the
`pa.audio` namespace (`pa.audio.transcriptions(...)` / `pa.audio.speech(...)`);
the routes below are what those methods call. Both are metered **per minute** of
audio and return `402` (`InsufficientCreditsError`) on an empty balance.

### `POST /v1/audio/transcriptions`

Speech-to-text (the `asr` lane). Body is JSON with base64 audio:

```json
{"audio_base64": "<base64 wav/mp3/m4a/webm>", "language": "en"}
```

`language` is optional. Returns `{"text": ..., "language": ..., "duration_s": ...}`;
debits per minute of **input** audio.

```bash
curl https://api.pareta.ai/v1/audio/transcriptions \
  -H "Authorization: Bearer $PARETA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"audio_base64\": \"$(base64 -i call.wav)\"}"
```

### `POST /v1/audio/speech`

Text-to-speech (the `tts` lane). Body is JSON:

```json
{"text": "Hello from Pareta.", "voice": "<optional kokoro voice id>"}
```

`text` is required (max 5000 chars); `voice` is optional (omit for the default
Kokoro voice). Returns
`{"audio_base64": ..., "sample_rate": ..., "duration_s": ..., "format": ...}`;
debits per minute of **output** audio.

```bash
curl https://api.pareta.ai/v1/audio/speech \
  -H "Authorization: Bearer $PARETA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello from Pareta."}'
```

## Evals

### `GET /v1/eval/frontier-models`

The vendor frontier roster you can evaluate against. Wrapped by
`evals.frontier_models(task=None)`. The server returns
`{"frontier_models": [...]}`; the SDK maps each to a `FrontierModel`
(`id`, `vendor`, `vision`, `benchmarked`). Pass `task` to annotate `benchmarked`
(already benchmarked on that task) and vision-filter for document tasks. Feed
the ids into `evals.runs.create(frontier=[...])`.

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    roster = pa.evals.frontier_models(task="contract-key-fields")
    for fm in roster:
        print(fm.id, fm.vendor, fm.vision, fm.benchmarked)
```

### `POST /v1/eval-sets`

Create an eval set from your rows. Wrapped by
[`evals.sets.create(task=..., items=...)`](../guide/evaluation.md). The rows go
over the wire as **JSONL** inside a `multipart/form-data` body (`items` file part
plus `task_id` and `name` form fields), not as a JSON array. An empty `items`
raises `ValueError`. The server returns `{"eval_set": {...}}`; the SDK maps it to
an `EvalSet` (`id`, `task_id`, `name`, `item_count`, `scoring_strategy`).

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    eval_set = pa.evals.sets.create(
        task="contract-key-fields",
        items=[
            {"input": "Agreement between A and B...", "expected": {"parties": ["A", "B"]}},
            {"input": "This SOW is by C for D...",     "expected": {"parties": ["C", "D"]}},
        ],
    )
    print(eval_set.id, eval_set.item_count, eval_set.scoring_strategy)
```

### `GET /v1/eval-sets` and `GET /v1/eval-sets/{eval_set_id}`

List your eval sets, or retrieve one. Wrapped by `evals.sets.list()` (server
returns `{"eval_sets": [...]}`) and `evals.sets.retrieve(eval_set_id)` (server
returns `{"eval_set": {...}}`). Both map to `EvalSet`.

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    for es in pa.evals.sets.list():
        print(es.id, es.name, es.item_count)
    one = pa.evals.sets.retrieve("evset_123")
```

### `DELETE /v1/eval-sets/{eval_set_id}`

Delete an eval set. Wrapped by `evals.sets.delete(eval_set_id)`, which returns
`None`.

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    pa.evals.sets.delete("evset_123")
```

### Uploading documents to a row (3 routes)

For document/image tasks, attach a binary blob to one row's input field. The SDK
collapses two upload paths into a single
`evals.sets.upload_document(eval_set_id, file, *, idx, field_name, mime=None)`
call. `file` may be a path, raw `bytes`, or a binary file-like; anything else
raises `TypeError`. `idx` is the 0-based row, `field_name` the blob input field.

The SDK picks the path by size:

- **Files under 5 MiB** go inline through
  `POST /v1/eval-sets/{id}/attach-blob` (`multipart/form-data`: the `file` part
  plus `idx`, `field_name`, `mime` form fields).
- **Larger files** use the signed-URL flow: mint a URL with
  `POST /v1/eval-sets/{id}/blob-upload-url`, `PUT` the bytes directly to storage
  (GCS), then confirm with `POST /v1/eval-sets/{id}/blob-upload-complete`.

Either way the method returns the response dict from the terminal call.

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    eval_set = pa.evals.sets.create(
        task="document-extraction",
        items=[{"expected": {"invoice_total": "1240.00"}}],
    )
    # Attach the PDF that row 0's blob field expects.
    pa.evals.sets.upload_document(
        eval_set.id, "invoice.pdf", idx=0, field_name="document"
    )
```

### `POST /v1/eval-runs`

Start an eval run. Wrapped by
[`evals.runs.create(...)`](../guide/evaluation.md). Pass either an existing
`eval_set=<id>` or an inline `task=...` + `items=...` (which the SDK turns into an
eval set first). `models` is the list of candidate ids to evaluate — pass
`["auto"]`; `frontier` adds vendor baselines.

The SDK resolves `frontier` to a list of ids before sending, then posts
`{"eval_set_id": ..., "candidate_model_ids": ["auto", ...frontier...]}`:

| `frontier=` value | Resolves to |
|---|---|
| `None` or `"none"` | `[]` (no baselines) |
| list of ids | the list, as-is |
| `"all"` | every id from `GET /v1/eval/frontier-models?task=...` |
| `"benchmarked"` | frontier models already benchmarked on the task |

A keyword (`"all"` / `"benchmarked"`) needs the task; if you passed `eval_set=`
only, the SDK looks up its `task_id` to resolve the roster, and raises
`ValueError` if the task is unknown. Metered: the org balance is debited for
auto and frontier compute, and an empty balance returns `402`.

The server responds with `{"run_id": ..., "status": ...}`. With `wait=False`
the SDK returns an `EvalRun` in its initial (running/queued) state. With
`wait=True` it polls `GET /v1/eval-runs/{run_id}` every `poll_interval` seconds
(default 3.0) until terminal, up to `timeout` seconds (default 900.0), then
returns the final `EvalRun`; exceeding the deadline raises `ParetaError` while the
run keeps going server-side.

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    run = pa.evals.runs.create(
        task="contract-key-fields",
        items=[{"input": "Agreement between A and B...", "expected": {"parties": ["A", "B"]}}],
        models=["auto"],                        # the candidate under test
        frontier="benchmarked",                 # vendor baselines benchmarked on the task
        wait=True,
    )
    print(run.status, run.cost)                 # "completed" Decimal("0.42")
    for r in run.results:                       # EvalResult per model
        print(r.model_id, r.kind, r.quality_mean, r.mean_cost_micro_usd)
```

### `GET /v1/eval-runs/{run_id}`

Retrieve full run state, including per-model results once terminal. Wrapped by
`evals.runs.retrieve(run_id)` and the `evals.runs.wait(run_id, ...)` poll helper
(same semantics as `create(..., wait=True)`). The server returns an envelope
`{"run": {...}, "results": [...]}` that the SDK maps to an `EvalRun`.

`EvalRun.cost` is the billed total as `Decimal` dollars **floored to cents**
(never rounded up), while `EvalRun.cost_micro_usd` keeps the raw integer
micro-USD value. A 5 micro-USD run reads `Decimal("0.00")`. Per-item unit rates
such as `EvalResult.mean_cost_micro_usd` stay in micro-USD so the auto-vs-frontier
comparison is not erased by flooring.

```python
from pareta import Pareta

with Pareta.from_env() as pa:
    run = pa.evals.runs.retrieve("run_456")
    if run.is_terminal:                         # status in ("completed", "failed")
        print(run.cost, run.cost_micro_usd)
        if run.status == "failed":
            print(run.error_detail)
    else:
        run = pa.evals.runs.wait("run_456", poll_interval=5.0, timeout=600.0)
```

## Status codes

The server is FastAPI, so error bodies are `{"detail": "<message>"}` with a
standard HTTP status. The SDK maps each status to a specific
`ParetaError` subclass so you catch by meaning.

| Status | Exception | When |
|---|---|---|
| 400, 422 | `BadRequestError` | request validation failed |
| 401 | `AuthenticationError` | invalid or missing API key |
| 402 | `InsufficientCreditsError` | org out of balance (top up in the dashboard) |
| 403 | `PermissionDeniedError` | authenticated, not allowed |
| 404 | `NotFoundError` | eval set / run / task id not found |
| 409 | `ConflictError` | transient lock/contention |
| 429 | `RateLimitError` | rate limited |
| 503 | `EndpointNotReadyError` | a serving backend behind `auto` is warming or briefly unavailable (retried automatically) |
| other 5xx | `APIStatusError` | generic server error |

Each `APIStatusError` exposes `status_code`, `detail`, `request_id` (from the
`x-request-id` response header), and the underlying `response`. The SDK
automatically retries `408, 409, 429, 500, 502, 503, 504` with exponential
backoff that honors `Retry-After`. Full treatment in
[Errors, retries & timeouts](../guide/errors-and-retries.md).

## Async over the same routes

`AsyncPareta` hits the identical routes with awaitable methods. Streaming routes
return async iterators; `evals.runs.wait()` is a coroutine. The wire format,
auth, status mapping, and retry policy are the same.

```python
import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        models = await pa.models.list()                 # GET /v1/models
        async for chunk in await pa.chat.completions.create(  # POST /v1/chat/completions
            model="auto",
            messages=[{"role": "user", "content": "Extract the parties."}],
            stream=True,
        ):
            piece = chunk.choices[0].delta.content
            if piece:
                print(piece, end="", flush=True)

asyncio.run(main())
```

## See also

- [Inference](../guide/inference.md) — OpenAI-compatible chat completions and streaming
- [Evaluation](../guide/evaluation.md) — eval sets, runs, `wait`, and `run.cost`
- [Tasks](tasks.md) — the benchmark catalog and `match()`
- [Errors, retries & timeouts](../guide/errors-and-retries.md) — the full exception hierarchy
- [Async](../guide/async.md) — the `AsyncPareta` client end to end
- [Configuration](../guide/configuration.md) — base URL, keys, timeout, and retry budget



---

<!-- reference/agent-api.md -->

# Agent API (`/agent/v1`) — OpenClaw and agent runtimes

The wire reference for Pareta's agent surface: an OpenAI-compatible chat
completions endpoint built for multi-turn tool loops. If you want the
narrative version — what the lane does per turn and why — read
[Connect OpenClaw to Pareta](../guide/agent-openclaw.md) first; this page is
the contract.

Base URL: `https://api.pareta.ai/agent/v1`. Authentication is a Bearer
`pareta_sk_…` key on every request, same as `/v1`.

## `GET /agent/v1/models`

Returns the one model the surface serves. Agent runtimes that size their
context accounting from `/models` (OpenClaw, Hermes) read the vLLM-style
extension fields.

```json
{
  "object": "list",
  "data": [{
    "id": "auto",
    "object": "model",
    "owned_by": "pareta",
    "created": 1784130000,
    "max_model_len": 131072,
    "context_window": 131072
  }]
}
```

## `POST /agent/v1/chat/completions`

One conversation turn. The transcript and tool schemas pass through to a
turn-routed fleet member verbatim; the response comes back in OpenAI shape
with `model: "auto"`.

### Request body

| Field | Behavior |
| --- | --- |
| `model` | `"auto"` or omitted. Any other value → `400`. |
| `messages` | Full transcript — `system`, `user`, `assistant` (including prior `tool_calls`), and `tool` results all pass through. Nothing is dropped or rewritten. |
| `tools`, `tool_choice`, `parallel_tool_calls` | OpenAI function-calling shape, passed through; `tool_calls` come back in the same shape. |
| `stream`, `stream_options` | SSE streaming (below). `include_usage` is forced on — the final chunk always carries `usage`. |
| `temperature`, `top_p`, `max_tokens`, `max_completion_tokens`, `stop`, `seed`, `presence_penalty`, `frequency_penalty`, `logprobs`, `top_logprobs`, `response_format` | Forwarded verbatim — the agent owns its sampling. `max_tokens` defaults to 8192 when omitted. |
| `n` | Only `1` (or omitted). `n > 1` → `400`. |
| anything else | Ignored, never a `400` — runtimes that attach vendor `extra_body` fields work unmodified. |

Context window: **131,072 tokens** per turn.

### Request headers

| Header | Behavior |
| --- | --- |
| `Authorization` | `Bearer pareta_sk_…` — required. |
| `Idempotency-Key` | Optional. A retried turn with the same key bills once. |
| `X-Pareta-Session` | Optional explicit conversation id. Pareta pins a conversation's route so consecutive turns don't re-derive routing (and an escalation can stick); without this header the pin key is derived from your org + system prompt + tool-schema names — the stable prefix agent runtimes re-send every turn. Pins expire after ~30 minutes idle; a modality change (an image turn in a text conversation) re-routes. |

### Response

A standard OpenAI chat completion. `model` is always `"auto"` — real model
ids never appear anywhere in the response. `usage` carries prompt and
completion tokens.

| Header | Meaning |
| --- | --- |
| `X-Pareta-Billed` | The turn's debit in micro-USD (non-streamed responses). A turn bills once no matter how it was routed or escalated; a failed turn bills nothing. |

### Streaming

Set `"stream": true`. The response is `text/event-stream` with OpenAI chunk
deltas — `tool_calls` stream in OpenAI shape — followed by a final chunk
carrying `usage`, then `data: [DONE]`.

### Routing and escalation

Every turn is routed by shape behind the single `auto` string: general
reasoning, coding, or vision. A turn that comes back low-confidence is re-run
on a frontier model **with your tools intact** and that answer is returned;
conversations that escalate repeatedly pin to the frontier route. None of
this changes the wire shape or the one-debit-per-turn billing.

### Errors

| Status | Meaning |
| --- | --- |
| `400` | Malformed JSON body, `model` other than `"auto"`, or `n > 1`. |
| `401` | Missing or invalid API key. |
| `402` | Insufficient balance. |
| `404` | The agent surface is not enabled. |
| `502` | The turn could not be completed (never a raw 500). |
| `503` | Temporarily unavailable — retry after the `Retry-After` header (30s). |

## OpenClaw provider block

OpenClaw configures models through OpenAI-compatible providers in
`~/.openclaw/openclaw.json`. This is a working provider entry (field names
can drift across OpenClaw versions — the three values that matter are the
base URL, the key, and `model: auto`):

```json
{
  "models": {
    "providers": {
      "pareta": {
        "baseUrl": "https://api.pareta.ai/agent/v1",
        "apiKey": "pareta_sk_…",
        "api": "openai-completions",
        "models": [{
          "id": "auto",
          "name": "Pareta Auto",
          "input": ["text", "image"],
          "contextWindow": 131072,
          "maxTokens": 8192
        }]
      }
    }
  }
}
```

Then point any agent role at it — as the primary model, keep a local
fallback if you like:

```json
{
  "agents": {
    "defaults": {
      "model": {
        "primary": "pareta/auto",
        "fallbacks": ["ollama/gpt-oss:120b"]
      }
    }
  }
}
```

Restart the OpenClaw gateway and every turn — chat, tool calls, images —
routes through Pareta. Because `auto` routes per turn, this one provider
entry covers coding, general reasoning, and vision; there is nothing else to
wire up.

