Metadata-Version: 2.4
Name: runlatent
Version: 0.3.3
Summary: Latent SDK: one call at startup posts every finished OpenAI / Anthropic / Google GenAI / Ollama call to Latent's scoring API, fire-and-forget.
License: Proprietary
Project-URL: Homepage, https://runlatent.ai
Project-URL: Documentation, https://github.com/runlatentai/core/blob/main/runlatent/README.md
Project-URL: Source, https://github.com/runlatentai/core
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# runlatent — ingest finished LLM calls into Latent's scoring API

```
pip install runlatent
```

A Python SDK that captures every finished call your app makes through the OpenAI, Anthropic,
Google GenAI, Ollama, Groq, Together, Fireworks, Mistral, Cohere, Azure AI Inference, Hugging Face or boto3
Bedrock Python SDK (and every OpenAI-compatible endpoint through the `openai` client: the provider
matrix with the `base_url` recipe per host is `docs/sdk-providers.md`) and posts the (prompt, answer)
pair to Latent's scoring API, which
is the sidecar's `POST /score` contract (`latent_sidecar/README.md`). The app's call is never
slower by a network round trip, never sees a different return value and never gets an
exception from here. Step 1 of the self-serve path: no gateway, no log tail, one line at startup.

```
your app ──openai / anthropic / google-genai / ollama SDK──▶ the provider
   │ (after the SDK returned, fire-and-forget, a daemon thread)
   └──POST <LATENT_API_URL>/score {request_id, messages, output, model, finish_reason}──▶ Latent
```

## Install

`pip install runlatent` (the `runlatent` distribution on PyPI, built from `runlatent/` in this
repository; from the repository, `pip install -e .`). It has no runtime dependencies: nothing
from numpy, torch or the core packages reaches your process. The provider SDKs are yours:
whichever of `openai`, `anthropic`, `google-genai`, `ollama`, `groq`, `together`, `fireworks-ai`, `mistralai`,
`cohere`, `azure-ai-inference`, `huggingface_hub`, `boto3` is importable gets patched, the others are
skipped (`docs/sdk-providers.md` lists what each one covers and the hosts behind the `openai` client).

## One-line setup

```python
import runlatent
runlatent.auto_instrument()        # reads LATENT_API_URL and LATENT_API_KEY
```

Call it once, before the clients are built or after, it does not matter. Supported SDKs: OpenAI,
Anthropic, Gemini, Ollama (and the rest of the list above). The patch is on the SDK classes
(`Completions.create`, `AsyncCompletions.create`, `Responses.create`, `AsyncResponses.create`;
`Messages.create` / `.stream` and the async pair; `Models.generate_content` / `.generate_content_stream`
and `AsyncModels`; ollama's `Client.chat` / `.generate` and `AsyncClient`, plus the module-level
`ollama.chat` / `ollama.generate`, which are bound to the module's own client at import), so every
client in the process is covered, including ones a framework builds for you. Keyword arguments:

| argument | default | meaning |
|---|---|---|
| `openai`, `anthropic`, `google_genai`, `ollama`, `groq`, `together`, `fireworks`, `mistralai`, `cohere`, `azure_ai_inference`, `huggingface_hub`, `bedrock` | `True` | which SDKs to patch (`docs/sdk-providers.md`) |
| `api_url` | `$LATENT_API_URL` | the API base; the SDK posts to `<api_url>/score` |
| `api_key` | `$LATENT_API_KEY` | sent as `Authorization: Bearer <key>` |
| `redact` | `None` | `callable(body) -> body or None`, run on every pair before it is queued (below) |
| `timeout` | `5.0` | seconds per POST |
| `max_queue` | `10000` | pairs waiting for the poster thread; a full queue drops, counted |

Idempotent: a second call patches nothing twice, and with the same destination it keeps the
same poster and its queue; a call with a different `api_url` / `api_key` / `timeout` /
`max_queue` swaps the poster at once and drains the old one on its own thread (its final
counters stay in `stats()`), never on the caller and never while the app's calls wait.
`runlatent.uninstrument()` restores every original (classes and wrapped instances). With no URL anywhere, `auto_instrument()` logs one
warning, patches nothing and returns `stats()` with `enabled: False`.

**Any OpenAI-compatible endpoint.** The patch is on the `openai` client, not on OpenAI's servers, so any
OpenAI-compatible server (Ollama's `/v1`, vLLM's OpenAI server, LM Studio, a Hugging Face Inference
Endpoint or TGI, Together, Fireworks and the like) is captured the moment the app talks to it through the
`openai` client with `base_url` pointed at it, chat and streaming alike, with the model name the app passed:

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")   # Ollama's /v1; vLLM: http://<host>:8000/v1
client.chat.completions.create(model="llama3.2", messages=msgs)          # captured like any OpenAI call
```

**Ollama's native client** (`pip install ollama`) is captured as is, no URL recipe: `ollama.chat(...)`,
`ollama.generate(...)`, `ollama.Client().chat` / `.generate`, `ollama.AsyncClient`, `stream=True` on each
(the chunks are assembled). A `generate` call posts its `system` as the system turn and the `prompt` as one
user turn (a fill-in-the-middle `suffix` follows it under a `[suffix]` marker, a prior-turn `context` of token
ids is a counted `[context omitted]` part); the finish is `done_reason`, usage is `prompt_eval_count` / `eval_count`, and since Ollama returns
no response id the pair's `request_id` is the SDK's own (`client.response_id` is null). `embed` and the model
management calls are not answers and are not patched.

```python
import ollama
ollama.chat(model="llama3.2", messages=msgs)                   # captured; so is ollama.Client().chat(...)
```

Anything else (in-process `transformers` generation, Replicate, SageMaker `invoke_endpoint`, a client the SDK
does not patch) uses `runlatent.check` directly after each generation: one POST, synchronous, the verdict in
hand (below). Or serve the model through vLLM with Latent's in-band plugin and skip the SDK altogether:

```python
answer = tokenizer.decode(model.generate(**inputs)[0], skip_special_tokens=True)
r = runlatent.check(prompt, answer, context=docs, model="meta-llama/Llama-3.1-8B-Instruct")
if r.flagged: answer = SAFE_FALLBACK
```

## Check before you deliver: `runlatent.check`

`auto_instrument` and the wrappers observe: the pair is posted after the provider returned, on a
daemon thread, and the read never blocks or fails your call. To **hold, replace or re-route** a
flagged answer before your user sees it, read it synchronously first. On hosted plans
`POST /score` answers the verdict in tens of milliseconds (the hosted service: `docs/hosted-score.md`),
so the whole pattern is two lines:

```python
r = runlatent.check(prompt, answer, context=docs)
if r.flagged: answer = SAFE_FALLBACK
```

`check` is the explicit way to hold before delivery on hosted plans; in front of your own vLLM the gateway
does it automatically from the in-band event. `auto_instrument` holds, replaces or re-routes only under the console
policy (its default `enforce="policy"`, the section "Act from the console" below); with `enforce=False` it never
changes a response.

```python
runlatent.check(prompt, answer, *, context=None, system=None, messages=None, model=None,
                timeout=2.0, request_id=None, raise_on_error=False, detail=False) -> CheckResult
await runlatent.acheck(...)      # the same read, awaited (run off the event loop)
```

| argument | meaning |
|---|---|
| `prompt` | the user's question (one user turn) |
| `answer` | the text about to be delivered; empty -> `verdict: "unscored"`, nothing posted |
| `context` | the retrieved sources; rendered before the question as `Knowledge: <context>\n<prompt>`, the shape the probes are calibrated on. Pass it: an answer with no source (no `context`, no `Knowledge:` line, no `system`) is stored and shown but gets no grounding verdict and is never flagged (`verdict: "unscored"`, the server's `unscored_reason: "no_source"`) |
| `system` | the system prompt, a string or Anthropic's block list, posted as a leading system turn (flattened as the capture flattens it) |
| `messages` | the conversation as the provider saw it (`[{role, content}, ...]`), instead of `prompt` / `context` |
| `model` | the generating model's name, for the console |
| `timeout` | seconds for the whole exchange (default 2.0); no reply in time is a refusal, never an exception |
| `request_id` | the console row's id (the provider's response id when you have one; else a uuid) |
| `raise_on_error` | `True` raises `runlatent.CheckError` (with `.result`) on a refusal instead of returning it |
| `session_id` / `user_id` | the call or conversation this turn belongs to and its end user (your own opaque ids, never personal data); else the ambient `runlatent.session(...)` block, else the fingerprint inferred from `messages` ("Group the turns of a call", below) |
| `detail` | `True` asks the same read for its metadata in the same reply (`detail: true` in the body; no second pass): `r.sentences`, `r.suite` / `r.suite_fired`, `r.threshold` and the raw block `r.read` (below, "Console-free use") |

It uses the destination and key `auto_instrument()` / `configure()` / the environment set
(`LATENT_API_URL`, `LATENT_API_KEY`), posts the same body `auto_instrument` posts
(`client.api: "check"`) through the same `redact` hook, and the route stores the pair in your
console like any other scored answer. With neither a URL nor a key configured it raises
`runlatent.ConfigurationError` at once, whatever `raise_on_error` says: nothing can be checked and
you need to know.

`CheckResult` is a small frozen dataclass:

| field | meaning |
|---|---|
| `risk` | the probe's probability that the answer is unreliable (`None` when not scored) |
| `verdict` | `pass` / `escalate` / `ood` from the reader; `unscored` (an empty answer, or `redact` returned None); `unknown` (the read did not happen, or came back unreadable) |
| `request_id` | the console row (`/events/<request_id>`) |
| `ood_distance` | the OOD guard's distance, when the reader sent one |
| `scored` | `True` exactly when the server answered 2xx with a risk and a verdict the SDK knows |
| `status` | the HTTP status; `0` when no reply came back (timeout, connection error, local skip) |
| `detail`, `resume` | the server's `detail` and how-to-resume sentence on a refusal (`limit_reached`, `rate_limited`, `unauthorized: ...`), or the transport error |
| `retry_after_s` | the server's `Retry-After` on a 429 / 503 |
| `error` | the exception class on a transport failure (`TimeoutError`, `ConnectionRefusedError`) |
| `.flagged` | `verdict in ("escalate", "ood")`: do not deliver as is |
| `.passed` | scored and `pass` |
| `.refused` | the read did not happen or was unreadable (`verdict == "unknown"`) |
| `read` | with `detail=True`: the server's raw `detail` block (`docs/hosted-score.md`, "The detail block"), the turn's `session` included when you passed `session_id` / `user_id` or an ambient `runlatent.session(...)`; else `None` |
| `sentences` | with `detail=True`: the sentence read, a tuple of frozen `Sentence(start, end, text, risk)` in answer order (`r.read["sentences_status"]` says why it is what it is: `"ok"`, `"absent"` = no sentence probe on the reader, `"failed"` = the configured read raised, with `r.read["sentences_reason"]`, `"skipped"` = nothing to read; a failed read is marked, never reported as absent) (`answer[start:end] == text` when `r.read["text_source"]` is `"output"`, which it is for a text-only check; `"decoded"` marks offsets into the reader's decoded answer when posted token ids did not round-trip; `risk` the sentence probe's probability that the sentence is unsupported); `None` when no sentence probe is attached to the reader, or without `detail` |
| `suite`, `suite_fired` | with `detail=True`: the probe heads' verdicts by name, frozen `SuiteHead(risk, threshold, over, enabled)` (grounding, arithmetic, a head calibrated on your traffic), and the names that fired; `None` / `()` when the artifact carries no heads, or without `detail` |
| `.threshold` | with `detail=True`: the cut the verdict was judged against; else `None` |

**A refusal never raises by default.** A 401 / 403 / 429 / 5xx, a timeout and a connection error
all return `scored=False` with the status and the server's words, so the fail-open / fail-closed
choice is yours and explicit:

```python
r = runlatent.check(prompt, answer, context=docs)
if not r.scored:                # the read did not happen: r.status, r.detail, r.resume say why
    deliver(answer)             # fail open: deliver unchecked (and alert on r.detail), or
    # answer = SAFE_FALLBACK    # fail closed: hold
elif r.flagged:
    answer = SAFE_FALLBACK
```

`if r.passed` is the fail-closed test in one word (a refusal is not a pass); `if not r.flagged` is
the fail-open one. `bool(r)` is `r.passed`, so `if runlatent.check(...)` holds a refusal AND an
empty answer (`unscored`); the properties say which. `raise_on_error=True` turns a refusal into
`runlatent.CheckError` for code that prefers an exception. A 2xx whose verdict is not one of
`pass` / `escalate` / `ood` is a refusal too (`scored=False`, `verdict unknown`, `error
ValueError`): a reply the SDK cannot read never delivers as "not flagged". The only other
exceptions are `ConfigurationError` (no URL or no key) and `ValueError` for a call that cannot
mean anything (`messages=` together with `prompt` / `context`; a `timeout` that is not positive).

**One scored answer, not two.** With `auto_instrument` on, the pair the instrumentation just
captured is scored once when the check can be paired with it. The pairing is by `request_id`
first (pass the provider's response id, `request_id=resp.id`: exact), then by the conversation
(`messages=` as the provider saw it, plus the answer, whitespace normalized; with Anthropic pass
`system=` too), never by the answer text alone: the grounding verdict depends on the prompt, so
"Paris" checked for "Capital of Germany?" must not inherit the pass of "Paris" captured for
"Capital of France?". Paired, `check` adopts the captured read's reply and its `request_id`
(`stats()["check_adopted"]`); a captured read still queued behind a slow one after half the
timeout is claimed and posted by `check` under the same id, the poster skipping its copy
(`skipped_checked`); one the poster is already sending, or retrying after a 429 / 5xx, is waited
for to the end of the timeout and adopted (`check_waited_in_flight`), never posted a second time;
a read that outlasts the timeout is a refusal (`status 0`, `error TimeoutError`, "still in
flight") and the poster's read lands in the console on its own. A check that runs before the
capture fires (Anthropic's `.stream()` manager: the app has the text inside the `with`) makes that
capture skip when it carries the same `request_id` (`get_final_message().id`) or the same
`messages=`. One capture pairs with one check, within 60 s, in a set bounded to 1,000 pairs.

What is not paired is scored twice, two console rows and two scored answers on the bill:
`check(prompt, answer, context=docs)` with no `request_id` next to `auto_instrument` (it renders
one user turn, `Knowledge: <context>\n<prompt>`, which equals the provider's messages only when
that is exactly what you sent; otherwise only the id pairs them: pass `resp.id`), and a stream
manager's check that carries neither the id nor the messages. A paired answer appears once in the
console and once on the bill.

### Console-free use: the risk score and its metadata in one reply

A service that wants no console, just "send the full request and get back a risk score plus its
metadata and classifications", asks for `detail`. The reply is the same read: nothing is computed
for the block, it is what the reader already had on the event (the cut, the OOD signal, the sentence
read, the probe heads, the finish, the token count, who read it), serialized. One POST, no second pass.

```python
r = runlatent.check(prompt, answer, context=docs, detail=True)
print(r.risk, r.verdict, r.threshold)             # 0.71 escalate 0.5
for s in r.sentences or ():                       # the sentence read, in answer order
    print(f"{s.risk:.2f}  {s.text}")              # 0.04  Paris is the capital of France.
                                                  # 0.91  It has two million people.
for name, head in (r.suite or {}).items():        # the classifications: grounding / arithmetic / your heads
    print(name, head.risk, head.over, head.enabled)
print(r.suite_fired)                              # ('grounding',)
```

The same with curl, against the hosted API (`docs/hosted-score.md`, "The detail block"; `?detail=1` for a
body you do not want to edit, the body's own `detail` winning when both are given):

```sh
curl -s https://api.runlatent.ai/score -H "Authorization: Bearer $LATENT_API_KEY" -H "Content-Type: application/json" \
  -d '{"request_id": "req-1", "messages": [{"role": "user", "content": "Knowledge: ...\nCapital of France?"}],
       "output": "Paris is the capital of France. It has two million people.", "detail": true}'
```

`r.sentences[i].text` is your own answer sliced at the read's sentence spans (echoed back, never stored
text), so the block is available on every plan. Not in the block, by design: the judge's rationale (the
console's request page), the per-token trace (its own keys on the reply, `token_risk` and friends) and the
explain record (`POST /explain` on the sidecar, `docs/explain.md`). Without `detail=True` the reply and the
result are exactly what they were.

A `detail=True` check never adopts a captured read (the instrumentation posts without `detail`, so the
adopted reply would carry none): when the poster has not started on the pair the check claims it and the
one post carries `detail` (`skipped_checked`); when the poster already sent it the check posts its own read
under the same `request_id` (`stats()["check_detail_reposted"]`), one console row and the answer scored and
billed twice. Pass `request_id=resp.id` and call `check` before the capture fires to keep it at one.
## Act from the console: the policy enforced in the SDK (`enforce="policy"`)

`check()` is for code you write. The console's **Act automatically** needs no code: `auto_instrument()` (and the
wrappers) default to `enforce="policy"`, which applies the project's policy to the answers your app is about to use.

```python
runlatent.auto_instrument()                 # enforce="policy": the console decides (the default)
runlatent.auto_instrument(enforce=False)    # never change a response; the console shows "enforcement disabled in the SDK"
```

How it works: the SDK polls your project's policy, `GET <LATENT_API_URL>/policy` with your key, every 60 s (ETag, so
an unchanged document costs one 304; a failed poll keeps the last document and backs off up to 5 min; a `401` drops it
and is logged once). The poller starts at `auto_instrument()` time and **no call ever waits for it**: until the first
document has arrived the SDK observes only (those pairs are posted with `enforce.reason: policy_not_loaded`), and
enforcement begins once the document is in and says to act. While the policy's Automatic response is **none** (or the
in-band plugin's early stop) nothing changes: the call path is the observe path, byte for byte (no read, no wait; the
pair is posted in the background as before). While it is **abstain**, **regenerate**, **route** or **judge**, every
non-streaming, single-sample captured call is scored **synchronously** before the response is returned to your app:
one `POST /score` within the policy's wait budget (`response.wait_ms`, default 500 ms, the same default as the
gateway's), then the decision the gateway takes (the same table, `runlatent/_decision.py`, a verbatim copy of
`latent_gateway/decision.py`) and the action in place:

| `response.action` | what your app gets |
|---|---|
| `abstain` | each choice's content replaced with the policy's `fallback_text` (ids, usage and the model kept; tool calls, function calls and logprobs dropped; a tool-call finish becomes `stop`); `response.latent.action == "abstain"` |
| `regenerate` | the same call re-issued once (`stream` off) and the NEW answer delivered: `latent.verdict == "regenerated"`, `latent.risk` null, the withheld answer's numbers under `original_risk` / `original_verdict`, `regenerated_id`; the new answer is posted to the console as its own row (`enforce.reason: reissued_answer`, never re-read before delivery). A re-issue that fails delivers the ORIGINAL flagged answer as `regenerate_failed` (the gateway's rule), unless the policy pins `response.fail_closed: true`, which holds it (the fallback text, `action: hold`) |
| `route` | the same call re-issued to `response.route_model` on the SAME provider (`latent.verdict == "routed"`, `routed_to`); a `route_model` of another provider (`anthropic:claude-...` under an OpenAI client) is `route_not_possible_in_sdk`, no model is `route_unconfigured`: the answer is delivered unchanged with its verdict. A re-issue that fails: as `regenerate`, `route_failed`, held under `fail_closed: true` |
| `judge` | delivered unchanged; the pair is queued for the console's explain judge through the report below |

The scope is **complete, single-choice answers**: a streamed answer and an `n > 1` call are delivered as the model
produced them, with the verdict, never changed (the console's Policy page says so beside Act automatically, and, when
every recent answer of yours was streamed, that the action is not applied to your traffic).
| below the flag cut | delivered unchanged, `latent.action == "none"`, `reason == "below_flag_cut"` |

**Providers covered.** OpenAI chat completions and Responses, Anthropic Messages, Google GenAI `generate_content`,
Ollama `chat` and `generate` (the native client): each provider's response shape is rewritten in place (OpenAI: every
choice's content, tool calls / function calls / logprobs dropped, a tool-call finish becomes `stop`; Anthropic: the first
text block kept with the replacement, tool-use and thinking blocks dropped, `stop_reason: tool_use` becomes `end_turn`;
GenAI: the first candidate's first text part, the other parts and candidates dropped; Ollama: `message.content` /
`response`, the message's `tool_calls` and the `thinking` text dropped, `model`, `done_reason`, the eval counts and a
`generate` call's `context` kept). A `route` on Ollama re-issues the call to the policy's model on the same client
(`llama3.1:70b` is a model tag, `ollama:llama3.1:70b` the same target spelled with its provider; `openai:gpt-5` from
an Ollama call is `route_not_possible_in_sdk`). A response shape the SDK cannot rewrite is delivered as the model
produced it with `latent.reason` ending in `+replace_failed` (counted `enforce_replace_failed`); the SDK never
substitutes a shape of its own.

**Latency.** Only the calls the policy acts on pay the synchronous read: one `/score` round trip, bounded by the
policy's wait budget (`response.wait_ms`, 500 ms default); the reader's own time depends on the deployment and is not
quoted here. A regenerate or a route adds one model call.
Streams never wait (a stream is never enforced; its capture says `stream_not_enforced:<action>`), `n > 1` calls never
wait (`n_not_enforced:<action>`), and with `enforce=False` nothing waits.

**Failure semantics are the policy's.** No read within the budget (a timeout, a 5xx, a 429, an empty answer): under a
policy that withholds flagged answers (`abstain`) the answer is HELD, the fallback text delivered with
`latent.action == "hold"`, `reason == "unmatched_timeout"` (fail-closed, the gateway's `auto` rule); under
`regenerate` / `route` / `judge` it is released unchanged with `latent.verdict == "unknown"` (fail-open);
`response.fail_closed: true | false` in the policy pins either. A read that did not come back is **re-posted in the
background** (hosted ingest is idempotent on `request_id`, so a row the slow reader stored anyway dedupes) and the
decision report waits for that post, retrying a `404` (the row not there yet) or an unreachable control plane after 1,
3 and 8 s, so the console shows the hold or the release on the row it was taken on (`enforce.reason:
read_not_returned` on the re-posted pair). A policy route that refuses the key (`401`) means nothing is enforced until
the key is accepted; a destination with no policy route at all (a sidecar: `404`) is said once and enforces nothing.
Your `redact` hook returning `None` for a pair the policy would have read is not silent either: under a withholding
policy (`abstain`) the answer is held (`latent.reason: redact_none`, the fallback text), under any other it is delivered
with `latent.reason: redact_none`; nothing is posted or reported for it, as the hook asked. **The app's calls are never
failed by any of this**: every failure on
the enforcement path (the read, the replacement, the report, the poller, a re-issue that raises) is swallowed and
counted (`stats()["errors"]`) and the provider's own response object is returned (pinned per failure path in
`tests/test_sdk_enforce.py`).

**What travels.** The decision is attached to the response as `latent` when the SDK's model takes an extra attribute
(openai's and anthropic's pydantic models do: `response.latent` / `response.model_extra["latent"]`; google-genai's
`GenerateContentResponse` and ollama's `ChatResponse` / `GenerateResponse` forbid extras, so there
`runlatent.last_decision()` is the way to read it, and the pair counts `enforce_block_not_attached`), and is always
`runlatent.last_decision()`: `{risk, verdict, action, reason, policy_version, tier, cut_basis, flag_cut, enforced_by:
"sdk", request_id, wait_ms, ...}`. Every posted pair carries `enforce: {by: "sdk", enabled, policy_version, applied,
reason}` so the console knows the SDK is enforcing (its Policy page then offers Act automatically) or that you turned
it off; each decision is reported to `POST <base>/score/action` (fire-and-forget, bounded) and shows on the request's
row as `action_applied`. `LATENT_SDK_ENFORCE=off` is the environment's spelling of `enforce=False`; an explicit
`enforce=` argument wins. With enforcement off the policy is not polled at all.

## Per-client wrappers

The same capture on one client object, nothing else in the process touched:

```python
client = runlatent.wrap_openai(OpenAI())          # or AsyncOpenAI(); any OpenAI-compatible base_url
client = runlatent.wrap_anthropic(Anthropic())    # or AsyncAnthropic(), AnthropicBedrock(), AnthropicVertex()
client = runlatent.wrap_genai(genai.Client(...))  # sync and .aio, Gemini API or vertexai=True
client = runlatent.wrap_ollama(ollama.Client())   # or ollama.AsyncClient(); chat and generate, streams included
client = runlatent.wrap_groq(Groq())              # likewise wrap_together, wrap_fireworks, wrap_mistral,
                                                  # wrap_cohere, wrap_azure_inference, wrap_huggingface,
                                                  # wrap_bedrock(boto3.client("bedrock-runtime"))
```

Each returns the client. Wrapping twice is a no-op; a wrapped instance under a patched class
captures once, not twice.

## Environment

| variable | meaning |
|---|---|
| `LATENT_API_URL` | the API base (the sidecar's address when you run it yourself: `http://sidecar:8090`) |
| `LATENT_API_KEY` | the bearer (the sidecar's `LATENT_SIDECAR_TOKEN`) |
| `LATENT_SIDECAR_URL`, `LATENT_SIDECAR_TOKEN` | accepted as fallbacks when the two above are unset. A sidecar has no `/policy` route: the policy poll answers `404`, said once at INFO, and nothing is enforced (the sidecar path is observe-only) |
| `LATENT_SDK_EXIT_FLUSH_S` | seconds the atexit handler waits for queued pairs at a normal interpreter exit (default `2.0`; `0` disables, see Limits) |
| `LATENT_SDK_ENFORCE` | `off` / `0` / `false` = never apply the console policy to a response (`auto_instrument(enforce=False)`); unset = `policy` ("Act from the console"); an explicit `enforce=` argument wins |

## What is posted

One JSON body per finished call, the sidecar's `/score` contract plus one extra block:

```json
{"request_id": "chatcmpl-abc123",
 "messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}],
 "output": "the full answer text",
 "model": "gpt-4.1", "finish_reason": "stop",
 "client": {"provider": "openai", "api": "chat.completions.create", "sdk": "openai", "sdk_version": "1.x",
            "runlatent": "0.3.3", "captured_at": "2026-10-02T12:00:00.000+00:00", "stream": false,
            "response_id": "chatcmpl-abc123", "finish_reason_raw": "stop",
            "usage": {"input_tokens": 120, "output_tokens": 48, "total_tokens": 168},
            "parts_dropped": 0, "items_dropped": 0}}
```

- `request_id` is the provider's response id (`chatcmpl-…`, `msg_…`, `resp_…`, Gemini's
  `response_id`) when it has one, else a uuid; the raw id is repeated in `client.response_id`.
- `messages` is the request as the app sent it, system prompt included, reduced to
  `{role, content}`: OpenAI `messages` (a `tool` turn is a text message; an assistant turn with
  `tool_calls` carries `[tool_calls omitted]`); the Responses API's `instructions` as `system`
  and its `input` string or message items (non-message items, function call outputs and the like,
  are skipped and counted in `items_dropped`; `previous_response_id` is recorded, the earlier
  turns are not fetched); Anthropic's `system` as `system` and `messages` (`tool_result` text
  kept, `thinking` blocks skipped); GenAI's `config.system_instruction` as `system` and `contents`
  (`model` role becomes `assistant`, loose strings and parts become one user turn).
- `output` is the whole answer text: the first choice's `message.content`; the Responses
  output's `message` items; Anthropic's `text` blocks; the first Gemini candidate's text parts
  (thoughts skipped). Non-text answer parts (a `tool_use` block, a Gemini `function_call`, an
  image) contribute nothing to it, so an answer with no text is not posted: counted
  `skipped_tool_only` when such parts were seen, `skipped_empty_output` otherwise (a blocked
  prompt). A Responses API response with `status: failed` or an `error` is not an answer and is
  never posted (`skipped_failed_response`), in the stream (`response.failed`) or not.
- `messages` is read BEFORE the SDK method runs, into fresh dicts and strings, sync and async
  alike. The pair is posted later (after the response, after a stream, after an await) but from
  that snapshot: an app that appends the assistant turn to its own list while the call is in
  flight or before the stream is exhausted does not leak the answer into the posted prompt. A
  one-shot iterable (a generator for `messages`, Responses `input`, Anthropic `messages` /
  `system`, GenAI `contents`) is not read ahead, which would empty it for the SDK: it reaches the
  SDK through a recording pass-through, so the provider receives exactly the elements the app
  produces (and, if the generator raises, the same exception at the same point; that request is
  counted `skipped_unreadable_request`), and the snapshot is what the SDK consumed.
- `finish_reason` is the plugin's vocabulary, so the service's `finish_to_review` rule reads
  it; the provider's word is kept in `client.finish_reason_raw`; a missing one stays unset so
  the sidecar marks `finish_unknown` rather than guess. The map:

  | posted | OpenAI chat | OpenAI Responses (`status[:reason]`) | Anthropic | Gemini |
  |---|---|---|---|---|
  | `stop` | `stop` | `completed` | `end_turn`, `stop_sequence` | `STOP` |
  | `length` | `length` | `incomplete:max_output_tokens` | `max_tokens`, `model_context_window_exceeded` | `MAX_TOKENS` |
  | `tool_calls` | `tool_calls`, `function_call` | | `tool_use` | `MALFORMED_FUNCTION_CALL` |
  | `content_filter` | `content_filter` | `incomplete:content_filter` | `refusal` | `SAFETY`, `RECITATION`, `BLOCKLIST`, `PROHIBITED_CONTENT`, `SPII`, `IMAGE_SAFETY`, `LANGUAGE` |
  | `abort` | (stream closed early) | `cancelled`, `in_progress`, `queued` | | |
  | passed through | | `failed` | `pause_turn` | `other`, `unknown` (`OTHER`, `FINISH_REASON_UNSPECIFIED`), `blocked:<reason>` for a blocked prompt |

  A word not in the table passes through lower-cased, cut to the sidecar's 32 characters.
- A conversation longer than the sidecar's 256-message limit keeps its system turn and the
  newest turns; the number dropped is in `client.messages_truncated` (without the cap the
  sidecar answers 422 and the pair is lost).
- `session_id` / `session_user` / `session_label` (under `runlatent.session(...)`), or `session_fingerprint`
  (inferred from the conversation when no id was given), and `session_turn` (the count of assistant turns in
  `messages` plus one) name the call or conversation the turn belongs to ("Group the turns of a call", below).
- Multimodal and tool parts in the PROMPT (images, audio, files, `inline_data`, function calls)
  are reduced to a marker in place, `[image_url omitted]`, so the prompt keeps its shape, and
  counted in `client.parts_dropped` and in `stats()["multimodal_parts_dropped"]`; in the answer
  they are dropped without a marker (above). The bytes never leave the process.
- `usage` is normalized to input / output / total tokens.

The sidecar validates the body with its own `ScoreRequest`; the `client` block is outside that
contract today and is ignored by pydantic, not refused (`tests/test_sdk.py` posts the body to
the real app). Token ids are not posted (the SDKs do not expose them).

## Tag the sources: `runlatent.context(docs)`

Latent checks an answer against the source it was written from, so the source must be separable from the question in
the posted pair: a tagged `context` (below, or `runlatent.check(prompt, answer, context=docs)`), a `Knowledge:` line
opening the user turn, or the system message. An answer with none of these (`context_source: none`, a bare question or
passages interleaved with no marker) is stored and shown in the console but gets no grounding verdict and is never
flagged (the reply reads `verdict: "unknown"`, `unscored_reason: "no_source"`, the reader's number as
`risk_informational`; the SDK result is `unscored`), and it is not judged, so the calibration and the first fit have
fewer rows. Apps that retrieve passages and build the prompt themselves can tag them without changing the prompt:

```python
with runlatent.context(docs):                      # a str, a list of str, or [{"title", "text"}]
    client.chat.completions.create(model="gpt-5", messages=messages)

@runlatent.context(lambda question, docs: docs)    # or a decorator: the docs from the call's own arguments
def answer(question, docs): ...
```

Every call captured inside posts the passages as `context` beside `messages` (at most 64 passages, 200,000
characters in all); nothing in the app's call changes; thread- and task-local, async-safe, never raises. A
`redact` hook sees `context` on the body like every other field: mask it too when the passages carry what the
messages carry. `runlatent.check(prompt, answer, context=...)` renders its own `context` into the user turn as a
`Knowledge:` line (one copy, redacted with the messages) and posts the ambient block only when no `context=` was given. Apps that
cannot wrap the call put the passages in the system message or under a `Knowledge:` line
(`docs/hosted-score.md`, "What the judge needs as the source").

## Group the turns of a call: `runlatent.session(...)`

A voice agent or a chat app makes many LLM calls per call or conversation. Usage is per scored answer (one per
turn), and the review queue can say "these answers are all from the same session" and roll a session up when the
turns are tagged. Wrap the call loop once:

```python
with runlatent.session(call_id, user_id=caller_hash):      # one block around the call loop
    for turn in call:                                      # every captured call and every check() inside is tagged
        reply = client.chat.completions.create(model="gpt-5", messages=history)
        history.append({"role": "assistant", "content": reply.choices[0].message.content})

@runlatent.session(lambda call, **kw: call.id, label="Support line")   # or a decorator: the id from the call's arguments
async def handle(call): ...

r = runlatent.check(prompt, answer, session_id=call_id, user_id=caller_hash)   # or per check, which wins over the block
```

Every pair posted inside carries `session_id` (and `session_user` / `session_label` when given) and `session_turn`,
the count of assistant turns in `messages` plus one. Per-turn scoring is the unit: each turn is one scored answer,
one console row and one on the bill; the session is how the rows are grouped. The queue shows a session chip on each
row ("Session, turn 7 of 12"), the request page the previous and next turn in the session, `GET /queue?session=<id>`
lists one session's turns in order (passes included), `GET /sessions` rolls each session up (turns, flagged turns,
max risk, last verdict, open reviews) and `/stats.sessions` counts them (`service/README.md`). The optional policy
rule `review.session_rule.flag_once_per_session` (off by default) routes a second flagged turn of a session into the
session's open review instead of opening a second queue row.

The ids are **your own opaque strings**, stored as given and shown in the console: they must not be personal data.
Hash a phone number or an account id before passing it as `user_id` (`caller_hash` above); the same goes for
`session_id` and `label`. A hash-only plan gains nothing readable from them beyond these ids. One line each, up to 256
characters (cut, never refused); thread- and task-local (`contextvars`), async-safe, never raises into the app.

**Inferred sessions.** With no explicit id, an app that passes the conversation history each turn (`messages`) gets
`session_fingerprint`: the sha256 of the system turn plus the first user turn, each whitespace-normalized the way the
pair fingerprint is, computed on the list the app sent (before the 256-message cap trims old turns), so it stays the
same for the whole conversation. An explicit id wins over it. Two conversations that open with the same system prompt
and the same first user message share a fingerprint, so pass an explicit id when that matters (a FAQ bot whose first
turn is "Hi"); the service never merges reviews on a fingerprint, only on an explicit id. Inference is on by default
and is the one thing the posted body gains on upgrade; turn it off with `runlatent.configure(infer_session=False)` or
`LATENT_INFER_SESSION=0` (then a conversation without an explicit id posts `session_turn` only). A bare
`check(prompt, answer)` has no conversation to infer from: only an explicit or ambient id names its session. The block
is `contextvars`-scoped: `asyncio.to_thread` (and `acheck`) inherit it, a `ThreadPoolExecutor` worker does not (pass
`session_id=` to a `check()` run on a pool). The streaming read (`stream_read=True`) records its pair through the
reader's stream session and carries no conversation session yet. Under the policy rule, the merge is by session id across
models: one call served by two models is one review.

## What is never posted

- API keys, headers, the request's other parameters (temperature, tools, response formats,
  metadata): only `model` and the message text are read.
- Image, audio and file bytes or URLs (reduced to a marker, see above).
- Thinking / reasoning blocks (Anthropic `thinking`, Gemini `thought` parts, Responses
  `reasoning` items).
- Anything when the call failed: the SDK's exception propagates unchanged and no pair exists.
- Anything the `redact` hook returned `None` for, or raised on (a failed masker never lets the
  unmasked pair through; the drop is counted).

## Streaming

A stream is returned to the app through a thin proxy that yields the same items and delegates
every other attribute (`.response`, `.close()`, `with …`), and the pair is posted **once, at
the end**: when the iterator is exhausted (OpenAI chat chunks and Responses events, Anthropic
raw events, Gemini chunks are assembled into the full text; the last chunk's finish and usage
are taken), or when **the proxy's own** `close()` or context exit runs before that, in which
case the partial text is posted with `finish_reason: "abort"` and `client.stream_complete:
false`. A stream the app simply stops reading (`for chunk in s: break` with no `close()` and no
`with` on `s`) is never posted: it is counted `stream_unfinished` when the proxy is
garbage-collected (a finalizer counts, it does not post). OpenAI's own helper managers,
`client.chat.completions.stream()` and `client.responses.stream()`, go through the patched
`create` and capture once when run to exhaustion; their `__exit__` closes the HTTP response,
not the stream, so an early exit from one of them is the unfinished case above, not `abort`.
One consequence: `chat.completions.stream()` can raise `LengthFinishReasonError` from its own
parsing after the final chunk arrived; the raw stream under it was exhausted, so the pair IS
posted, as `length`, although the app got an exception rather than a completion (E-R149). `abort` is only ever
the app closing a healthy stream early: a stream that raised mid-way, or a `with` block an
exception left (the SDK raising on an `error` event, Anthropic's `overloaded_error` included,
or the app raising inside the block), posts nothing and is counted `stream_errored`. Anthropic's
`messages.stream()` manager hands the app the SDK's `MessageStream` through a thin proxy (same
events, `text_stream`, `get_final_message()` and every attribute delegated) that remembers a
failure raised while reading, so an `overloaded_error` the app catches inside the block still
posts nothing; a clean exit reads `current_message_snapshot` (no extra network read), and a
healthy stream the app stops reading early posts the partial text as `abort`. Only the first
choice / candidate is assembled when `n > 1`.

## Streaming read (opt-in): the risk while the answer streams

```python
runlatent.auto_instrument(stream_read=True)      # or runlatent.wrap_openai(client, stream_read=True); process-wide;
                                                 # auto_instrument() without the argument leaves an earlier opt-in on

stream = client.chat.completions.create(model="gpt-4.1", messages=msgs, stream=True)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
    read = runlatent.current_read()              # the live read of THIS stream, or None
    if read and read["flagged"]:
        print("\n[withheld: this answer is being flagged]")
        stream.close()                           # the app decides; runlatent never withholds a chunk itself
        break
```

With `stream_read=True` a streamed call is also fed, delta by delta, to Latent's reader as the app
reads it: the reader runs its streaming read session (`<LATENT_API_URL>/stream/open`, `/feed`,
`/close`: a sidecar with `LATENT_STREAM_READ=1`; `latent_sidecar/README.md`, "The streaming read
session"), which scores every token of the answer so far and runs the plugin's early-stop rule on
the running mean. `runlatent.current_read()` returns that read for the stream the calling code is
iterating (the proxy's `latent_read()` is the same object): `risk` (the artifact's quantity at the
last read), `mean_risk` (the running mean, what the rule reads), `n_tokens`, `threshold`, `rule`,
`early_stop` (the reader's decision record once the rule fired, else `None`), `flagged` (the
decision fired, or the running mean is at or above the cut), `status` (`pending` before the first
delta reached the reader, `reading`, `unavailable`, `closed`). The read lags the stream by the
reader's step and one round trip (a few tokens); the feeding runs on a daemon thread behind a queue,
so the app's iteration never waits on it, and deltas that arrive while a feed is in flight ride the
next one. **The SDK never withholds**: every chunk the provider yielded reaches the app when it was
yielded; stopping is the app's `break` (posted as `finish_reason: "abort"`, like any early close).

The session's close is the record: the reader writes the event with the per-token trace
(`meta.capture: "sidecar-stream"`, `meta.trace`, `meta.early_stop`), so nothing is posted to
`/score` for that stream. A session that cannot be opened (the reader down, at its cap, the stream
read off: `404`), fails on the way or is refused at close falls back to the one-pass `/score` post of
the whole stream, counted `stream_read_fallback`; nothing is lost. Counters in `stats()`:
`stream_read` (on / off), `stream_read_sessions`, `stream_read_fed`, `stream_read_stops`,
`stream_read_closed`, `stream_read_unavailable`, `stream_read_failed`, `stream_read_fallback`, `stream_read_aborted` (a
stream that raised mid-way, whose capture failed, or that the app DROPPED without `close()` / `with` (a bare
`break`, an exception in a plain `for` body, a dropped `async for`) releases its session unscored when the proxy is
collected: nothing is recorded, as the one-pass path records nothing for such a stream, and `current_read()` is
None for it from then on), `stream_gateway_read` (below). `current_read()` is task-scoped under asyncio: a task
that iterates no stream reads None, never another task's stream; and it is None once the stream has ended,
whichever way it ended.

**With `runlatent.check()`:** a stream read and a `check()` of the same pair are ONE reader pass in either order:
`check()` by the provider's response id or by the conversation adopts the stream's close reply (`check_adopted`),
and a stream whose pair `check()` already recorded releases its session unscored (`skipped_checked`). The reader
itself answers a `/score` of a request id its stream read just closed from memory (`deduplicated: "stream_close"`,
bound to the same messages and output; other content under that id is read fresh and marked `conflicting_replay`),
so the fallback after a close reply that timed out on this side costs no second pass either.

**Behind Latent's gateway with the shadow read on** (`LATENT_GATEWAY_STREAM_READ=shadow`): the gateway's reader
already records the stream and its trailing chunk's `latent` block says so. Equal ids inside one provider stream do
not authenticate a gateway, so the SDK adopts that tail as the record only when the app has configured a TRUSTED
gateway: `runlatent.configure(gateway_hosts=["gateway.internal"])` or `LATENT_GATEWAY_HOSTS=gateway.internal,...`
(comma list), the provider client's base URL host (`base_url` on the OpenAI / Anthropic client; `http_options.base_url`
on a `google.genai.Client`, default `generativelanguage.googleapis.com`; read at every call, so a client whose
`base_url` changes after the wrap is judged by where it points now) is one of them, AND the tail carries a completed scored read (`stream_read.status: "read"` with a `risk`) bound to this request
(its `request_id` equals the stream's own id; the gateway stamps it on every shadow tail). Then the SDK releases its own
session unscored and posts nothing (`stream_gateway_read`), whatever `stream_read` is. With no trusted gateway
configured (the default) every tail is ignored and the SDK records as usual, counted `stream_gateway_tail_untrusted`
(the service keeps the first row per request id, so a gateway + SDK pair is still one stored record); a tail from a
trusted host that is not this request's is `stream_gateway_tail_unbound`. A chunk carrying `stop_reason:
"latent_early_stop"` is read as the plugin's early stop (`finish_reason: "early_stop"`, its replacement text not the
model's answer), not as a `length` finish. Behind a gateway running the shadow read, leave `stream_read` off and read
the tail's `latent` block instead; `stream_read=True` is for the app that calls the provider directly.

**Redaction:** redaction and the live stream read are mutually exclusive: a redacting hook
(`runlatent.configure(redact=...)`) disables live reads, and the recorded pair is redacted. A whole-pair hook has no
safe incremental contract (a lookahead decides on text that arrives later), and text once fed to the reader cannot be
retracted, so with `redact=` configured the stream's deltas are never fed; the fully redacted pair is posted once at
the end exactly as without `stream_read`, counted `stream_read_redact_fallback`.

**Not fed:** Anthropic's `messages.stream()` manager (`with client.messages.stream(...) as s`) reads
`current_message_snapshot` at exit and has no delta path through the proxy, so it is scored once at the end as before
and `current_read()` is None inside it; the raw event stream (`messages.create(..., stream=True)`) is fed. The
Responses API and Gemini streams are fed from the same assembled text pieces as the OpenAI chat stream. On
the hosted service (`docs/hosted-score.md`, "The streaming read") the `/stream/*` routes are carried for Pro and Team
with usage billing on, for a project that turned the streaming read on in the console (Policy > Advanced > "Read
streamed answers sentence by sentence"); a Free key, usage billing off, or a project with the read off falls back to
the one-pass post on every stream (counted `stream_read_fallback`). Every sentence read there is one scored answer on
the bill beside the answer, whether or not the stop below is on.

**Against the hosted route, with usage billing on, the read can stop the stream.** With the project's
`policy.stream_stop` on (the console's Policy page, "Stop a streamed answer"), a feed reply may carry `stop` when the
sentence head's read over the completed sentences crosses the project's cut; the proxy then ends the model's stream,
delivers the policy's fallback text as a final chunk where the provider's chunk shape can carry it (OpenAI chat
chunks: `action: stop`) or ends the iteration with nothing more (`action: hold`), and closes the session with
`finish_reason: "early_stop"`; `runlatent.last_decision()` and the read's `action` say what happened, counted
`stream_stop_stop` / `stream_stop_hold`; `auto_instrument(enforce=False)` leaves the stream untouched
(`stream_stop_ignored`). This is the one case where the SDK ends a stream itself, and the customer turns it on.

## Redaction (default: none)

By default the SDK posts the text as the app sent and received it, because masking changes what
Latent reads: E-R143 (2026-10-01) ran a default de-identification tool over logged prompts and
answers masked separately and a third of the good answers turned into judged failures (the
answer named "Felica Cannon", the masked source only "Felica"), the probe lost up to 0.07 AUROC
and the flagged list overlapped the unmasked one about 50%. Identifiers-only masking with ONE
dictionary derived from the source and applied to prompt and answer alike kept the judge at
0.96 and the read within 0.02. If you must mask before posting, do that, and do it in one place:

```python
def redact(body):
    table = my_entities(body["messages"])             # names from the source, one dictionary
    body["messages"] = [dict(m, content=mask(m["content"], table)) for m in body["messages"]]
    body["output"] = mask(body["output"], table)      # the same table on the answer
    return body                                        # or None to skip this pair

runlatent.auto_instrument(redact=redact)
```

The hook sees the full body (the `client` block included) and may rewrite or drop it. Prefer
raw text under the processing terms (retention is the service's `policy.retention`) and mask
after the read where a report needs it.

## Counters and shutdown

```python
runlatent.stats()
# {"enabled": true, "api_url": "...", "patched": ["openai:Completions.create", ...],
#  "captured": 120, "posted": 118, "failed": 0, "dropped": 2, "queued": 0,
#  "skipped_empty_output": 3, "multimodal_parts_dropped": 7, "redact_skipped": 0,
#  "checked": 40, "check_scored": 38, "check_adopted": 30, "check_unscored": 1, "check_refused": 1, "skipped_checked": 2,
#  "errors": {"post:ConnectionRefusedError": 0}, "poster": {... the poster's own counters ...}}
runlatent.flush(timeout=5.0)   # at shutdown: wait for the queue to be attempted
```

`enforce_read` counts the synchronous reads the policy asked for ("Act from the console"), `enforce_<action>` the
decisions (`enforce_abstain`, `enforce_regenerate`, `enforce_route`, `enforce_judge`, `enforce_none`, `enforce_hold`,
`enforce_regenerate_failed`, `enforce_route_failed`), `enforce_fail_open` / `enforce_fail_closed` the reads that did not come
back in the budget by how they ended, `enforce_replace_failed` a response shape the SDK could not rewrite (delivered as it
was, said so in `latent.reason`), `enforce_reported` / `enforce_report_failed` / `enforce_report_dropped` the decision
reports; `stats()["enforce"]` carries `enabled`, the poller's reading (`policy`: `url`, `version`, `fetched`,
`not_modified`, `failures`, `unauthorized`, `last_status`) and `last_decision`.
`checked` counts `check()` calls; `check_scored` those answered with a risk (`check_adopted` of them
read from the instrumentation's own POST rather than a second one), `check_unscored` the empty
answers, `check_refused` the refusals and transport failures (by class under `errors`,
`check_post:TimeoutError`), `skipped_checked` the captured pairs the poster did not post because a
`check` recorded them, `check_detail_reposted` the `check(detail=True)` calls that posted their own read
because the captured one had already gone out without `detail`.

`captured` counts pairs assembled; `posted` / `failed` / `dropped` / `queued` are the poster's
(`latent_vllm.events.SinkPoster` through `latent_sidecar.tap.FireAndForget`: daemon thread,
bounded queue, three attempts per pair, no spill file: a pair that cannot be delivered is
lost and counted). `dropped` includes pairs captured with no destination configured.
`errors` is keyed by stage and exception class (`observe:…` a shape the adapter could not read,
`redact:…`, `post:…`, `stream_feed:…`); nothing in it ever reached the app. Only the class is
logged (at DEBUG), never the exception's message: a masker that raises with its input in the
message must not put answer text in your log.

## Limits

- `stream_read=True` is process-wide (one reader client for the API base), reads sample 0 of a
  stream only, and needs a reader with the streaming read session; the Responses, Anthropic and
  Gemini streams are fed from the same assembled text pieces as the OpenAI chat stream.
- Only the methods above are patched. `chat.completions.parse()` and `responses.parse()` post
  through their own path and are not captured; `beta.*`, the Assistants and Realtime APIs are not
  in this version; wrap a gateway or use `latent_sidecar.tap` for those. A client the SDK does not
  patch (Replicate, SageMaker `invoke_endpoint`, Bedrock `invoke_model`, in-process `transformers`)
  goes through `runlatent.check(prompt, answer, context=...)`: `docs/sdk-providers.md` has the matrix.
  Captured because they route through the patched methods: `chat.completions.stream()` and
  `responses.stream()` (exhaust-to-capture, see Streaming) and Gemini `chats.send_message`.
- Arguments must be passed by keyword (the SDKs require it; a positional `messages` is not read), with two
  exceptions: `InferenceClient.chat_completion(messages, ...)` takes `messages` positionally and ollama's
  `chat(model, messages)` / `generate(model, prompt, suffix)` take theirs so, and the SDK binds them (class
  patch and `wrap_huggingface` / `wrap_ollama` alike; `docs/sdk-providers.md`).
- `with_raw_response.create(...)` and `with_streaming_response.create(...)` return the HTTP
  response object, not the parsed completion; reading it would consume a streaming body, so
  those calls are not captured and are counted under `stats()["skipped_raw_response"]`.
- At a normal interpreter exit an atexit handler waits up to `LATENT_SDK_EXIT_FLUSH_S`
  seconds (default 2.0) for the queued pairs to be posted, so a short script that never calls
  `flush()` still delivers; with it set to `0` nothing waits and pairs still queued at exit are
  lost without a counter (the process is gone). A `kill -9` or an `os._exit()` loses them either
  way: call `runlatent.flush()` in your shutdown path when it matters.
- The proxy around a stream is not an instance of the SDK's `Stream` class; `isinstance`
  checks on it fail (attribute access does not).
- There is no on-disk spill: a Latent outage longer than the retries loses those pairs, counted.
- `max_queue` counts pairs, not bytes, behind one serial poster thread (ingest tops out near
  1 / POST latency pairs per second). A full queue of 10,000 pairs with 100 KB RAG prompts is
  about 1 GB held in the process before anything is dropped: size `max_queue` to the burst you
  are willing to hold in memory, and lower it on memory-tight hosts.
- Multi-choice (`n > 1`) and multi-candidate responses post the first only.
- Gemini thinking models count their thought tokens against `max_output_tokens`: a short budget
  yields many empty answers (not posted, `skipped_empty_output`) and truncated ones, posted as
  `length` and routed to review by the service's default `finish_to_review` rule (E-R149's
  first Gemini run at 160 tokens produced 44 of them). Give thinking models a budget that covers
  the thinking, or disable it, before reading the review queue as a quality signal.
