Metadata-Version: 2.4
Name: llmswitch-sdk
Version: 0.1.4
Summary: One Python client for chat completions, embeddings and the Responses API across OpenAI, Azure OpenAI, Azure AI Foundry, Anthropic and Vertex AI.
Author: Rituraj Raman
License-Expression: MIT
Project-URL: Homepage, https://github.com/RITURAJRAMAN/llmswitch
Project-URL: Repository, https://github.com/RITURAJRAMAN/llmswitch
Project-URL: Issues, https://github.com/RITURAJRAMAN/llmswitch/issues
Project-URL: Changelog, https://github.com/RITURAJRAMAN/llmswitch/blob/main/CHANGELOG.md
Keywords: llm,openai,azure,anthropic,vertex-ai,embeddings,chat-completions,responses-api
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3,>=2.14.1
Requires-Dist: openai<4.0.0,>=3.36.1
Requires-Dist: anthropic<3,>=2.1.0
Requires-Dist: azure-ai-inference==1.0.0b9
Requires-Dist: aiohttp<4.0.0,>=3.14.5
Requires-Dist: google-genai<3,>=2.32.0
Requires-Dist: tiktoken<0.15,>=0.14.0
Provides-Extra: anthropic
Provides-Extra: azure-foundry
Provides-Extra: vertexai
Provides-Extra: redis
Requires-Dist: redis<9,>=8.1.0; extra == "redis"
Provides-Extra: agentic
Requires-Dist: langchain-core<2,>=1.6.9; extra == "agentic"
Provides-Extra: all
Requires-Dist: llmswitch-sdk[agentic,redis]; extra == "all"
Provides-Extra: dev
Requires-Dist: build>=1.6.1; extra == "dev"
Requires-Dist: pytest>=9.1.1; extra == "dev"
Requires-Dist: pytest-asyncio>=1.4.0; extra == "dev"
Requires-Dist: pytest-cov>=7.1.0; extra == "dev"
Requires-Dist: mypy>=2.4.0; extra == "dev"
Requires-Dist: ruff>=0.17.1; extra == "dev"
Requires-Dist: twine>=7.0.0; extra == "dev"
Dynamic: license-file

# llmswitch

One Python client for chat completions, embeddings and the Responses API across OpenAI, Azure OpenAI,
Azure AI Foundry, Anthropic and Vertex AI. Switch providers by changing the constructor arguments; the
call sites stay the same. Rate limits, retries, a circuit breaker, backup failover and context handling
are built in, and they behave the same for every provider.

Supported in 0.1.4: sync and async chat completions (streaming and structured output), embeddings, the
Responses API, rate limiting, retries, a circuit breaker, backup failover, context truncation, batching,
observability hooks, and the agentic features (the optional `agentic` extra): a chat model with a bounded
tool loop, and an agent with reflection, tool execution, checkpoints and resume, model routing, specialists
and aggregation.

Supports Python 3.11, 3.12 and 3.13.

## Contents

- [Which feature to use](#which-feature-to-use)
- [Installation](#installation)
- [Quick start](#quick-start)
- [Providers and their settings](#providers-and-their-settings)
- [Chat completions](#chat-completions)
- [Embeddings](#embeddings)
- [Responses API](#responses-api)
- [Agentic features](#agentic-features)
- [Rate limits](#rate-limits)
- [Retries, timeouts and the circuit breaker](#retries-timeouts-and-the-circuit-breaker)
- [Backup failover](#backup-failover)
- [Context, truncation and token counting](#context-truncation-and-token-counting)
- [Performance: clients, sharing and lazy loading](#performance-clients-sharing-and-lazy-loading)
- [Observability](#observability)
- [Configuration reference](#configuration-reference)
- [Errors](#errors)
- [Development](#development)
- [License](#license)

## Which feature to use

| If you need to... | Use | Why |
|---|---|---|
| Get a reply from a model | `chat.completions.create` | The same call works for every provider. |
| Show the reply as it is generated | `create(..., stream=True)` | The first words arrive sooner, and long answers feel faster. |
| Get a typed object back | `chat.completions.parse` with a Pydantic class | You get validated data instead of parsing text yourself. |
| Turn text into vectors | `embeddings.create` | For search, clustering and similarity. Use `batching=True` for large lists. |
| Use OpenAI's newer Responses API | `responses.create` | Stateful and tool-rich features. Native for OpenAI and Azure OpenAI; emulated for Anthropic and Vertex AI. |
| Run tools in a loop with a model | `ChatLLMSwitch` and `run_tool_loop` | A bounded loop that runs the tools the model asks for, and feeds the results back. |
| Let an agent work through a task with limits | `Agent` | Budgets, tool timeouts and retries, reflection on the answer, and checkpoints to pause and resume. |
| Route steps between a main and a backup model | `Agent(fallback_model=..., routing_policy=...)` | Escalate after a tool error, alternate models, or switch on a late step. |
| Ask several agents and combine their answers | `invoke_specialists` and `aggregate_specialists` | Contracts with capabilities and weights, and majority, weighted or JSON-merge consensus. |
| Limit cost and traffic per user | `rpm`, `tpm`, `user_id` | Stops one user or job from using the whole quota. |
| Share limits across many processes | `redis_url` | In-process limits cannot see other processes. |
| Survive short provider outages | `max_retries`, `retry_base_delay` | Retries transient errors with exponential backoff. |
| Stop hammering a broken provider | `circuit_breaker=True` | Fails fast after repeated failures, then probes again later. |
| Keep serving when a provider is down | `backup={...}` | Moves the request to a second provider. |
| Fit long conversations into a context window | `truncate=True` | Trims the oldest content instead of failing. |
| Measure what happens on each call | `on_event` | Gives you retries, failovers, latency and token usage. |

## Installation

```bash
pip install llmswitch-sdk
```

This installs the SDKs for OpenAI, Azure OpenAI and Foundry, Anthropic and Vertex AI. An SDK is loaded
the first time you use its provider, so `import llmswitch` stays fast.

Optional extras:

| Extra | Install | Use it when |
|---|---|---|
| `redis` | `pip install "llmswitch-sdk[redis]"` | You run several processes that must share rate limits. |
| `agentic` | `pip install "llmswitch-sdk[agentic]"` | You want the agentic chat model, the tool loop, or the agent with specialists. |
| `all` | `pip install "llmswitch-sdk[all]"` | You want Redis and the agentic extra together. |

The extras `anthropic`, `azure-foundry` and `vertexai` are still accepted so existing install commands
keep working. They add nothing new.

## Quick start

```python
from llmswitch import LLMSwitch

client = LLMSwitch(provider="openai", api_key="sk-...", model="gpt-4o")

reply = client.chat.completions.create(
    messages=[{"role": "user", "content": "Say hello in French."}],
)
print(reply.choices[0].message.content)

vectors = client.embeddings.create(["first text", "second text"], model="text-embedding-3-small")
print(len(vectors.data), len(vectors.data[0].embedding))

client.close()
```

`LLMSwitch` can also be used as a context manager, which closes it on exit:

```python
with LLMSwitch(provider="openai", api_key="sk-...", model="gpt-4o") as client:
    print(client.chat.completions.create(messages=[{"role": "user", "content": "Hi"}]).choices[0].message.content)
```

Async works the same way with `AsyncLLMSwitch`:

```python
import asyncio
from llmswitch import AsyncLLMSwitch

async def main():
    async with AsyncLLMSwitch(provider="anthropic", api_key="sk-ant-...", model="claude-opus-4-5") as client:
        reply = await client.chat.completions.create(messages=[{"role": "user", "content": "Hello"}])
        print(reply.choices[0].message.content)

asyncio.run(main())
```

## Providers and their settings

| `provider` | Required settings | Chat (sync / async) | Embeddings | Responses |
|---|---|---|---|---|
| `openai` | `api_key`, `model` | yes / yes | yes | native |
| `azure_openai` | `api_key`, `endpoint`, `api_version`, `deployment_name` | yes / yes | yes | native |
| `azure_foundry` | `api_key`, `endpoint`, `api_version`, `deployment_name` | yes / yes | yes | no |
| `anthropic` | `api_key`, `model` | yes / yes | no | emulated |
| `vertexai` | `location`, `credentials_path`, `model` | yes / yes | no | emulated |

Asynchronous embeddings use the native async client for `azure_foundry`. The other providers run their
synchronous embedding code in a worker thread, so the event loop stays free.

**Credentials.** The library never reads environment variables or `.env` files. Load your keys in your
application and pass them to the constructor. `credentials_path` for Vertex AI points to a service account
JSON file.

**Azure OpenAI example:**

```python
client = LLMSwitch(
    provider="azure_openai",
    api_key="...",
    endpoint="https://your-resource.openai.azure.com/",
    api_version="2024-10-21",
    deployment_name="your-gpt4o-deployment",
)
```

**Vertex AI example:**

```python
client = LLMSwitch(
    provider="vertexai",
    location="us-central1",
    credentials_path="/path/to/service_account.json",
    model="gemini-2.5-pro",
)
```

**Model selection.** A configured `deployment_name` is always the model used, and a per-call `model` cannot
replace it. Without a deployment name, a per-call `model` overrides the configured `model`. Use this to
send one client's requests to several models.

**Token limits.** `max_tokens`, `max_completion_tokens` and `max_output_tokens` are interchangeable on every
provider. The library sends the name each provider expects. This matters for newer OpenAI models, which
reject `max_tokens`, and for Anthropic, which requires it.

**Gemini and the output budget.** Gemini 2.5 models use part of the output budget for internal reasoning.
A small limit can end the reply early: the response has `finish_reason="length"` in chat,
`status="incomplete"` in Responses, and `parse()` may return no parsed value. In testing, a
`max_output_tokens` of 300 was not enough for a short structured answer, while 2000 was. Set a generous
limit for Gemini calls, and check `finish_reason` or `status` before trusting the output.

**Anthropic prompt caching.** `enable_prompt_caching` (default `True`) adds cache markers to system messages
and tool definitions, so repeated prompts cost less. Set it to `False` if you need the raw request.

## Chat completions

**When to use it:** any time you need a model reply. Use `create` for a complete reply, `create(stream=True)`
for progressive output, and `parse` when you need a typed object.

```python
reply = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "Answer in one sentence."},
        {"role": "user", "content": "Why is the sky blue?"},
    ],
    temperature=0.2,
    max_tokens=200,
)
print(reply.choices[0].message.content, reply.usage.total_tokens)
```

**Streaming** returns an iterator of chunks. Stop early by breaking out of the loop, or call `close()` on
the stream:

```python
for chunk in client.chat.completions.create(messages=messages, stream=True):
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

With `AsyncLLMSwitch`, await the call first, then iterate with `async for`:

```python
stream = await client.chat.completions.create(messages=messages, stream=True)
async for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

**Structured output** takes a Pydantic model class. The parsed object is on `choice.message.parsed`:

```python
from pydantic import BaseModel

class City(BaseModel):
    name: str
    country: str

parsed = client.chat.completions.parse(
    messages=[{"role": "user", "content": "Largest city in Japan?"}],
    response_format=City,
)
print(parsed.choices[0].message.parsed)
```

**Provider-specific arguments.** Per-call keyword arguments such as `seed`, `top_k` or `extra_body` are
passed to the provider request unchanged. Constructor options that no provider defines are rejected with a
suggestion, so a typo such as `max_retires` fails at construction time.

**Per-call overrides.** `max_retries`, `truncate` and `live_logs` can be passed to any single call.

## Embeddings

**When to use it:** to turn text into vectors for search, clustering or comparison. Pass a string or a list.

```python
vectors = client.embeddings.create(
    ["first text", "second text"],
    model="text-embedding-3-small",
)
print(vectors.data[0].embedding[:5], vectors.usage.total_tokens)
```

**Batching** splits a large list by token budget and by `batchsize`, so each request stays within the
provider's limits. Results come back in input order unless you pass `restore_order=False`:

```python
vectors = client.embeddings.create(texts, batching=True, batchsize=32)
```

**Failover for embeddings is all or nothing.** If the primary fails part-way through a batched request, the
backup embeds the whole input again. Every vector then comes from one model, because mixing vectors from two
embedding models would make them incomparable.

Embeddings are available for `openai`, `azure_openai` and `azure_foundry`.

## Responses API

**When to use it:** when you need the Responses API's features: server-side conversation state, background
responses, hosted tools, or its structured output and streaming events. For `openai` and `azure_openai` these
are the service's own features. For `anthropic` and `vertexai`, Responses is emulated on top of chat
completions, so the same code runs on every provider that supports it.

```python
client = LLMSwitch(provider="openai", api_key="sk-...", model="gpt-4o")

reply = client.responses.create(input="Summarize the llmswitch README in one line.")
print(reply.output_text)

for event in client.responses.stream(input="Write a haiku about Python."):
    if event.type == "response.output_text.delta":
        print(event.delta, end="")

parsed = client.responses.parse(input="Largest city in Japan?", text_format=City)
print(parsed.output_parsed)
```

Other calls:

- `retrieve(response_id)`, `delete(response_id)` and `cancel(response_id)` manage stored responses.
- `compact()` shortens a conversation.
- `input_items.list(response_id)` lists the input items of a response.
- `input_tokens.count()` counts the tokens a request would use.
- `conversations` creates, reads, updates and deletes conversations and their items.
- `create_background(...)` starts a background response. `wait(response_id)` polls it until it finishes.
- `submit_mcp_approval(previous_response_id, approval)` answers an MCP approval request.

Which of these work depends on the model and the endpoint. Token counting and the conversations API are
provided by the service and may return an error for a given deployment. Service errors are raised as
`ResponsesError` with the service's message.

**Checks before sending.** Responses requests are validated first. Unknown request fields raise
`InvalidResponseRequestError`. Remote MCP servers and connectors are rejected unless you allow them:

```python
client = LLMSwitch(
    provider="openai",
    api_key="sk-...",
    model="gpt-4o",
    mcp_policy={
        "enabled": True,
        "allowed_server_labels": ["docs"],        # or allowed_connector_ids / allowed_origins
    },
)
```

**Capabilities.** `client.responses.capabilities()` reports what the configured target supports. A feature
can be turned on or off with `responses_capabilities`, for example `{"background": True}`.

**API version.** Azure OpenAI calls to `/responses` need API version `2025-03-01-preview` or later. Older
`api_version` values are raised to that minimum automatically. Set `responses_api_version` to choose a
different version for Responses calls only.

`azure_foundry` has no Responses API. Calls to it raise `ResponsesError` with status 501.

### Responses on Anthropic and Vertex AI

For `anthropic` and `vertexai`, `responses.create()` (and `stream()`, `parse()` and the async client)
translate the request into a chat completion and the result back into a `Response`:

```python
client = LLMSwitch(provider="anthropic", api_key="sk-ant-...", model="claude-opus-4-5")
reply = client.responses.create(input="Hello", instructions="Be brief.")
print(reply.output_text, reply.usage.total_tokens)
```

**What works:** text and image input, `instructions`, function tools and `tool_choice`, function-call round
trips (`function_call` and `function_call_output` items), `text` formats and `text_format` for structured
output, `temperature`, `top_p`, `max_output_tokens`, streaming, and the usual rate limits, retries and
failover. Streams produce the Responses events (`response.created`, `response.output_text.delta`,
`response.function_call_arguments.delta`, the `.done` events, and `response.completed` or
`response.incomplete`). A response cut off by the token limit has `status="incomplete"`.

**What does not work**, because chat completions cannot provide it: server-side state
(`previous_response_id`, `conversation`, `store=True`, `background`), stored objects (`retrieve`, `delete`,
`cancel`, `compact`, `input_items`, `input_tokens.count`, `conversations`) and hosted tools (web search, file
search, code interpreter, MCP and the rest). These raise `UnsupportedCapabilityError` before any request is
sent. Keep the conversation history in your application and pass it as `input` items.

Fields with no chat equivalent (`metadata`, `user`, `truncation`, `include`, `reasoning` and similar) are
accepted and ignored. Response ids are generated locally and cannot be used to retrieve anything.

`client.responses.capabilities().emulated` is `True` for these providers.

## Agentic features

**When to use it:** when a model should call your functions in a loop, for example to look things up and
then answer. Use `ChatLLMSwitch` when you want a chat model object; use `run_tool_loop` when you want the
whole loop run for you. Install the `agentic` extra first.

```python
from llmswitch.agentic import ChatLLMSwitch

llm = ChatLLMSwitch(provider="anthropic", api_key="sk-ant-...", model="claude-opus-4-5", temperature=0.2)
print(llm.invoke("Say hello in French.").content)
```

`ChatLLMSwitch` accepts the same provider options as `LLMSwitch`. The model's own options are `temperature`,
`max_tokens`, `stop` and `call_options` (extra keyword arguments sent with every request). Every other
keyword argument is a switch option, so `backup`, `on_event`, the rate-limit settings and the rest work the
same way as they do for `LLMSwitch`. It supports `invoke`, `ainvoke`, `stream`, `astream`, `bind_tools`
and `close`/`aclose`. It is also a standard runnable, so it composes with prompts and parsers using `|`.

Set `api="responses"` to send requests through the Responses API instead of chat completions. It is
available for `openai` and `azure_openai`. The agent works with either API.

**Binding tools.** `bind_tools` sends tool objects, Pydantic classes, plain functions or OpenAI-style dicts
with every request:

```python
def get_weather(city: str) -> str:
    """Get the current weather for a city."""
    return f"Sunny in {city}"

llm_with_tools = llm.bind_tools([get_weather], tool_choice="auto")
reply = llm_with_tools.invoke("Weather in Paris?")
reply.tool_calls   # [{"name": "get_weather", "args": {...}, "id": "...", "type": "tool_call"}]
```

`tool_choice` accepts `"auto"`, `"none"`, `"required"` (or `"any"`), or a tool name to force that tool.
Arguments that are not valid JSON appear in `reply.invalid_tool_calls` instead of raising.

### Tool loop

`run_tool_loop` runs the tools the model asks for and returns the whole conversation:

```python
from llmswitch.agentic import run_tool_loop

history = run_tool_loop(llm, "Weather in Paris, and 17 plus 25?", [get_weather, add], max_steps=4)
print(history[-1].content)
```

- Each turn that requests tools runs them in order. The results go back to the model as tool messages.
- A tool that raises, an unknown tool name, and arguments that fail the tool's schema are sent back as
  error tool messages, so the model can recover.
- `max_steps` is the most model calls allowed. If the model still asks for tools on the last call,
  `ToolLoopLimitError` is raised and no tools run for that last turn.
- `arun_tool_loop` is the async version. Tools run one after another in both versions, so their order and
  side effects are predictable.
- Plain functions need a name. Functions created with `functools.partial` have none, so wrap them in a
  `def` or use a tool object.

`run_tool_loop` has no budgets beyond `max_steps`, no checkpoints and no routing. For those, use `Agent`
below.

### Message conversion limits

- Images are supported in user messages only. An image in a system, assistant or tool message raises
  `ValueError` with a clear message.
- With Anthropic, the signed `thinking` and `redacted_thinking` blocks of an answer are kept and sent back
  unchanged in the next request, first in the assistant turn, as Anthropic asks for extended thinking with
  tools. Tool calls are sent through `tool_calls`. Other providers' reasoning blocks are not sent back.
- With Vertex AI, the thought signature of each function call is kept and sent back with that call, as Gemini
  requires for tool use. Streamed answers keep them too, so a streamed turn can be sent back with its
  thinking blocks and signatures intact.
- `tool_choice=False` is sent as `"none"`, which tells the model not to call tools.

### Closing the agentic model

Call `llm.close()`, or `await llm.aclose()` for async use, to close the provider connections. Put the call in
a `finally` block, as the examples do.

### Agent

Use `Agent` when a run needs more than a fixed number of tool calls: budgets, tool timeouts and retries,
pausing and resuming, routing between models, or several specialists. An agent is a loop of model calls
and tool calls.

```python
from llmswitch.agentic import Agent, AgentConfig

agent = Agent(model=llm, tools=[get_weather], config=AgentConfig(max_iterations=6))
result = agent.invoke("Weather in Paris, then a short summary.")
print(result["status"], result["output"])
```

Each step calls the model. If the model asks for tools, they run and their results go back to the model.
If it answers, the answer is checked by **reflection**: a second model call, with no tools, returns a JSON
verdict on whether the answer does what was asked. A rejected answer is revised, up to
`reflection_max_retries` times (default 2). Set `reflection_enabled=False` to skip the check.

`invoke` and `ainvoke` return a dictionary. Its `status` tells you what happened:

| `status` | Meaning | Other keys |
|---|---|---|
| `completed` | The answer passed reflection, or reflection is off. | `output`, `messages`, `steps` |
| `completed_with_reflection_limit` | The answer was still rejected after the revisions; the last one is returned. | `output`, `messages`, `steps` |
| `interrupted` | The run paused. Resume it with `resume(checkpoint_id)`. | `reason`, `checkpoint_id`, `messages`, `steps` |
| `stopped` | A limit ended the run. | `reason`, `messages`, `steps` |

An `interrupted` run has a `reason` of `tool_error` (a tool failed and `interrupt_on_tool_error=True`) or
`uncertain_tool_execution` (a tool may have started before the pause and is not safe to run again). A
`stopped` run has a `reason` of `max_iterations_reached`, `max_runtime_exceeded`,
`max_input_tokens_per_step_exceeded`, `tool_retry_exhausted` (a failure remained after the retries), `tool_failed`
(a failure, with no retries configured) or `reflection_verdict_unreadable` (the reflection check gave no readable
verdict in three tries, so the run stops with the answer it had).

`steps` is a list of `AgentStep` objects, one per model call. Each has `step`, `assistant_content`,
`tool_calls`, `model_used` (the name of the model that answered) and `tool_results`.

The main limits, all in `AgentConfig`:

| Setting | Default | What it does |
|---|---|---|
| `max_iterations` | 12 | The most model calls in one run to answer or request tools. Reflection checks are not counted. |
| `max_runtime_seconds` | 180 | The wall-clock limit for one run. |
| `max_input_tokens_per_step` | none | Stops before a call whose input is over this many tokens (an estimate). |
| `reflection_enabled` | `True` | Turns the reflection check on or off. |
| `reflection_max_retries` | 2 | How many revisions a rejected answer may get. |

You can also pass `max_runtime_seconds=` to `invoke` and `ainvoke` to give one run a shorter limit. It cannot
make the limit longer than the config.

### Tool execution

Every tool call is run with the limits below, and a failure is returned to the model as an error tool
message, so the model can recover. Each failure has a kind:

- `parse`: the arguments are not a JSON object. Never retried.
- `not_found`: no tool has that name. Retried only with `retry_on_tool_not_found=True`.
- `timeout`: the tool did not finish within `tool_timeout_seconds` (default 30). Never retried, because the
  tool may still be running or may already have acted.
- `tool_error`: the tool raised. Retried only if the tool is marked idempotent, or with
  `retry_non_idempotent_tools=True`.

| Setting | Default | What it does |
|---|---|---|
| `tool_timeout_seconds` | 30 | The most time one tool call may take. |
| `parallel_tools` | `True` | Runs the tool calls of one step at the same time. |
| `tool_isolation` | `"thread"` | `"process"` stops a `def` tool at its limit by killing its forked process (see below). |
| `max_concurrency` | 4 | The most tool calls running at once. |
| `tool_retry_attempts` | 0 | Extra rounds for the retryable failures. |
| `tool_retry_backoff_seconds` | 0 | The pause between retry rounds. |
| `interrupt_on_tool_error` | `False` | Pauses the run after a failed tool, so you can decide what to do. |
| `escalate_on_retry_exhausted` | `False` | Stops the run when a tool fails and the failure remains after the retries (reason `tool_retry_exhausted`, or `tool_failed` when no retries are configured). |

Mark a tool safe to retry with `idempotent`. Only do this when running the tool twice has the same effect as
running it once:

```python
from llmswitch.agentic import idempotent

@idempotent
def get_weather(city: str) -> str:
    """Get the current weather for a city."""
    return f"Sunny in {city}"
```

A timeout stops the wait, not the work. A tool written as `async def` is cancelled at the timeout. A plain
`def` tool runs in a worker thread, and Python cannot stop a running thread, so by default it keeps running
after the timeout, in sync and async runs alike.

To stop a plain `def` tool at its limit, set `tool_isolation="process"`. The tool then runs in a forked child
process, which is killed at the timeout, so its work stops. Its arguments and its result cross the process
boundary: the arguments are inherited by the fork, but the result must be picklable. The tool must not depend on state that only exists in the parent
process. It is available on POSIX systems (Linux and macOS), not on Windows. Python warns that a fork from a
program with several threads can deadlock in the child, so use this setting in a program that starts its
threads after the fork, or accept that risk.

### Checkpoints and resume

The agent saves its state at each phase of a run, under a checkpoint id. An `interrupted` run is continued
with that id.

```python
from llmswitch.agentic import Agent, AgentConfig, FileCheckpointStore

store = FileCheckpointStore("./checkpoints")
agent = Agent(model=llm, tools=[lookup], config=AgentConfig(interrupt_on_tool_error=True),
              checkpoint_store=store)

result = agent.invoke("Look up the population of Lyon.")
if result["status"] == "interrupted":
    result = agent.resume(result["checkpoint_id"], user_feedback="Try the other service.")
```

- `InMemoryCheckpointStore` (the default) keeps checkpoints in the process. `FileCheckpointStore(directory)`
  keeps one JSON file per checkpoint, written atomically and checked with a checksum, so a damaged file is
  rejected instead of resumed. A new process can resume from the same directory.
- Any object with `save`, `load` and `delete` methods works as a store.
- `user_feedback` is added to the conversation. It replaces the model's last answer, or the tools that
  have not started yet, and the step is run again.
- A run that is `uncertain_tool_execution` cannot take feedback, because the tool may already have acted.
  `resume` returns it as an `interrupted` result with the same reason, and the run does not continue.

### Routing between models

Give the agent a `fallback_model`, and a routing policy decides which model answers each step:

```python
from llmswitch.agentic import Agent, EscalationRoutingPolicy

agent = Agent(model=primary_llm, fallback_model=backup_llm, tools=[lookup],
              routing_policy=EscalationRoutingPolicy())
```

| Policy | Behaviour |
|---|---|
| `EscalationRoutingPolicy` | Uses the fallback after a tool error, and the primary otherwise. |
| `RoundRobinRoutingPolicy` | Alternates between the primary and the fallback. |
| `CostAwareRoutingPolicy(fallback_after_step=n)` | Uses the primary, switching to the fallback after a tool error or from step `n` on. |

For your own rule, subclass `BaseRoutingPolicy` and implement `select_llm`, or pass `router=` a function
that takes `(messages, step)` and returns the model. A `router` takes precedence over a routing policy.
Routing state, such as a round-robin position, belongs to the agent: each run starts where the last one stopped. It is also saved in checkpoints, so a resumed run continues its own rotation. A resume does not change the agent's routing state, so a run that happens in between is not undone. Use one agent per concurrent run if the order of the models matters.
Each decision is reported to the event handler as `on_routing_decision`.

### Specialists, contracts and shared state

To ask several agents the same task, use `invoke_specialists` (or `ainvoke_specialists`). It returns one row
per specialist, in the order you gave them:

```python
from llmswitch.agentic import SpecialistContract, invoke_specialists

rows = invoke_specialists(
    task="Review this plan.",
    specialists={"legal": legal_agent, "finance": finance_agent},
    contracts={
        "legal": SpecialistContract(capabilities={"review", "legal"}, weight=2.0),
        "finance": SpecialistContract(capabilities={"review"}, weight=1.0),
    },
    required_capabilities={"review"},
    max_runtime_seconds=120,
)
```

- A `SpecialistContract` lists what a specialist can do (`capabilities`) and how much its answer counts
  (`weight`, a finite non-negative number). With `required_capabilities`, only specialists whose contracts
  cover all of them are asked. A specialist with no contract is not asked in that case.
- Each row has `specialist`, `status`, `output`, `reason`, `steps`, `checkpoint_id` and `raw` (the full
  result). A specialist that raised has status `error`. One that ran out of the shared time limit has status
  `timeout`. Rows from a contract also have `weight` and `capabilities`.
- `max_runtime_seconds` is shared: each specialist gets what is left of it. `parallel=False` runs them one
  after another, and `max_concurrency` limits the parallel ones. In a parallel run, a specialist still running
  when the limit ends is reported as `timeout`, and its work is not written to the shared state. A sequential
  run works the same way, one specialist at a time. The async version cancels a specialist at the limit. The
  sync version reports the timeout on time, but the specialist's thread keeps running in the background, as
  any sync call does.
- A parallel specialist reads the shared state when it starts, so what it sees depends on timing.
- `SharedState` is a thread-safe store. Each specialist's output is written to it under
  `specialist_outputs.<name>`. Its contents are added to the task prompt, so a large state is trimmed to
  `shared_state_max_bytes` (default 16,000). Pick the keys to share with `shared_state_keys`.

An agent can also be a tool for another agent. `make_subagent_tool(name=..., description=..., agent=...)`
returns a function that runs the agent and returns a JSON summary (`status`, `output`, `reason`, `steps`).
Give it to the parent's `tools` list, and the parent decides when to delegate.

### Aggregation

`aggregate_specialists` (and `aaggregate_specialists`) runs the specialists and combines their rows. Pick
the strategy with `aggregator=`:

| Aggregator | Consensus |
|---|---|
| `MajorityVoteAggregator()` | The output most specialists gave. |
| `WeightedScoreAggregator()` | The output with the largest total `weight`. A row without a weight counts as 1. |
| `JsonMergeAggregator(conflict_policy=...)` | Merges outputs that are JSON objects, key by key. The default. |

```python
from llmswitch.agentic import WeightedScoreAggregator, aggregate_specialists

result = aggregate_specialists(
    task="What is the capital of Australia? Answer with the city name only.",
    specialists={"geography": geo_agent, "trivia": trivia_agent},
    contracts={"geography": SpecialistContract({"facts"}, weight=2.0),
               "trivia": SpecialistContract({"facts"}, weight=1.0)},
    aggregator=WeightedScoreAggregator(),
)
result.consensus, result.confidence   # the answer, and the share of the weight behind it
```

- Rows with status `error`, `timeout`, `interrupted` or `stopped` are left out, unless you pass
  `include_failed=True`.
- An output that is a JSON string is read as JSON first, and so is a JSON object inside a Markdown code
  fence. `{"a": 1}` and `'{"a": 1}'` count as the same answer.
- An output of `None` is not a vote, but it still counts in the denominator of `confidence`.
- Ties go to the answer that appeared first in the rows.
- `JsonMergeAggregator` falls back to a majority vote when no output is an object. Its conflict policies are
  `last_writer_wins` (the default), `first_writer_wins` and `error`. Each conflict is listed in
  `metadata["conflicts"]`.

### Events

Pass an `event_handler` to see what the agent does. It is any object with some of these methods, and a
missing method is skipped: `on_step_start`, `on_step_end`, `on_tool_start`, `on_tool_end`, `on_tool_error`
and `on_routing_decision`. `CollectingEventHandler` keeps the events in its `events` list. `LoggingEventHandler` writes
them to the `llmswitch.agentic` logger, and it logs tool names and the names of exception classes, never
argument or result values. Tool events can come from worker threads when tools run in parallel, so a handler
must be safe to call from several threads. A call that is refused before it runs, with arguments that are not
a JSON object or an unknown tool name, still gets an `on_tool_start` and an `on_tool_error` event, and it appears in the
step's `tool_results`.

## Rate limits

**When to use it:** always, when more than one user, job or thread shares a provider account. Limits stop one
caller from exhausting the quota, and keep requests inside the provider's own limits.

Limits are counted per `user_id` (default `"global"`) in a fixed one-minute window:

```python
client = LLMSwitch(
    provider="openai",
    api_key="sk-...",
    model="gpt-4o",
    user_id="team-search",
    rpm=60,                  # requests per minute
    tpm=200_000,             # tokens per minute
    embedding_rpm=300,
    embedding_tpm=1_000_000,
)
```

The defaults are `rpm=10000` and `tpm=1000000`, for chat and for embeddings. Lower them for shared accounts.

**What happens when a limit is reached.** By default the request waits for the window to reopen. If the budget
is still exhausted, it fails with `RateLimitExceededError`:

- `on_rate_limit="raise"` fails immediately instead of waiting.
- `rate_limit_max_wait` (seconds) waits only for short delays. A longer wait raises `RateLimitExceededError`.

**Denied requests do not use budget.** A denied request gives its reservation back, so a later request that
fits in the window is still allowed.

**Shared limits across processes.** In-process limits are per process. To share them, set `redis_url` and
install the `redis` extra. `redis_connect_timeout` (default 1.0 second) bounds how long each check waits for
Redis. If Redis is unavailable, `rate_limit_failure_policy` decides what happens:

- `"memory_fallback"` (default): use in-process limits, so you keep protection without Redis.
- `"fail_open"`: allow the request.
- `"fail_closed"`: raise `RateLimitBackendUnavailableError`.

```python
client = LLMSwitch(
    provider="openai", api_key="sk-...", model="gpt-4o",
    redis_url="redis://cache.internal:6379/0",
    rate_limit_failure_policy="memory_fallback",
)
```

**In-process table size.** In-process limits keep at most 10,000 user windows. When that table is full, the
oldest window is dropped and a warning is logged once. That user's window restarts, so the limit is weaker for
that window, but no user is refused for lack of room. Use Redis for exact limits with many users.

**Streams without usage.** When a stream does not report token usage, its completion is counted from the text
it streamed (content and thinking), with the same tokenizer that counts prompts. This is an estimate, so it can
differ a little from the provider's own count. When the provider does report usage, that count is used.

## Retries, timeouts and the circuit breaker

**Retries** repeat a call after a transient error. This absorbs short provider hiccups without bothering the
caller.

- Retried: rate limits, connection errors and 5xx responses.
- Not retried: timeouts. The provider may already have processed the request, so a retry could bill it twice.
- `max_retries` (default 3) is the number of retries, not attempts. `retry_base_delay` (default 1.0 seconds)
  doubles each time, with jitter.
- `retry_deadline` (seconds) caps the total time one provider call may keep retrying. When the next wait would
  pass the deadline, the last error is raised.

```python
client = LLMSwitch(provider="openai", api_key="sk-...", model="gpt-4o",
                   max_retries=2, retry_base_delay=0.5, retry_deadline=30)
```

**Timeouts** still count as failures for the circuit breaker, and they trigger backup failover, so a
timed-out provider is treated as unavailable.

**The circuit breaker** stops calls to a provider that keeps failing. Use it when you would rather fail fast
than wait through retries against a provider that is down.

```python
client = LLMSwitch(provider="openai", api_key="sk-...", model="gpt-4o",
                   circuit_breaker={"failure_threshold": 5, "reset_timeout": 30})
```

- It counts timeouts, connection errors and 5xx responses. Client errors (4xx) and rate limits do not count.
- After `failure_threshold` failures it opens and refuses calls. After `reset_timeout` seconds it lets one trial
  call through. If the trial succeeds, the breaker closes; if it fails, it opens again.
- A stream you stop reading early still holds the trial slot until you call `close()` on it or let it go out of
  scope. Close streams you abandon.

`circuit_breaker=True` uses the default thresholds.

## Backup failover

**When to use it:** when a second provider can serve the same operation and you want requests to keep working
through a primary outage.

```python
client = LLMSwitch(
    provider="openai",
    api_key="sk-...",
    model="gpt-4o",
    backup={
        "provider": "azure_openai",
        "api_key": "...",
        "endpoint": "...",
        "api_version": "2024-10-21",
        "deployment_name": "gpt-4o",
    },
)
```

- The backup is used when the primary fails with a retryable error or times out.
- The backup must support the same operation. This is checked before the primary is called, so a backup that cannot serve the operation fails clearly.
- For streams, the backup is used only if the primary fails before the first output reaches the caller. After
  that, switching providers would repeat text, so the stream fails instead.
- Embedding failover is all or nothing, as described in [Embeddings](#embeddings).
- Responses requests that create side effects (background requests, MCP calls, computer use) are not retried or
  failed over, so the same action is not performed twice.

`client_options` retries are separate. A `max_retries` inside `client_options` goes to the SDK, while
`max_retries` on the client is the library's own retry loop.

## Context, truncation and token counting

**When to use it:** long conversations or documents that might exceed the model's context window.

By default a request that is too long raises `ContextLengthExceededError`. With `truncate=True`, the oldest
content is trimmed so the request fits:

```python
reply = client.chat.completions.create(messages=long_history, truncate=True)
```

The window is controlled by:

- `completion_context_window` (default 123000 tokens): the budget for input and output together.
- `context_output_reserve_tokens` (default 0): output room kept free for every request, so the answer is not cut.
- `context_overhead_tokens` (default 0): extra protocol overhead you know about, counted against the budget.

**Token counting** uses `tiktoken`. OpenAI model names use their own encoding. Any other name uses the newest
tiktoken encoding, so counts for other providers are estimates. For an exact or provider-specific count, pass
your own function with `token_counter=fn`, where `fn(text, model_name)` returns an integer. It is used for
rate limits and context checks. Truncation still cuts text with tiktoken.

## Performance: clients, sharing and lazy loading

- **Client reuse.** Create one client per configuration and reuse it. Each client keeps its SDK connection pool.
- **`share_clients=True`** reuses one SDK client across clients that have the same configuration. Use it when
  you create many clients, for example one per request.
- **Lazy loading.** A provider's SDK is loaded the first time that provider is used, so unused providers cost
  nothing at import time.
- **Batching** for embeddings (see [Embeddings](#embeddings)) keeps large jobs inside provider limits.
- **Async** clients let one process serve many concurrent requests. Embeddings for providers without native
  async run in worker threads, so they do not block the event loop.

## Observability

**When to use it:** to log, measure or alert on what the library does: retries, failovers, latency and tokens.

Pass `on_event` to receive a dict for each step of a call:

```python
def log_event(event):
    print(event["event"], event["operation"], event["provider"], event.get("latency"))

client = LLMSwitch(provider="openai", api_key="sk-...", model="gpt-4o", on_event=log_event)
```

| `event` | When | Extra fields |
|---|---|---|
| `request` | a call is about to be sent | `stream` |
| `response` | the call returned (for a stream, once it is established) | `stream`, `latency`, `usage` |
| `error` | the call failed | `error`, `status_code`, `latency` |
| `retry` | a transient error is about to be retried | `attempt`, `max_retries`, `delay`, `error`, `status_code` |
| `failover` | the primary provider failed and the backup is used | `error`, `status_code`, and `from_provider` and `to_provider` for calls that are not streams |

Every event also has `operation` (`chat.completions`, `embeddings`, `responses`, or `responses.management`
with a `verb` such as `retrieve`), `provider` and `model`. The callback runs on the calling thread. If it
raises, the error is logged and the call continues.

**Request logging.** Request details are not logged by default. Pass `live_logs=True` on a call to log it at
INFO. The messages or input are replaced with a placeholder, and the other parameters are logged with secrets
redacted. Keep sensitive text out of parameters such as `instructions` when you turn it on.

## Configuration reference

Constructor options for `LLMSwitch` and `AsyncLLMSwitch`. Options that a provider does not use are accepted, so
the same settings work with any provider.

**Identity and provider**

| Option | Default | Meaning |
|---|---|---|
| `provider` | required | One of `openai`, `azure_openai`, `azure_foundry`, `anthropic`, `vertexai`. |
| `model` | provider default | Model name. A `deployment_name` takes precedence. |
| `deployment_name` | none | Azure deployment name. Always used as the model when set. |
| `api_key` | none | Provider API key. |
| `endpoint` | provider default | Custom endpoint or gateway URL. |
| `api_version` | provider default | API version (Azure). |
| `responses_api_version` | derived | API version for Responses calls only. |
| `location`, `credentials_path`, `scopes` | none | Vertex AI settings. |
| `organization`, `project` | none | OpenAI organization and project. |
| `timeout` | SDK default | Request timeout in seconds. |
| `enable_prompt_caching` | `True` | Anthropic cache markers on system messages and tools. |
| `client_options` | none | Extra keyword arguments for the native SDK client, for example `{"max_retries": 3}`. These cannot set `api_key`, `api_version`, `azure_endpoint`, `azure_deployment`, `base_url`, `endpoint`, `organization`, `project`, `timeout`, `credential`, `credentials`, `location`, `vertexai`, `http_options` or `responses_api_version`; use the unified options for those. |
| `user_id` | `"global"` | The key that rate limits are counted against. |

**Rate limits**

| Option | Default | Meaning |
|---|---|---|
| `rpm`, `tpm` | 10000, 1000000 | Requests and tokens per minute for chat. |
| `embedding_rpm`, `embedding_tpm` | 10000, 1000000 | Requests and tokens per minute for embeddings. |
| `on_rate_limit` | `"wait"` | `"wait"` or `"raise"`. |
| `rate_limit_max_wait` | none | Longest wait, in seconds, before raising. |
| `redis_url` | none | Shared limits across processes. |
| `redis_connect_timeout` | 1.0 | Seconds to wait for Redis. |
| `rate_limit_failure_policy` | `"memory_fallback"` | `"memory_fallback"`, `"fail_open"` or `"fail_closed"`. |

**Reliability**

| Option | Default | Meaning |
|---|---|---|
| `max_retries` | 3 | Retries for transient errors. |
| `retry_base_delay` | 1.0 | Seconds before the first retry; doubles each time. |
| `retry_deadline` | none | Maximum total seconds one call may keep retrying. |
| `circuit_breaker` | `False` | `True`, or a dict with `failure_threshold` (5) and `reset_timeout` (30). |
| `backup` | none | Backup provider settings, used on failover. |

**Context and tokens**

| Option | Default | Meaning |
|---|---|---|
| `completion_context_window` | 123000 | Token budget for input and output together. |
| `context_output_reserve_tokens` | 0 | Output room kept free for every request. |
| `context_overhead_tokens` | 0 | Extra protocol overhead to count. |
| `token_counter` | none | Your own function `fn(text, model_name)` for counting. |

**Responses**

| Option | Default | Meaning |
|---|---|---|
| `mcp_policy` | none | Allow-list for remote MCP servers and connectors. |
| `responses_capabilities` | none | Turn Responses features on or off, for example `{"background": True}`. |

**Other**

| Option | Default | Meaning |
|---|---|---|
| `on_event` | none | Callback that receives one dict per event. |
| `share_clients` | `False` | Reuse SDK clients across clients with the same configuration. |

Per-call arguments on `create()` include `stream`, `truncate`, `live_logs`, `max_retries` and provider-specific
keyword arguments. Embedding calls also accept `batching`, `batchsize` and `restore_order`.

## Errors

All errors derive from `LLMSwitchError`. Every error has `status_code`, `provider`, `user_id` and, for rate
limits, `retry_after`. These are `None` when the value is not known.

| Error | Raised when |
|---|---|
| `ConfigurationError` | Constructor settings are invalid, missing, or name an option no provider defines. |
| `RateLimitExceededError` | The budget is exhausted and waiting is not allowed, or the wait is too long. |
| `RateLimitBackendUnavailableError` | Redis is unavailable and the policy is `fail_closed`, or the rate-limit backend is disabled. |
| `ContextLengthExceededError` | The request is too long and `truncate` is off. |
| `ChatCompletionError`, `EmbeddingGenerationError` | A provider failed. `status_code` carries the HTTP status when known. |
| `AuthenticationError` | The provider rejected the credentials. |
| `APIConnectionError` | The provider could not be reached. Retried, and counted by the breaker. |
| `TimeOutError` | The provider did not answer in time. Not retried; fails over to the backup. |
| `ContentFilterError` | The provider blocked the content, including a prompt Vertex AI blocked. |
| `InvalidResponseError` | The provider returned something the library cannot read. |
| `ResponsesError` | A Responses call failed, or the provider has no Responses API (status 501). |
| `InvalidResponseRequestError` | A Responses request is invalid, or uses an MCP server that the policy does not allow. |
| `UnsupportedCapabilityError` | A Responses feature is not available on the configured provider. |
| `ToolLoopLimitError` | `run_tool_loop` reached `max_steps` while the model still asked for tools. |

Calling an operation a provider does not support (for example embeddings on `anthropic`) raises
`LLMSwitchError` before any request is sent.

## Development

```bash
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all,dev]"
pytest
ruff check src tests examples
mypy
```

The tests use mocks and do not contact any provider. The agentic tests skip when the `agentic` extra is not
installed, so `pip install -e ".[redis,dev]"` runs the core suite on its own. CI runs the full suite, the core
suite without the extra, types and lint. The release workflow also tests the built wheel in a clean environment
before anything is published.

## License

MIT. See [LICENSE](LICENSE).
