Metadata-Version: 2.5
Name: dot-inference
Version: 3.1.0
Summary: Unified async Python library for LLM inference and embeddings across multiple providers
Project-URL: Homepage, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference
Project-URL: Repository, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference
Project-URL: Issues, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference/-/issues
Author-email: Kannon For Deep Tech <louis.letarnec@deepika.ai>
License-Expression: MIT
License-File: LICENSE.md
Keywords: embeddings,inference,llm,providers
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: <3.14,>=3.12
Requires-Dist: httpx>=0.27.0
Requires-Dist: openai>=1.0.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: typing-extensions>=4.16
Provides-Extra: mistral
Requires-Dist: mistralai>=2.0.0; extra == 'mistral'
Description-Content-Type: text/markdown

# dot-inference

[![PyPI](https://img.shields.io/pypi/v/dot-inference)](https://pypi.org/project/dot-inference/)
![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)
[![Licence: MIT](https://img.shields.io/badge/licence-MIT-blue)](LICENSE.md)
[![Pipeline](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference/badges/main/pipeline.svg)](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference/-/pipelines)

**One async interface for LLM inference and embeddings, across every provider.**

```python
from dot_inference import InferenceSettings, Message, Role, get_llm_client

client = get_llm_client(InferenceSettings())  # provider and model come from env
response = client.call_sync([Message(role=Role.USER, content="How tall is the Eiffel Tower?")])
print(response.content)
```

## Why dot-inference

Switching LLM provider usually means rewriting call sites: each SDK has its own
client, its own message shape, its own way of doing structured output, streaming
and tool calls. Retries, cost accounting and rate limiting end up reinvented in
every project.

dot-inference puts one async interface in front of OpenAI, Mistral, OpenRouter
and any OpenAI-compatible server. Changing provider is an environment variable,
not a refactor. Retries honouring `Retry-After`, USD cost tracking, rate limiting
and structured output come with it, and a model registry records what each model
can actually do — so an unsupported `reasoning_effort` fails with a clear error
instead of silently doing nothing.

## Features

- Multi-provider: OpenAI, Mistral, OpenRouter, and any OpenAI-compatible server (vLLM, SGLang, Ollama)
- Structured outputs backed by Pydantic models
- Rich event streaming: text, reasoning, incremental tool-call arguments, tool calls, usage and cost
- Image input: URLs, local files or bytes, with per-model vision support in the registry
- Native tool calling with `tool_choice` control (`auto` / `required` / `none` / a tool name) and strict tool schemas
- Provider-specific request fields per call (`extra_body`)
- Embeddings with automatic batching
- Retries on transient errors, honouring `Retry-After`, with exponential backoff
- Cost tracking in tokens and USD
- Rate limiting through an async sliding window
- YAML model registry carrying pricing and per-model capabilities
- Per-model default provider resolution — pick a model, let the registry choose the provider
- Configurable HTTP timeouts (`DOTI_GENERATION__TIMEOUT`, `DOTI_EMBEDDING__TIMEOUT`)
- Mock clients for testing without API calls
- Sync wrappers (`call_sync`, `embed_sync`) for non-async codebases

## Installation

```bash
pip install dot-inference

# With the native Mistral SDK:
pip install 'dot-inference[mistral]'
```

Requires Python 3.12+.

## Configuration

Settings come from `DOTI_`-prefixed environment variables, using `__` as the
nesting delimiter — or from an env file, modelled on `.env.template`:

```bash
DOTI_GENERATION__PROVIDER=openai
DOTI_GENERATION__OPENAI__MODEL_NAME=openai/gpt-5-mini
DOTI_GENERATION__OPENAI__API_KEY=sk-...
```

Model names are registry ids (`openai/gpt-5-mini`, whatever the provider); each
provider is sent its own API name. A name missing from the registry is sent as
is, with a warning, and without the registry's capabilities and prices.

API keys are held as `SecretStr`, so a settings object caught in a traceback or
an error report never carries the key in clear text.

## Quick start

### From settings

```python
import asyncio
from dot_inference import InferenceSettings, Message, Role, get_llm_client

settings = InferenceSettings()  # reads from DOTI_* env vars
client = get_llm_client(settings)

messages = [Message(role=Role.USER, content="What is the height of the Eiffel Tower?")]
response = asyncio.run(client.call(messages))
# or synchronously:
response = client.call_sync(messages)
print(response.content)
```

### Explicit provider and model

```python
from dot_inference import InferenceSettings, Message, Provider, Role, get_llm_client

settings = InferenceSettings()
client = get_llm_client(settings, provider=Provider.OPENAI, model="openai/gpt-5-mini")
response = asyncio.run(client.call([Message(role=Role.USER, content="Hello!")]))
```

### Reasoning models

Use `ReasoningEffort` to control thinking budget on reasoning models (e.g. `gpt-5*`, `o-series`):

```python
from dot_inference import (
    InferenceSettings,
    Message,
    Provider,
    ReasoningEffort,
    Role,
    get_llm_client,
)

settings = InferenceSettings()
client = get_llm_client(
    settings,
    provider=Provider.OPENAI,
    model="openai/gpt-5-mini",
    reasoning_effort=ReasoningEffort.HIGH,
)
response = asyncio.run(client.call([Message(role=Role.USER, content="Solve this step by step...")]))
```

### Structured output

```python
from pydantic import BaseModel
from dot_inference import InferenceSettings, Message, Provider, Role, get_llm_client


class Answer(BaseModel):
    answer: str
    confidence: float


settings = InferenceSettings()
client = get_llm_client(settings, provider=Provider.OPENAI, model="openai/gpt-5-mini")
response = asyncio.run(
    client.call(
        [Message(role=Role.USER, content="What is 2+2?")],
        response_format=Answer,
    )
)
print(response.structured_output)  # Answer(answer='4', confidence=1.0)
```

OpenAI's strict mode wants every object closed (`"additionalProperties": false`) and every property listed in `required`. When the request reaches OpenAI (the `openai` provider, or an `openai/...` model on OpenRouter), a Pydantic model's schema is rewritten to those rules: an optional field becomes required but nullable, so the model answers `null` and Pydantic still gives `None`. A dict schema is sent as written and must follow the rules itself. Details and examples: [docs/provider-quirks/strict-schemas.md](docs/provider-quirks/strict-schemas.md).

### Images

Give a message a list of content parts: `text_part()` for text, `image_part()`
for an image — an `http(s)://` or `data:` URL, a local file path, or raw bytes
(inlined as base64).

```python
from dot_inference import Message, Role, image_part, text_part

message = Message(
    role=Role.USER,
    content=[
        text_part("What is on this receipt?"),
        image_part("receipt.jpg"),
        image_part("https://example.com/logo.png", detail="low"),
    ],
)
response = client.call_sync([message])
```

The registry marks the models that read images (`supports_vision`). Sending an
image to a model registered without it raises `ValueError` before any request;
an unregistered model gets the request as-is.

### Streaming

`stream()` yields typed `StreamEvent` objects:

| Event | Description |
| ----- | ----------- |
| `TextDelta` | A chunk of text content (`content: str`) |
| `ReasoningDelta` | A chunk of reasoning text, when the provider exposes it |
| `InputJsonDelta` | A chunk of tool-call argument JSON (`tool_call_index`, `tool_name`, `partial_json`) — emitted incrementally before `ToolCallDone` |
| `ToolCallDone` | A fully assembled tool call (`tool_call: ToolCall`) |
| `StreamUsage` | Token counts and cost (`prompt_tokens`, `completion_tokens`, `cost`) |
| `StreamDone` | Terminal event carrying the final `LLMResponse` — identical to what `call()` returns |

**Simple text streaming** — use `stream_text()` for plain `str` chunks:

```python
from dot_inference import InferenceSettings, Message, Role, get_llm_client

settings = InferenceSettings()
client = get_llm_client(settings)


async def main():
    async for chunk in client.stream_text([Message(role=Role.USER, content="Tell me a story")]):
        print(chunk, end="", flush=True)


asyncio.run(main())
```

**Rich event streaming** — consume all events for text + usage + tool calls:

```python
from dot_inference import (
    InferenceSettings,
    Message,
    Role,
    get_llm_client,
    ToolDef,
    TextDelta,
    InputJsonDelta,
    ToolCallDone,
    StreamUsage,
    StreamDone,
)


async def main():
    settings = InferenceSettings()
    client = get_llm_client(settings)

    async for event in client.stream(
        [Message(role=Role.USER, content="What's the weather in Paris?")],
        tools=[ToolDef(name="get_weather", description="Get weather", parameters={...})],
    ):
        if isinstance(event, TextDelta):
            print(event.content, end="", flush=True)
        elif isinstance(event, InputJsonDelta):
            print(event.partial_json, end="", flush=True)
        elif isinstance(event, ToolCallDone):
            print(f"Tool call: {event.tool_call.name}({event.tool_call.arguments})")
        elif isinstance(event, StreamUsage):
            print(f"Tokens: {event.prompt_tokens} in / {event.completion_tokens} out")
        elif isinstance(event, StreamDone):
            # event.response is a full LLMResponse, works with CostTracker
            tracker.record(event.response)


asyncio.run(main())
```

### Embeddings

```python
from dot_inference import InferenceSettings, get_embedding_client

settings = InferenceSettings()
client = get_embedding_client(settings)
vectors = asyncio.run(client.embed(["Hello world", "Bonjour le monde"]))
print(len(vectors), len(vectors[0]))  # 2, 1536
```

### Cost tracking

```python
from dot_inference import CostTracker, InferenceSettings, Message, Provider, Role, get_llm_client

settings = InferenceSettings()
tracker = CostTracker()
client = get_llm_client(settings, provider=Provider.OPENROUTER, model="openai/gpt-5-mini")

response = asyncio.run(client.call([Message(role=Role.USER, content="Hi")]))
tracker.record(response)
print(f"Tokens: {tracker.total_tokens}, Cost: ${tracker.total_cost:.4f}")
```

### Retries

Transient provider errors (5xx, rate limits, timeouts) are retried automatically with exponential backoff. When the provider returns a `Retry-After` header, the wait is honored (clamped to 60s). Client-side errors (400, 401, 403, 404, 422) are not retried.

Attach `on_retry` to observe retry events — useful for UI feedback:

```python
client = get_llm_client(settings)
client.on_retry = lambda attempt, max_attempts, err: print(f"Retry {attempt}/{max_attempts}: {err}")
```

`max_attempts` sets the total number of attempts per request (default 4 for OpenAI-compatible clients, 2 for Mistral). Set it to 1 to surface every error immediately, e.g. when measuring latency:

```python
client = get_llm_client(settings, max_attempts=1)
```

### Strict tools, forced tool, provider-specific fields

```python
class WeatherArgs(BaseModel):
    city: str
    unit: Literal["C", "F"] | None = None


tools = [
    ToolDef(name="get_weather", description="Get weather", parameters=WeatherArgs, strict=True)
]

response = await client.call(
    messages,
    tools=tools,
    tool_choice="get_weather",          # force this tool; or "auto" / "required" / "none"
    extra_body={"ignore_eos": True},    # sent as-is (vLLM, SGLang, llama.cpp…)
)
args = tools[0].parse_arguments(response.tool_calls[0].arguments)  # WeatherArgs(city='Paris', unit=None)
```

`parameters` is a JSON schema dict, sent as written, or a Pydantic model. A model's schema is sent, rewritten to OpenAI's strict rules when `strict=True` (for every provider: OpenAI and Claude both reject a strict tool with an open object). A call's `arguments` stays the JSON string the provider sent; `ToolDef.parse_arguments()` validates it into the model, raising `ValidationError` when it does not match.

`strict=True` asks the server to constrain the arguments to the schema exactly; it is only sent when set, and OpenAI then imposes its own schema rules ([strict-schemas.md](docs/provider-quirks/strict-schemas.md)). Some models cannot be forced to call a tool (Claude Opus 5.5 and Sonnet 5.5): `"required"` or a tool name raises `ValueError` there ([tool-choice.md](docs/provider-quirks/tool-choice.md)). `extra_body` fields win over those the client adds itself (OpenRouter routing, `cache_control`); the Mistral client rejects `extra_body`.

## On-premise inference (vLLM, SGLang, Ollama)

Point the `local` provider at any OpenAI-compatible endpoint:

```shell
export DOTI_GENERATION__PROVIDER=local
export DOTI_GENERATION__LOCAL__MODEL_NAME=openai/gpt-oss-120b
export DOTI_GENERATION__LOCAL__BASE_URL=http://localhost:8000/v1
```

```python
from dot_inference import InferenceSettings, Message, Role, get_llm_client

client = get_llm_client(InferenceSettings())
response = client.call_sync([Message(role=Role.USER, content="How tall is the Eiffel Tower?")])
print(response.content)
```

`list_served_models(base_url)` returns the model ids the server actually loaded
(`GET /models`), for any model. To send `reasoning_effort` to a model the
registry does not know, declare it (`temperature` is still sent alongside
it, as for any model with `supports_temperature`, the default):

```python
from dot_inference import ModelConfig, ProviderModelConfig, list_served_models, register_model

base_url = "http://localhost:8000/v1"
served = (await list_served_models(base_url))[0]
register_model(ModelConfig(
    model_id=served,
    supports_reasoning=True,
    supports_vision=True,  # if the served model reads images
    default_provider="local",
    providers={"local": ProviderModelConfig(api_model_name=served)},
))
```

## Testing without API calls

```python
from dot_inference.mock import MockLLMClient, MockEmbeddingClient
```

Both return deterministic responses — including structured output built from
your Pydantic schema — so tests never reach a provider.

## Stability

`dot-inference` follows semantic versioning: everything exported from the top-level
package is covered, anything underscore-prefixed is internal and may change in
any release. Public names are never removed without a deprecation period.

```toml
dependencies = ["dot-inference>=3.0,<4"]
```

See [docs/VERSIONING.md](docs/VERSIONING.md) for the full policy.

## Roadmap

- [ ] Azure OpenAI provider
- [ ] Logging integration (dot-logger)
- [ ] Jinja templates for prompt management

## Documentation

| Document | Contents |
|---|---|
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Internal design and module layout |
| [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) | Environment setup, tests, code style |
| [docs/VERSIONING.md](docs/VERSIONING.md) | Versioning, deprecation policy, how to depend on this package |
| [docs/PUBLISHING.md](docs/PUBLISHING.md) | Cutting a release |
| [docs/provider-quirks/](docs/provider-quirks/) | Provider behaviours that differ from the OpenAI format: `tool_choice` support, strict schemas, OpenRouter error bodies |
| [CHANGELOG.md](CHANGELOG.md) | Release history |

## Contributing

Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the
licensing terms that apply to contributions and how to submit a change.

## Licence

Copyright (C) 2026 Kannon For Deep Tech (deepika)

This software is distributed under the MIT License — see
[LICENSE.md](LICENSE.md).
