Metadata-Version: 2.5
Name: dot-inference
Version: 2.0.2
Summary: Unified async Python library for LLM inference and embeddings across multiple providers
Project-URL: Homepage, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference
Project-URL: Repository, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference
Project-URL: Issues, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference/-/issues
Author-email: Kannon For Deep Tech <louis.letarnec@deepika.ai>
License-Expression: AGPL-3.0-or-later
License-File: LICENSE.md
Keywords: embeddings,inference,llm,providers
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: <3.14,>=3.12
Requires-Dist: httpx>=0.27.0
Requires-Dist: openai>=1.0.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: typing-extensions>=4.16
Provides-Extra: mistral
Requires-Dist: mistralai>=2.0.0; extra == 'mistral'
Description-Content-Type: text/markdown

# dot-inference

[![PyPI](https://img.shields.io/pypi/v/dot-inference)](https://pypi.org/project/dot-inference/)
![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)
[![Licence: AGPL v3](https://img.shields.io/badge/licence-AGPL--3.0--or--later-blue)](LICENSE.md)
[![Pipeline](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference/badges/main/pipeline.svg)](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference/-/pipelines)

**One async interface for LLM inference and embeddings, across every provider.**

```python
from dot_inference import InferenceSettings, Message, Role, get_llm_client

client = get_llm_client(InferenceSettings())  # provider and model come from env
response = client.call_sync([Message(role=Role.USER, content="How tall is the Eiffel Tower?")])
print(response.content)
```

## Why dot-inference

Switching LLM provider usually means rewriting call sites: each SDK has its own
client, its own message shape, its own way of doing structured output, streaming
and tool calls. Retries, cost accounting and rate limiting end up reinvented in
every project.

dot-inference puts one async interface in front of OpenAI, Mistral, OpenRouter
and any OpenAI-compatible server. Changing provider is an environment variable,
not a refactor. Retries honouring `Retry-After`, USD cost tracking, rate limiting
and structured output come with it, and a model registry records what each model
can actually do — so an unsupported `reasoning_effort` fails with a clear error
instead of silently doing nothing.

## Features

- Multi-provider: OpenAI, Mistral, OpenRouter, and any OpenAI-compatible server (vLLM, SGLang, Ollama)
- Structured outputs backed by Pydantic models
- Rich event streaming: text, reasoning, incremental tool-call arguments, tool calls, usage and cost
- Native tool calling with `tool_choice` control (`auto` / `required` / `none`)
- Embeddings with automatic batching
- Retries on transient errors, honouring `Retry-After`, with exponential backoff
- Cost tracking in tokens and USD
- Rate limiting through an async sliding window
- YAML model registry carrying pricing and per-model capabilities
- Per-model default provider resolution — pick a model, let the registry choose the provider
- Configurable HTTP timeouts (`DOTI_GENERATION__TIMEOUT`, `DOTI_EMBEDDING__TIMEOUT`)
- Mock clients for testing without API calls
- Sync wrappers (`call_sync`, `embed_sync`) for non-async codebases

## Installation

```bash
pip install dot-inference

# With the native Mistral SDK:
pip install 'dot-inference[mistral]'
```

Requires Python 3.12+.

## Configuration

Settings come from `DOTI_`-prefixed environment variables, using `__` as the
nesting delimiter — or from an env file, modelled on `.env.template`:

```bash
DOTI_GENERATION__PROVIDER=openai
DOTI_GENERATION__OPENAI__MODEL_NAME=gpt-5-mini
DOTI_GENERATION__OPENAI__API_KEY=sk-...
```

API keys are held as `SecretStr`, so a settings object caught in a traceback or
an error report never carries the key in clear text.

## Quick start

### From settings

```python
import asyncio
from dot_inference import InferenceSettings, Message, Role, get_llm_client

settings = InferenceSettings()  # reads from DOTI_* env vars
client = get_llm_client(settings)

messages = [Message(role=Role.USER, content="What is the height of the Eiffel Tower?")]
response = asyncio.run(client.call(messages))
# or synchronously:
response = client.call_sync(messages)
print(response.content)
```

### Explicit provider and model

```python
from dot_inference import InferenceSettings, Message, Provider, Role, get_llm_client

settings = InferenceSettings()
client = get_llm_client(settings, provider=Provider.OPENAI, model="gpt-5-mini")
response = asyncio.run(client.call([Message(role=Role.USER, content="Hello!")]))
```

### Reasoning models

Use `ReasoningEffort` to control thinking budget on reasoning models (e.g. `gpt-5*`, `o-series`):

```python
from dot_inference import (
    InferenceSettings,
    Message,
    Provider,
    ReasoningEffort,
    Role,
    get_llm_client,
)

settings = InferenceSettings()
client = get_llm_client(
    settings,
    provider=Provider.OPENAI,
    model="gpt-5-mini",
    reasoning_effort=ReasoningEffort.HIGH,
)
response = asyncio.run(client.call([Message(role=Role.USER, content="Solve this step by step...")]))
```

### Structured output

```python
from pydantic import BaseModel
from dot_inference import InferenceSettings, Message, Provider, Role, get_llm_client


class Answer(BaseModel):
    answer: str
    confidence: float


settings = InferenceSettings()
client = get_llm_client(settings, provider=Provider.OPENAI, model="gpt-5-mini")
response = asyncio.run(
    client.call(
        [Message(role=Role.USER, content="What is 2+2?")],
        response_format=Answer,
    )
)
print(response.structured_output)  # Answer(answer='4', confidence=1.0)
```

### Streaming

`stream()` yields typed `StreamEvent` objects:

| Event | Description |
| ----- | ----------- |
| `TextDelta` | A chunk of text content (`content: str`) |
| `ReasoningDelta` | A chunk of reasoning text, when the provider exposes it |
| `InputJsonDelta` | A chunk of tool-call argument JSON (`tool_call_index`, `tool_name`, `partial_json`) — emitted incrementally before `ToolCallDone` |
| `ToolCallDone` | A fully assembled tool call (`tool_call: ToolCall`) |
| `StreamUsage` | Token counts and cost (`prompt_tokens`, `completion_tokens`, `cost`) |
| `StreamDone` | Terminal event carrying the final `LLMResponse` — identical to what `call()` returns |

**Simple text streaming** — use `stream_text()` for plain `str` chunks:

```python
from dot_inference import InferenceSettings, Message, Role, get_llm_client

settings = InferenceSettings()
client = get_llm_client(settings)


async def main():
    async for chunk in client.stream_text([Message(role=Role.USER, content="Tell me a story")]):
        print(chunk, end="", flush=True)


asyncio.run(main())
```

**Rich event streaming** — consume all events for text + usage + tool calls:

```python
from dot_inference import (
    InferenceSettings,
    Message,
    Role,
    get_llm_client,
    ToolDef,
    TextDelta,
    InputJsonDelta,
    ToolCallDone,
    StreamUsage,
    StreamDone,
)


async def main():
    settings = InferenceSettings()
    client = get_llm_client(settings)

    async for event in client.stream(
        [Message(role=Role.USER, content="What's the weather in Paris?")],
        tools=[ToolDef(name="get_weather", description="Get weather", parameters={...})],
    ):
        if isinstance(event, TextDelta):
            print(event.content, end="", flush=True)
        elif isinstance(event, InputJsonDelta):
            print(event.partial_json, end="", flush=True)
        elif isinstance(event, ToolCallDone):
            print(f"Tool call: {event.tool_call.name}({event.tool_call.arguments})")
        elif isinstance(event, StreamUsage):
            print(f"Tokens: {event.prompt_tokens} in / {event.completion_tokens} out")
        elif isinstance(event, StreamDone):
            # event.response is a full LLMResponse, works with CostTracker
            tracker.record(event.response)


asyncio.run(main())
```

### Embeddings

```python
from dot_inference import InferenceSettings, get_embedding_client

settings = InferenceSettings()
client = get_embedding_client(settings)
vectors = asyncio.run(client.embed(["Hello world", "Bonjour le monde"]))
print(len(vectors), len(vectors[0]))  # 2, 1536
```

### Cost tracking

```python
from dot_inference import CostTracker, InferenceSettings, Message, Provider, Role, get_llm_client

settings = InferenceSettings()
tracker = CostTracker()
client = get_llm_client(settings, provider=Provider.OPENROUTER, model="openai/gpt-5-mini")

response = asyncio.run(client.call([Message(role=Role.USER, content="Hi")]))
tracker.record(response)
print(f"Tokens: {tracker.total_tokens}, Cost: ${tracker.total_cost:.4f}")
```

### Retries

Transient provider errors (5xx, rate limits, timeouts) are retried automatically with exponential backoff. When the provider returns a `Retry-After` header, the wait is honored (clamped to 60s). Client-side errors (400, 401, 403, 404, 422) are not retried.

Attach `on_retry` to observe retry events — useful for UI feedback:

```python
client = get_llm_client(settings)
client.on_retry = lambda attempt, max_attempts, err: print(f"Retry {attempt}/{max_attempts}: {err}")
```

## On-premise inference (vLLM, SGLang, Ollama)

Point the `local` provider at any OpenAI-compatible endpoint:

```shell
export DOTI_GENERATION__PROVIDER=local
export DOTI_GENERATION__LOCAL__MODEL_NAME=openai/gpt-oss-120b
export DOTI_GENERATION__LOCAL__BASE_URL=http://localhost:8000/v1
```

```python
from dot_inference import InferenceSettings, Message, Role, get_llm_client

client = get_llm_client(InferenceSettings())
response = client.call_sync([Message(role=Role.USER, content="How tall is the Eiffel Tower?")])
print(response.content)
```

## Testing without API calls

```python
from dot_inference.mock import MockLLMClient, MockEmbeddingClient
```

Both return deterministic responses — including structured output built from
your Pydantic schema — so tests never reach a provider.

## Stability

`dot-inference` follows semantic versioning: everything exported from the top-level
package is covered, anything underscore-prefixed is internal and may change in
any release. Public names are never removed without a deprecation period.

```toml
dependencies = ["dot-inference>=1.0,<2"]
```

See [docs/VERSIONING.md](docs/VERSIONING.md) for the full policy.

## Roadmap

- [ ] Azure OpenAI provider
- [ ] Logging integration (dot-logger)
- [ ] Jinja templates for prompt management

## Documentation

| Document | Contents |
|---|---|
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Internal design and module layout |
| [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) | Environment setup, tests, code style |
| [docs/VERSIONING.md](docs/VERSIONING.md) | Versioning, deprecation policy, how to depend on this package |
| [docs/PUBLISHING.md](docs/PUBLISHING.md) | Cutting a release |
| [CHANGELOG.md](CHANGELOG.md) | Release history |

## Contributing

Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the DCO
sign-off requirement, the licensing terms that apply to contributions, and how to
submit a change.

## Licence

Copyright (C) 2026 Kannon For Deep Tech (deepika)

This software is distributed under the GNU Affero General Public License,
version 3 or later — see [LICENSE.md](LICENSE.md).

A commercial licence is available for use in proprietary environments.
Contact: louis.letarnec@deepika.ai
