Metadata-Version: 2.4
Name: model-calling
Version: 0.3.0
Summary: A unified API for calling multiple LLM providers through a consistent, OpenAI-compatible interface
Author: Wes Jackson
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/rdwj/model-calling
Project-URL: Source, https://github.com/rdwj/model-calling
Project-URL: Issues, https://github.com/rdwj/model-calling/issues
Project-URL: Documentation, https://github.com/rdwj/model-calling/tree/main/docs
Keywords: llm,api,ollama,vllm,openai,anthropic,ai,language-model
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: httpx==0.28.1
Requires-Dist: pydantic==2.14.1
Provides-Extra: server
Requires-Dist: fastapi==0.150.0; extra == "server"
Requires-Dist: uvicorn==0.54.0; extra == "server"
Requires-Dist: python-dotenv==1.2.4; extra == "server"
Requires-Dist: PyYAML==6.0.3; extra == "server"
Provides-Extra: moderation
Requires-Dist: torch==2.14.1; extra == "moderation"
Requires-Dist: transformers==5.20.2; extra == "moderation"
Provides-Extra: examples
Requires-Dist: requests==2.34.2; extra == "examples"
Requires-Dist: aiohttp==3.14.5; extra == "examples"
Requires-Dist: numpy==2.4.6; python_version < "3.12" and extra == "examples"
Requires-Dist: numpy==2.5.4; python_version >= "3.12" and extra == "examples"
Requires-Dist: scikit-learn==1.9.1; extra == "examples"
Dynamic: license-file

# model-calling

[![PyPI version](https://img.shields.io/pypi/v/model-calling.svg)](https://pypi.org/project/model-calling/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

An LLM client library for Python that speaks OpenAI Chat Completions and Anthropic Messages over Ollama, vLLM, OpenAI-compatible, and OpenRouter backends. Use it in-process or as a gateway.

## Install

```bash
pip install model-calling              # library only
pip install 'model-calling[server]'    # adds the gateway (FastAPI + uvicorn)
pip install 'model-calling[moderation]' # adds moderation endpoint (torch + transformers)
```

The library depends on httpx and pydantic, pinned exactly (`httpx==0.28.1`, `pydantic==2.14.1`) so every release is tested against the versions it installs; an app that pins other versions of either will conflict. Importing it never loads FastAPI and never reads `.env`.

## Status

0.3.0 covers Chat Completions, Messages, and the stateless OpenAI Responses API, sync and async with streaming, over Ollama, vLLM, OpenAI, OpenRouter, Anthropic, and Cohere v2, plus the alias-routed gateway. Responses also works as a backend, through OpenRouter. Cohere has no live coverage: its tests replay fixtures written by hand from Cohere's API reference. Not yet available: retries and fallbacks. Upgrading from 0.1.x involves breaking changes; see the [migration guide](https://github.com/rdwj/model-calling/blob/main/docs/migration-guide.md). Remaining work is tracked in [#8](https://github.com/rdwj/model-calling/issues/8).

## Quick start

```python
from model_calling import Client

client = Client()

# Chat Completions (OpenAI format)
response = client.completion(
    "ollama/qwen3:4b",
    [{"role": "user", "content": "What is the capital of France?"}]
)
print(response.choices[0].message.content)

# Anthropic Messages
message = client.messages(
    "vllm/gpt-oss-20b",
    [{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=256
)
# Reasoning models put a thinking block first; print the text.
print(next(b.text for b in message.content if b.type == "text"))

# Embeddings
vectors = client.embedding("ollama/nomic-embed-text", ["hello", "world"])
print(vectors.data[0].embedding[:5])

client.close()
```

### Streaming

```python
from model_calling import Client

client = Client()

# Chat Completions stream
for chunk in client.completion(
    "ollama/qwen3:4b",
    [{"role": "user", "content": "Count to five."}],
    stream=True
):
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

# Messages stream
for event in client.messages(
    "vllm/gpt-oss-20b",
    [{"role": "user", "content": "Count to five."}],
    max_tokens=256,
    stream=True
):
    if event.type == "content_block_delta" and event.delta.type == "text_delta":
        print(event.delta.text, end="", flush=True)

client.close()
```

### Async

```python
from model_calling import Client
import asyncio

async def main():
    async with Client() as client:
        response = await client.acompletion(
            "ollama/qwen3:4b",
            [{"role": "user", "content": "Hi!"}]
        )
        print(response.choices[0].message.content)

asyncio.run(main())
```

Module-level functions are also available:

```python
import model_calling

response = model_calling.completion("ollama/qwen3:4b", [...])
vectors = model_calling.embedding("ollama/nomic-embed-text", ["hello"])
```

## Backends

Model routing uses `prefix/name`. Unprefixed names go to Ollama.

| Prefix | Default base_url | Credentials |
|--------|------------------|-------------|
| `anthropic` | `https://api.anthropic.com/v1` | `ANTHROPIC_API_KEY` |
| `ollama` | `http://localhost:11434` | none |
| `vllm` | `http://localhost:8000/v1` | `api_key` in config if needed |
| `openai` | `https://api.openai.com/v1` | `OPENAI_API_KEY`, optional `OPENAI_ORG_ID` |
| `openrouter` | `https://openrouter.ai/api/v1` | `OPENROUTER_API_KEY` |
| `cohere` | `https://api.cohere.com/v2` | `COHERE_API_KEY` |

The library reads only those environment variables. Override per call with `api_base=` / `api_key=`, or per client via `config`.

The `anthropic` prefix speaks Anthropic Messages to any server that serves it at `{base_url}/messages`: Anthropic, OpenRouter (`https://openrouter.ai/api/v1`), or vLLM (`http://localhost:8000/v1`). A `messages()` call reaches it unchanged. A `completion()` call is translated, and since Messages requires `max_tokens`, one sent without it gets 4096.

The `cohere` prefix speaks Cohere's v2 chat and embed APIs. It has no live coverage: the project has no Cohere API key, so its tests replay fixtures written by hand from the [v2 API reference](https://docs.cohere.com/reference/chat) rather than recorded. Requests v2 can't express are refused: a named `tool_choice` (v2 has only required and none), `parallel_tool_calls: false`, `strict` on some tools but not all, a thinking effort without a token budget, and a response format alongside tools. Embeddings send `input_type: search_document` unless the request names another `input_type`.

Both formats work over every backend. You can call `client.messages()` against Ollama, vLLM, OpenAI, or OpenRouter; the library translates to their Chat Completions APIs. Likewise for `client.completion()` against an Anthropic backend.

## Strictness

Strict by default: a parameter the backend cannot honor raises `UnsupportedParamsError` (a `BadRequestError` with a `.params` list) before anything is sent. Pass `drop_params=True` (client-wide or per call) to drop them instead.

Example: Ollama cannot force a tool call (`tool_choice` "required" or `{"type": "any"}`). Chat Completions backends cannot take `thinking.budget_tokens`.

Some parameters are always dropped silently: `cache_control` (a caching hint that changes no output) and `user` for Ollama embeddings.

## Errors and timeouts

Every exception the library raises subclasses `model_calling.APIError` and carries `.provider`, `.status` (the backend's HTTP status, or `None` when no response arrived), and `.body`. The subclasses mirror the `openai` SDK's: `BadRequestError` (including `UnsupportedParamsError`), `AuthenticationError`, `PermissionDeniedError`, `NotFoundError`, `RateLimitError`, `InternalServerError`, `APIConnectionError`, `APITimeoutError`, and `APIResponseValidationError`.

```python
import model_calling

try:
    response = model_calling.completion("openrouter/openai/gpt-4o-mini", messages)
except model_calling.RateLimitError:
    ...  # back off and retry; the library does not retry for you
except model_calling.APIError as e:
    print(e.provider, e.status, e.message)
```

Requests time out after 300 seconds by default, including a stalled stream. Set `timeout` (in seconds) per prefix in the client's config, or pass it to a single call.

## Gateway

The gateway is a FastAPI server running on the same library code. Apps talk to it with the official `openai` or `anthropic` SDK and name models by alias; a YAML file decides which backend serves each alias. Moving from Ollama in development to vLLM in production is an edit to that file. The app doesn't change, and the gateway picks up the edit without a restart.

```yaml
# model-calling.yaml (or the file MODEL_CALLING_CONFIG names)
backends:
  prod: {provider: vllm, base_url: https://vllm.example.com/v1, api_key_env: VLLM_API_KEY}
models:
  chat: {backend: ollama, model: qwen3:4b}   # development
  # chat: {backend: prod, model: gpt-oss-20b}   # production
  embed: {backend: ollama, model: nomic-embed-text, dimensions: 768}
```

```bash
model-calling  # or: python -m model_calling
```

Binds `0.0.0.0`, port from `PORT` (default 8000). The gateway reads `.env` (python-dotenv) at startup.

### Endpoints

- `POST /v1/chat/completions` (OpenAI Chat Completions; streaming SSE ending `data: [DONE]`)
- `POST /v1/messages` (Anthropic Messages; streaming SSE with `event:` lines; errors as Anthropic error bodies)
- `POST /v1/messages/count_tokens` (Anthropic token counting, answered with a local estimate since no backend here has a counter)
- `POST /v1/responses` (OpenAI Responses, stateless: nothing is stored, and requests that need stored state or built-in tools get a 400 naming the field)
- `POST /v1/embeddings` (OpenAI Embeddings)
- `GET /v1/models` (the configured aliases), `POST /v1/moderations`, `GET /` health

Point the official SDKs at it:

```python
import openai
import anthropic

openai_client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
anthropic_client = anthropic.Anthropic(base_url="http://localhost:8000", api_key="unused")

reply = openai_client.chat.completions.create(
    model="chat", messages=[{"role": "user", "content": "Hello"}]
)
```

The gateway does not check client keys; it uses its own backend keys from its environment. Any name that isn't an alias, including a backend-prefixed one such as `ollama/qwen3:4b`, gets 404. See [Model Aliases](https://github.com/rdwj/model-calling/blob/main/docs/advanced.md#model-aliases) for the file format and reload behavior.

### Strict mode

The gateway drops unsupported parameters by default. Set `MODEL_CALLING_STRICT_PARAMS=true` (accepts true/false/1/0/yes/no) to turn on strict mode: 400 with message "model '\<alias\>' cannot honor \<param\>".

### Backend URLs

For backends the alias file names without a `base_url`, the gateway takes URLs from its environment: `OLLAMA_BASE_URL` is Ollama's native root (`http://ollama:11434`), while `VLLM_BASE_URL` and `MODEL_CALLING_ANTHROPIC_BASE_URL` are API roots with `/v1`. `VLLM_API_KEY` is sent to vLLM if set.

Environment: `MODEL_CALLING_CONFIG`, `OPENAI_API_KEY`, `OPENAI_ORG_ID`, `OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY`, `OLLAMA_BASE_URL`, `VLLM_BASE_URL`, `VLLM_API_KEY`, `MODEL_CALLING_ANTHROPIC_BASE_URL`, `PORT`, `MODEL_CALLING_STRICT_PARAMS`. The `model-calling` command also reads a `.env` file from the directory it starts in.

## Containers

Build with podman:

```bash
./run_podman.sh         # Containerfile directly
./run_podman_compose.sh # podman-compose.yml
```

UBI 9 Python 3.12 base, runs as non-root. FIPS variant in `Containerfile-FIPS`.

## Testing

```bash
uv sync
uv run pytest                 # offline suite; no network
uv run pytest -m live         # live smoke tests via OpenRouter (OPENROUTER_API_KEY)
```

The offline suite runs the gateway in-process against recorded backend fixtures and validates output with the official `openai` and `anthropic` SDK types. See [tests/README.md](https://github.com/rdwj/model-calling/blob/main/tests/README.md) for the harness and fixture recorder.

## Documentation

The [documentation](https://github.com/rdwj/model-calling/blob/main/docs/README.md) covers the library API, the gateway, model aliases, and migrating from 0.1.x.

## License

Apache-2.0. See [LICENSE](https://github.com/rdwj/model-calling/blob/main/LICENSE) and [NOTICE](https://github.com/rdwj/model-calling/blob/main/NOTICE).

## Acknowledgements

This project was developed with assistance from AI tools. It was inspired by the need for a flexible approach to building LLM applications with locally-hosted and cloud models.
