Metadata-Version: 2.4
Name: llm-route
Version: 0.8.0
Summary: Intelligent LLM routing with weighted least-outstanding selection, 429 cooldown, and request class isolation
License-Expression: MIT
Keywords: azure,openai,anthropic,claude,routing,load-balancing,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openai<4,>=2.30
Requires-Dist: pydantic<3,>=2.12
Requires-Dist: pydantic-settings<3,>=2.14
Provides-Extra: anthropic
Requires-Dist: anthropic<2,>=0.92; extra == "anthropic"
Provides-Extra: entra
Requires-Dist: azure-identity<2,>=1.15; extra == "entra"
Provides-Extra: redis
Requires-Dist: redis==7.4.0; extra == "redis"
Provides-Extra: dev
Requires-Dist: pre-commit==4.5.1; extra == "dev"
Requires-Dist: pytest==9.0.3; extra == "dev"
Requires-Dist: pytest-asyncio==1.3.0; extra == "dev"
Requires-Dist: pytest-cov==7.1.0; extra == "dev"
Requires-Dist: ruff==0.15.9; extra == "dev"
Requires-Dist: mypy==1.20.0; extra == "dev"
Dynamic: license-file

# llm-route

Intelligent LLM routing library for Python. Drop-in replacement for direct Azure OpenAI calls with automatic load balancing, 429 failover, and request class isolation.

Built as a minimal-dependency alternative to LiteLLM Router, focused on security and transparency.

## Features

- **Weighted least-outstanding routing** — routes to the deployment with the lowest load relative to its capacity
- **429-aware cooldown** — respects `Retry-After` headers, automatically fails over to the next backend
- **Request class isolation** — separate concurrency budgets for light/medium/heavy requests prevent expensive operations from starving fast ones
- **Token-aware capacity tracking** — tracks actual token usage per deployment per minute window
- **Request deadline** — enforces a total timeout across all retry attempts
- **Health reporting** — exposes deployment health, inflight counts, 429 rates, and TPM usage

## Install

```bash
pip install llm-route
```

Or with uv:

```bash
uv add llm-route
```

Optional extras: `llm-route[entra]` (Microsoft Entra ID token auth via `azure-identity`),
`llm-route[anthropic]` (Claude deployments). The OpenAI SDK is supported on 2.30 and on 3.x.

## Quick Start

```python
import asyncio
from llm_route import SmartRouter, RequestClass, RouterConfig
from llm_route.config import DeploymentConfig

config = RouterConfig(
    deployments=[
        DeploymentConfig(
            name="eastus-1",
            endpoint="https://my-resource.openai.azure.com/",
            api_key="your-key",
            deployment_name="my-gpt6-deployment",  # your own name; never used to detect the model
            model="gpt-6-sol",  # what the deployment serves
            tpm_quota=120_000,
        ),
    ],
)

router = SmartRouter(config=config)

async def main():
    response = await router.complete(
        messages=[{"role": "user", "content": "Hello"}],
        request_class=RequestClass.LIGHT,
    )
    print(response.choices[0].message.content)

asyncio.run(main())
```

A legacy (non-reasoning) deployment declares its model the same way and keeps passing
sampling parameters through unchanged:

```python
DeploymentConfig(
    name="eastus-legacy",
    endpoint="https://my-resource.openai.azure.com/",
    api_key="your-key",
    deployment_name="prod-fast",
    model="gpt-4o",
    tpm_quota=60_000,
)
# await router.complete(messages=[...], temperature=0.2, max_tokens=500)  # sent as-is
```

## Configuration

### JSON config file

```json
{
  "deployments": [
    {
      "name": "eastus-1",
      "endpoint": "https://my-eastus-1.openai.azure.com/",
      "api_key": "your-api-key",
      "deployment_name": "my-gpt6-deployment",
      "model": "gpt-6-sol",
      "tpm_quota": 120000
    },
    {
      "name": "eastus-2",
      "endpoint": "https://my-eastus-2.openai.azure.com/",
      "api_key": "your-api-key",
      "deployment_name": "my-gpt6-deployment-2",
      "model": "gpt-6-sol",
      "tpm_quota": 60000
    }
  ],
  "default_timeout": 60.0,
  "max_retries": 3,
  "cooldown_seconds": 10.0,
  "concurrency": {
    "light": 20,
    "medium": 10,
    "heavy": 3
  }
}
```

Load it:

```python
config = RouterConfig.from_file("config.json")
router = SmartRouter(config=config)
```

### Environment variables

All settings can be set via env vars with `LLM_ROUTE_` prefix:

```bash
LLM_ROUTE_DEFAULT_TIMEOUT=60.0
LLM_ROUTE_MAX_RETRIES=3
LLM_ROUTE_COOLDOWN_SECONDS=10.0
```

## Authentication

Exactly one credential source per deployment (config validation rejects zero or several):

- **API key** — `api_key="..."`.
- **Microsoft Entra ID** — `entra=True` (needs `pip install 'llm-route[entra]'`; uses
  `DefaultAzureCredential` and `entra_scope`, default `https://ai.azure.com/.default`), or pass your
  own `token_provider` callable.
- **APIM / gateway header** — put the gateway's key in `default_headers`
  (`Ocp-Apim-Subscription-Key`, `api-key` or `Authorization`) and set no other credential. Header-only
  auth is **v1-surface only** (it is rejected when `api_version` is set), and the SDK's own
  `Authorization: Bearer` placeholder is omitted from every request so it never reaches the gateway.
  Non-auth headers (tracing, routing) never count as a credential source.

Credentials are excluded from `repr`, logs and the health report.

## API surfaces

By default Azure deployments use the api-version-less **v1** surface (`/openai/v1/`), which also
serves non-OpenAI Foundry models. Setting `api_version` (for example `"2024-12-01-preview"`) opts a
deployment into the legacy dated surface via `AsyncAzureOpenAI`; it is only needed for older
resources. `base_url` can replace `endpoint` for a gateway URL.

`router.complete()` uses Chat Completions. `router.respond()` uses the Responses API (needed by
Responses-only models) with the same selection, cooldown, concurrency and failover:

```python
result = await router.respond(
    input="Summarize this clause in one sentence.",
    reasoning={"effort": "low"},
    max_output_tokens=500,
)
print(result.response.output_text)
```

## How llm-route knows what a deployment accepts

Models differ in which parameters they take (sampling, `max_tokens` vs `max_completion_tokens`,
`reasoning_effort`, tools). The router resolves a **capability record** per deployment and shapes each
request to it. The deployment name is never an input.

Resolution order for the base record:

1. the model id the service **returned** (`response.model` on Chat Completions, from the warmup probe
   or the latest real response);
2. the deployment's configured `model`;
3. **unknown** — resolved conservatively (treated as a reasoning model: sampling parameters stripped,
   `max_completion_tokens`, effort passed through).

An explicit `capabilities` block on the deployment is merged on top, field by field, whatever the base
came from. The service outranks the config because Azure can change the model under a deployment name
on auto-upgrade; a disagreement is logged (`config_model_stale`, `deployment_model_changed`), never
silently followed, and capabilities are re-resolved when the returned model changes.

- **Legacy passthrough** — non-reasoning models (`gpt-4o`, `gpt-4.1`, `gpt-5-chat`, ...) receive every
  parameter they accept unchanged; only `reasoning_effort` is stripped.
- **Learning from a 400** — if the service rejects a parameter by name (`unsupported_parameter`), that
  parameter is stripped and the same deployment is re-called once per distinct parameter per request,
  then remembered for that deployment.
- **Warmup capability probe** — for a deployment whose model the built-in table does not know (or
  whose row is unverified), warmup sends a short matrix of one-token requests and builds the row from
  what the service actually accepted (`probe_capabilities`: `"unknown"` default, `"always"`,
  `"never"`; `probe_on_start` controls warmup).
- **Strict mode** — `strict_param_validation: true` raises `ReasoningModelParamError` instead of
  stripping anything, and re-raises a learnable 400 instead of learning it.
- **Anthropic** — Claude-family rows strip `temperature` / `top_p` / `top_k` where the model rejects
  them; `supports_sampling` and `capabilities` override. `reasoning_model` is deprecated (use
  `capabilities`).

Stripped parameters are reported on the router result (`sanitized_params`) and logged. Under strict
mode the Anthropic provider also raises for parameters with no Messages API equivalent
(`response_format`, `seed`, `logprobs`, `top_logprobs`, `n`, `presence_penalty`,
`frequency_penalty`) and untranslated ones (`tools`, `tool_choice`, `thinking`, ...).

**Claude refusals** — a safety-classifier refusal returns HTTP 200 with empty content
and `finish_reason == "content_filter"`. Callers that assume a successful response has
text should check `finish_reason` before using `choices[0].message.content`.

**`finish_reason == "length"` has two causes** — the `max_tokens` output cap
(`stop_reason: "max_tokens"`) and context-window exhaustion
(`stop_reason: "model_context_window_exceeded"`, logged as
`anthropic_context_window_exceeded`). Don't blindly retry with a larger `max_tokens`:
that fixes the first cause and makes the second worse. Shorten the input instead.

## Request Classes

Request classes provide concurrency isolation per deployment. Heavy requests (full document review) won't starve light ones (quick text search).

| Class | Default concurrency / deployment | Use case |
|-------|----------------------------------|----------|
| `LIGHT` | 20 | Short extraction, find-text, quick Q&A |
| `MEDIUM` | 10 | Clause analysis, section review |
| `HEAVY` | 3 | Full document review, redline, long synthesis |

```python
await router.complete(
    messages=[...],
    request_class=RequestClass.HEAVY,  # uses the heavy concurrency budget
)
```

## How Routing Works

1. **Filter** — remove disabled, cooled-down, and already-tried deployments
2. **Filter** — remove deployments with no available concurrency for the request class
3. **Score** — `inflight / weight` where `weight = tpm_quota / min_tpm`. Lower is better.
4. **Select** — pick lowest score; break ties by remaining TPM headroom
5. **Execute** — acquire semaphore, call Azure OpenAI with remaining deadline
6. **Failover** — on 429 or 5xx, mark cooldown, try next deployment
7. **Deadline** — if total timeout expires, raise `RouterExhaustedError`

## Health Monitoring

```python
health = router.health()
for dep in health.deployments:
    print(f"{dep.name}: healthy={dep.healthy}, inflight={dep.inflight}, "
          f"tpm={dep.tpm_used}/{dep.tpm_quota}, 429s={dep.total_429s}")
```

Expose as a FastAPI endpoint:

```python
@app.get("/health/llm")
async def llm_health():
    return router.health().model_dump()
```

## Dependencies

Minimal by design:

- `openai` — OpenAI / Azure OpenAI SDK (2.30 and 3.x)
- `pydantic` — data validation
- `pydantic-settings` — configuration management

Optional:

- `azure-identity` — Entra ID auth (`pip install llm-route[entra]`)
- `anthropic` — Claude deployments (`pip install llm-route[anthropic]`)
- `redis` — reserved for planned shared-state support (`pip install llm-route[redis]`). **Not yet implemented**: cooldown/TPM/concurrency state is currently per-process (the `redis_url` config field is accepted but unused). Multi-replica or multi-app deployments sharing the same backends will each track 429s independently until this lands.

## Security

This library was built in response to the [LiteLLM supply chain attack](https://github.com/BerriAI/litellm/issues/24518) (March 2026). Design principles:

- **Minimal dependencies** — 3 required packages, all well-maintained
- **No build-time code execution** — pure Python, no compiled extensions
- **`uv.lock` committed** — exact dependency tree is auditable
- **OIDC publishing** — no stored PyPI tokens in CI

## License

MIT
