Metadata-Version: 2.4
Name: ai-switch
Version: 0.4.0
Summary: Free-tier AI provider router: automatic fallback across Groq, OpenRouter, NVIDIA and more.
Author: Dhinakaran Thangaraj
License-Expression: MIT
Project-URL: Homepage, https://github.com/Dheena731/llmswitch
Project-URL: Documentation, https://dheena731.github.io/llmswitch/
Project-URL: Repository, https://github.com/Dheena731/llmswitch
Project-URL: Issues, https://github.com/Dheena731/llmswitch/issues
Keywords: llm,ai,groq,openrouter,nvidia,fallback,router,rate-limit
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# LLMSwitch

Free-tier AI provider router for developers. Write AI code once; LLMSwitch handles provider fallback across Groq, OpenRouter, NVIDIA, and more.

## Install

```bash
pip install ai-switch
```

The distribution is named `ai-switch`; the import stays `llmswitch`.

## Configure

Set an API key for any provider you want LLMSwitch to use — it only tries providers with a key present:

```bash
export GROQ_API_KEY="..."
export CEREBRAS_API_KEY="..."
export GEMINI_API_KEY="..."
export OPENROUTER_API_KEY="..."
export NVIDIA_API_KEY="..."
export MISTRAL_API_KEY="..."
export HF_TOKEN="..."
```

| Provider | Env var | Order |
|---|---|---|
| Groq | `GROQ_API_KEY` | 1 |
| Cerebras | `CEREBRAS_API_KEY` | 2 |
| Gemini | `GEMINI_API_KEY` | 3 |
| OpenRouter | `OPENROUTER_API_KEY` | 4 |
| NVIDIA | `NVIDIA_API_KEY` | 5 |
| Mistral | `MISTRAL_API_KEY` | 6 |
| Hugging Face | `HF_TOKEN` | 7 |
| Ollama (local) | `OLLAMA_HOST` | 8 — last resort |

### Ollama as a local last resort

Ollama needs no API key and never rate-limits, so it makes a good final fallback when every
cloud free tier is spent. It is **opt-in** — set `OLLAMA_HOST` to enable it, so users without a
local server don't pay for a failed attempt on every request:

```bash
export OLLAMA_HOST="http://localhost:11434/v1"
```

## Usage

```python
from llmswitch import AIClient

client = AIClient()
response = client.chat("Write a Python web scraper")
print(response)
```

Providers are tried in order (Groq, OpenRouter, NVIDIA by default). If one is rate-limited, unreachable, or errors, LLMSwitch automatically falls back to the next.

### Model names

`model` accepts either a friendly alias (`"fast-model"`, `"smart-model"` — see [llmswitch/registry.py](llmswitch/registry.py)) that's resolved per-provider, or a raw provider-specific model name passed straight through:

```python
client.chat("hello", model="fast-model")
client.chat("hello", model="llama-3.1-8b-instant")  # Groq-specific name
```

### Health tracking

LLMSwitch remembers which providers just failed, so a rate-limited provider isn't retried on every call — it's rested and moved to the back of the queue:

```python
client.chat("hi")      # groq is rate-limited -> falls back to openrouter
client.chat("hi")      # groq skipped entirely; openrouter served directly
```

Cooldowns respect the provider's own `Retry-After` header when present, and otherwise back off exponentially (60s, 120s, 240s… capped at 15 min for rate limits; shorter for transient outages). A success clears the streak.

If *every* provider is resting, LLMSwitch still tries them — closest-to-recovery first — rather than failing without an attempt.

Inspect current state at any time:

```python
client.status()
# {'groq': {'available': False, 'cooldown_remaining': 57.0, ...},
#  'openrouter': {'available': True, 'total_successes': 2, ...}}
```

### Streaming

```python
from llmswitch import AIClient, StreamInterrupted

with AIClient() as client:
    try:
        for token in client.stream("Write a haiku about failover"):
            print(token, end="", flush=True)
    except StreamInterrupted as exc:
        print(f"\n[broke on {exc.provider}, kept: {exc.partial!r}]")
```

Fallback is transparent **only up to the first token**. Once tokens have been handed to you they
can't be un-yielded, so a later failure raises `StreamInterrupted` carrying the partial text rather
than silently restarting the answer on another provider mid-sentence.

### Async

```python
import asyncio
from llmswitch import AsyncAIClient

async def main():
    async with AsyncAIClient() as client:
        print(await client.chat("hello"))
        async for token in client.stream("count to five"):
            print(token, end="")

asyncio.run(main())
```

`AsyncAIClient` shares the sync client's routing core, so fallback, cooldowns, and retry behave
identically — only the transport differs.

### Connections and cleanup

Clients pool connections per provider, so closing them matters. Use a context manager, or call
`close()` / `await aclose()` yourself:

```python
with AIClient() as client:      # closes on exit
    client.chat("hi")
```

### Retries

A timeout usually means one unlucky request rather than a dead provider, so LLMSwitch retries the
same provider once before failing over. A `429` is never retried — the provider explicitly said
stop.

### Choosing providers explicitly

```python
from llmswitch import AIClient
from llmswitch.providers import GroqProvider, NVIDIAProvider

client = AIClient(providers=[GroqProvider(), NVIDIAProvider()])
```

See [examples/](examples/) for more.

## Tests

```bash
pip install -e ".[dev]"
pytest
```

## Changelog

See [CHANGELOG.md](CHANGELOG.md) for what changed between releases.

## Status

v0.4.0 on PyPI — eight providers, health tracking, streaming, async, 150 tests. Still 0.x: the API may shift before
1.0, and five of the eight adapters have verified endpoints but have not completed a request with
a real key. See [roadmap.md](roadmap.md) for the plan and [todo.md](todo.md) for current progress.
