Metadata-Version: 2.4
Name: pyinferencemanager
Version: 1.0.0
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Typing :: Typed
Requires-Dist: anthropic>=0.100.0
Requires-Dist: pytest>=7.0 ; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21 ; extra == 'dev'
Requires-Dist: maturin>=1.0 ; extra == 'dev'
Requires-Dist: ruff>=0.5.0 ; extra == 'dev'
Provides-Extra: dev
License-File: LICENSE
Summary: Intelligent AI workload orchestrator. Multi-provider routing, cost optimization, semantic caching, dynamic routing with budget enforcement. Production observability included.
Keywords: ai,orchestrator,llm,ollama,anthropic,openai,local-first,cost-optimization,caching,multi-provider,routing,inference,workload-orchestration,hardware-aware,semantic-caching,distributed-tracing,observability,budget-enforcement
Author-email: Georgi Mammen Mullassery <mullassery@gmail.com>
Requires-Python: >=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/Mullassery/pyinferencemanager/releases
Project-URL: Documentation, https://github.com/Mullassery/pyinferencemanager/blob/main/README.md
Project-URL: Homepage, https://github.com/Mullassery/pyinferencemanager
Project-URL: Issues, https://github.com/Mullassery/pyinferencemanager/issues
Project-URL: Repository, https://github.com/Mullassery/pyinferencemanager.git

# PyInferenceManager

**Use any LLM. Switch models without rewriting code. Cut inference costs 40-60%.**

Intelligently route requests across Claude, GPT-4, Gemini, Llama, Mistral, and more based on cost, speed, or availability. One line of code. Automatic failover. No vendor lock-in.

[![PyPI](https://img.shields.io/pypi/v/pyinferencemanager)](https://pypi.org/project/pyinferencemanager)
[![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-blue)](https://www.python.org)
[![Tests Passing](https://img.shields.io/badge/tests-passing-success)](./tests)
[![License: Proprietary](https://img.shields.io/badge/License-Proprietary-blue.svg)](./LICENSE)

---

## 30-Second Start

```python
from pyinferencemanager import Manager

# Create manager (routes across all providers)
mgr = Manager()

# Same code. Different models. Different costs.
response = mgr.chat(
    "Which LLM should I use for this task?",
    prefer="cheapest"  # or "fastest" or "highest-quality"
)

print(response.text)
print(f"Used: {response.model}")  # Which provider was chosen?
print(f"Cost: ${response.cost:.4f}")  # How much did it cost?
```

---

## Why PyInferenceManager?

**The Problem:**
- Each LLM has different APIs (Claude, OpenAI, Google, Anthropic, etc.)
- Costs vary wildly (GPT-4 is 10x more expensive than Llama)
- You can't switch models without rewriting your code
- Provider outages break your application

**The Solution:**
- One unified API for all LLM providers
- Automatic routing based on cost, speed, or quality
- Provider failover (if Claude is down, switch to GPT-4 automatically)
- Easy cost comparison and optimization

---

## Key Features

- **11 Providers:** Claude (Anthropic), GPT-4/3.5 (OpenAI), Gemini (Google), Llama (Meta), Mistral, Cohere, PaLM, Falcon, and more
- **Smart Routing:** Automatic selection based on cost/speed/quality
- **Cost Tracking:** Real-time cost estimation and reporting
- **Failover:** Automatic provider switching if one goes down
- **Batch Processing:** Process 1000s of requests with automatic optimization
- **Streaming Support:** Get responses as they arrive
- **Rate Limiting:** Built-in quotas and backoff

---

## Real-World Use Cases

**Cost Optimization:**
```python
# Cheap tasks use Llama, complex tasks use Claude
response = mgr.chat(prompt, prefer="cheapest")
# Llama for summarization: $0.0001
# Claude for reasoning: $0.001
# Automatic choice based on task difficulty
```

**Reliability:**
```python
# If Claude API is down, automatically use GPT-4
response = mgr.chat(prompt, fallback="gpt-4")
```

**Multi-Model Comparison:**
```python
# Test a prompt across all providers
for model in ["claude", "gpt-4", "gemini", "llama"]:
    result = mgr.chat(prompt, model=model)
    print(f"{model}: ${result.cost}")
```

---

## Provider Comparison

| Provider | Speed | Cost | Quality | Notes |
|----------|-------|------|---------|-------|
| Claude 3 Opus | Fast | $$ | Excellent | Best reasoning |
| GPT-4 | Medium | $$$ | Excellent | General purpose |
| Gemini | Fast | $ | Good | Great value |
| Llama 2 | Slow | $ | Good | Local option |
| Mistral | Fast | $ | Good | European option |

---

## Installation

```bash
pip install pyinferencemanager
# or with uv
uv pip install pyinferencemanager
```

Set API keys (one time):
```bash
export ANTHROPIC_API_KEY=sk-...
export OPENAI_API_KEY=sk-...
export GOOGLE_API_KEY=goog-...
```

---

## Documentation

- [Quick Start](docs/QUICKSTART.md) — Get your first request working
- [Providers](docs/PROVIDERS.md) — How to connect to each service
- [Routing Strategies](docs/ROUTING.md) — Cost vs. speed vs. quality
- [Examples](examples/) — Real-world applications

---

## License

Proprietary License - Free to use with explicit attribution. See [LICENSE](LICENSE).

---

**PyInferenceManager v2.0.0** | Smart LLM routing | Python 3.10+

## License

MIT

---

**MCP 2.0 Mega-Platform | v2.0.0 | Wheels-Only Distribution**

