Metadata-Version: 2.4
Name: prospect-cache-ai
Version: 1.0.1
Summary: Semantic cache + reverse proxy for LLM inference
Project-URL: Homepage, https://github.com/CraftedWithIntent/prospect-ai
Project-URL: Repository, https://github.com/CraftedWithIntent/prospect-ai.git
Project-URL: Issues, https://github.com/CraftedWithIntent/prospect-ai/issues
Project-URL: Documentation, https://github.com/CraftedWithIntent/prospect-ai#readme
Author-email: CraftedWithIntent <dev@crafted.ai>
License: MIT
License-File: LICENSE
Keywords: cache,llm,openai,proxy,semantic
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Requires-Dist: fastapi>=0.104.0
Requires-Dist: fastembed>=0.0.50
Requires-Dist: httpx>=0.25.0
Requires-Dist: pydantic>=2.0
Requires-Dist: typer>=0.9.0
Requires-Dist: uvicorn[standard]>=0.24.0
Provides-Extra: dev
Requires-Dist: build>=1.0.0; extra == 'dev'
Requires-Dist: pyright>=1.1.320; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
Requires-Dist: pytest>=7.4.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Description-Content-Type: text/markdown

# Prospect AI: Semantic Cache & Reverse Proxy for LLM Inference

![License](https://img.shields.io/badge/License-MIT-blue) ![Python](https://img.shields.io/badge/Python-3.11%2B-blue) ![Status](https://img.shields.io/badge/Status-Production%20Ready-green)

**High-performance reverse proxy for LLM APIs. Combines two-tier semantic caching (exact-match + embedding-based similarity) to cut token costs by 8-11x and latency to under 15ms.**

## The Problem

LLM inference is expensive. Applications pay full token costs even for semantically duplicate queries:
- "How do I reset my password?" 
- "I forgot my password, how to reset?"

Both queries mean the same thing, but standard caches achieve <5% hit rates on natural language.

**Result:** Wasted tokens, high costs, slow responses.

## The Solution

Prospect AI is an intelligent gateway between your app and LLM providers (OpenAI, Anthropic, etc.). It:
1. **Intercepts requests** — Acts as a drop-in reverse proxy
2. **Deduplicates** — Exact-match cache (L1) + semantic similarity (L2)
3. **Caches** — Stores responses with embeddings
4. **Returns** — Cached response in <15ms

**Result:** 40-55% cache hit rate, 8-11x fewer tokens, <15ms latency on hits.

## Installation

```bash
# Via pip
pip install prospect-cache-ai

# Via Docker
docker pull ghcr.io/craftedwithintent/prospect-ai:1.0.0
```

## Quick Start

### 1. Start the Proxy

```bash
prospect-ai start --port 8000 --similarity 0.92
```

### 2. Point Your App to the Proxy

**Before:**
```python
from openai import OpenAI
client = OpenAI(api_key="sk-...", base_url="https://api.openai.com/v1")
```

**After:**
```python
from openai import OpenAI
client = OpenAI(api_key="sk-...", base_url="http://localhost:8000/v1")
```

### 3. Monitor Performance

```bash
prospect-ai stats
```

**Output:**
```
Cache Statistics:
  Total Requests: 1,248
  Cache Hits: 742 (59.4%)
  Avg Latency (Hit): 8.2ms
  Avg Latency (Miss): 1,850ms
  Cost Saved: $18.50
```

## Key Features

| Feature | Details |
|---------|---------|
| **Two-Tier Cache** | L1: Exact match (<1ms) + L2: Semantic similarity (<15ms) |
| **OpenAI-Compatible** | Drop-in proxy for `/v1/chat/completions` |
| **Streaming Support** | Full `stream: true` support with zero overhead |
| **Smart Fallback** | Automatic failover on 429/5xx errors |
| **Storage Backends** | In-memory, SQLite-vec (zero external dependencies) |
| **CLI & Metrics** | Live dashboard + Prometheus export |
| **Production Ready** | 50+ tests, 80%+ coverage, strict typing |

## Performance

| Scenario | Latency | Improvement |
|----------|---------|------------|
| Cache Hit (L1 exact) | 0.8ms | 99% faster |
| Cache Hit (L2 semantic) | 12ms | 99% faster |
| Cache Miss (upstream) | 1,200–3,500ms | — |
| **Hit Rate** | **40–55%** | **8–11x token savings** |

## Architecture

```
Your App (OpenAI SDK)
    ↓
Prospect Proxy
    ├─→ L1: Exact match hash lookup (<1ms)
    ├─→ L2: Semantic similarity check (<15ms)
    ├─→ Cache Miss: Forward to OpenAI/Anthropic/etc
    ├─→ Store response + embedding
    └─→ Return to app
```

**Functional Core + Imperative Shell:**
- Pure logic: similarity scoring, request normalization, fallback routing
- I/O: FastAPI async server, Async httpx gateway, pluggable storage

## Usage Examples

### Basic Cache Monitoring

```bash
# Start proxy
prospect-ai start --port 8000 --similarity 0.92

# Check stats
prospect-ai stats

# List cached entries
prospect-ai cache list

# Clear cache
prospect-ai cache clear
```

### Docker Compose

```yaml
version: '3.9'
services:
  prospect:
    image: ghcr.io/craftedwithintent/prospect-ai:latest
    ports:
      - "8000:8000"
    environment:
      OPENAI_API_KEY: ${OPENAI_API_KEY}
  
  app:
    build: .
    ports:
      - "9000:9000"
    environment:
      OPENAI_BASE_URL: "http://prospect:8000/v1"
    depends_on:
      - prospect
```

## Full Production Example

Complete, tested example in **[examples/llm-chat-with-caching/](https://github.com/CraftedWithIntent/prospect-ai/tree/main/examples)**:

- ✅ FastAPI chat service with caching
- ✅ Real-time metrics endpoint
- ✅ Cost tracking and savings calculation
- ✅ 14 comprehensive tests
- ✅ Docker + Kubernetes deployment

**Run locally:**

```bash
cd examples/llm-chat-with-caching
pip install -r requirements.txt
export OPENAI_API_KEY="your-key"

# Terminal 1: Start proxy
prospect-ai start --port 8000

# Terminal 2: Start app
python app.py

# Terminal 3: Make requests
curl -X POST http://localhost:9000/chat \
  -H "Content-Type: application/json" \
  -d '{"content": "What is AI?"}'
```

## Configuration

### Start Options

```bash
prospect-ai start [OPTIONS]

Options:
  --port PORT              Listening port (default: 8000)
  --similarity THRESHOLD   L2 threshold 0.0–1.0 (default: 0.92)
  --workers WORKERS        Worker threads (default: 4)
  --backend BACKEND        Cache backend: memory, sqlite-vec (default: memory)
  --log-level LEVEL        Logging: debug, info, warning, error (default: info)
```

### cache.yaml (Optional)

```yaml
server:
  port: 8000
  workers: 4

cache:
  similarity_threshold: 0.92
  backend: "memory"
  max_entries: 10000
  ttl_seconds: 3600

upstream:
  provider: "openai"
  api_key: "${OPENAI_API_KEY}"
  fallback:
    - provider: "anthropic"
      api_key: "${ANTHROPIC_API_KEY}"
```

## Testing

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run all tests
pytest tests/ -v --cov=src/prospect_ai

# Run specific test suite
pytest tests/test_semantic_cache.py -v
```

**Coverage:** 80%+ (enforced)

## Deployment

### Local Development

```bash
pip install -e ".[dev]"
prospect-ai start --port 8000
```

### Docker

```bash
docker run -p 8000:8000 \
  -e OPENAI_API_KEY=sk-... \
  ghcr.io/craftedwithintent/prospect-ai:1.0.0
```

### Kubernetes

See [docs/DEPLOYMENT.md](https://github.com/CraftedWithIntent/prospect-ai/blob/main/docs/DEPLOYMENT.md) for full K8s manifest.

## Documentation

- **[Full README](https://github.com/CraftedWithIntent/prospect-ai)** — Comprehensive guide
- **[Deployment Guide](https://github.com/CraftedWithIntent/prospect-ai/blob/main/docs/DEPLOYMENT.md)** — Production setup
- **[Architecture](https://github.com/CraftedWithIntent/prospect-ai/blob/main/docs/adr/001-architecture.md)** — Design decisions
- **[CONTRIBUTING.md](https://github.com/CraftedWithIntent/prospect-ai/blob/main/CONTRIBUTING.md)** — Development guide
- **[Examples](https://github.com/CraftedWithIntent/prospect-ai/tree/main/examples)** — Production-ready code

## Support

- 🐛 [GitHub Issues](https://github.com/CraftedWithIntent/prospect-ai/issues) — Bug reports, features
- 💬 [GitHub Discussions](https://github.com/CraftedWithIntent/prospect-ai/discussions) — Questions, ideas
- 📖 [Full Repository](https://github.com/CraftedWithIntent/prospect-ai) — Source, docs, examples

## License

MIT License — See [LICENSE](https://github.com/CraftedWithIntent/prospect-ai/blob/main/LICENSE) for details.

---

**Completely standalone:** Works with any OpenAI-compatible API. Part of the [CraftedWithIntent](https://github.com/CraftedWithIntent) ecosystem for production AI systems.
